AIMultiple ResearchAIMultiple Research

Cloud GPUs for Deep Learning: Availability& Price / Performance

Written by
Cem Dilmegani
Cem Dilmegani
Cem Dilmegani

Cem is the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per Similarweb) including 60% of Fortune 500 every month.

Cem's work focuses on how enterprises can leverage new technologies in AI, automation, cybersecurity(including network security, application security), data collection including web data collection and process intelligence.

View Full Profile
Cloud GPUs for Deep Learning: Availability& Price / PerformanceCloud GPUs for Deep Learning: Availability& Price / Performance

If you are flexible about the GPU model, identify the most cost-effective cloud GPU

If you prefer a specific model (e.g. A100), identify the GPU cloud providers offering it.

If undecided between on-prem and the cloud, explore whether to buy or rent GPUs on the cloud.

Cloud GPU price per throughput

Two common pricing models for GPUs are “on-demand” and “spot” instances.

On-demand GPUs from big tech cloud providers

This is the most straightforward pricing model where you pay for the compute capacity by the hour or second, depending on what you use with no long-term commitments or upfront payments. These instances are recommended for users who prefer the flexibility of a cloud GPU platform without any up-front payment or long-term commitment. On-demand instances are usually more expensive than spot instances, but they provide guaranteed uninterrupted capacity.

CloudGPU Type / Memory*# of GPUsOn-demand $Throughput**Throughput** / $***
AzureA100 / 80 GB13.6723263
AzureA100 / 80 GB414.6982156
AzureA100 / 80 GB27.3540655
GCPA100 / 40 GB13.6717949
AzureA100 / 80 GB837.181,36237
AWSA100 / 80 GB840.971,36233
GCPV100 / 16 GB12.954214
AWSV100 / 16 GB13.064214
AzureV100 / 16 GB13.064214
GCPV100 / 16 GB25.917713
GCPV100 / 16 GB411.8115313
AWSV100 / 16 GB412.2415313
AzureV100 / 16 GB26.127713
AzureV100 / 16 GB412.2415313
GCPV100 / 16 GB823.6328912
AWSV100 / 16 GB824.4828912

* Memory and GPU model are not the only parameters. CPUs and RAM can also be important, however, they are not the primary criteria that shape cloud GPU performance. Therefore, for simplicity, we have not included number of CPUs or RAM in these tables.

** Training throughput is a good metric to measure relative GPU effectiveness. It measures the number of tokens processed per second by the GPU for a language model (i.e. bert_base_squad).1 Please note that these throughput values should serve as high level guidelines. The same hardware would have a significantly different throughput for your workload since there is significant throughput difference even between LLMs running on the same hardware.2

*** Excludes cost of storage, network performance, ingress/egress etc. This is only the GPU cost.3

On-demand GPUs from other cloud providers

CloudGPU Type / Memory*# of GPUsOn-demand $Throughput**Throughput** / $***
TensorDockA100 / 80 GB41.200821684
Vast.aiV100 / 16 GB80.944289306
Vast.aiV100 / 16 GB20.35877215
TensorDockA100 / 80 GB11.200232193
TensorDockV100 / 16 GB10.22042191
TensorDockA100 / 80 GB11.400232165
JarvislabsA100 / 40 GB11.1179163
LambdaA100 / 40 GB11.1179163
LambdaH100 / 80 GB11.99322162
Crusoe CloudA100 / 80 GB11.650232140
FluidStackA100 / 40 GB11.290179139
FluidStackA100 / 40 GB11.400179128
Vast.aiA100 / 40 GB11.400179128
DatacrunchA100 / 80 GB11.85232125
Crusoe CloudA100 / 80 GB46.600821124
Crusoe CloudA100 / 40 GB11.450179123
Crusoe CloudA100 / 80 GB23.300406123
SeewebRTX A6000 / 48 GB21.480179121
Vast.aiA100 / 80 GB12.000232116
LambdaA100 / 80 GB8121,362114
DatacrunchA100 / 80 GB47.4821111
DatacrunchA100 / 80 GB23.7406110
CoreWeaveA100 / 80 GB12.210232105
CoreWeaveA100 / 80 GB12.210232105
FluidStackA100 / 80 GB12.210232105
FluidStackA100 / 80 GB12.210232105
SeewebA100 / 80 GB12.220232104
Crusoe CloudA100 / 80 GB813.2001,362103
Vast.aiA100 / 80 GB48.000821103
Vast.aiA100 / 80 GB24.000406101
DatacrunchA100 / 80 GB814.81,36292
SeewebA100 / 80 GB48.88082192
Oblivus CloudA100 / 80 GB12.5523291
SeewebA100 / 80 GB24.44040691
RunPodA100 / 80 GB815.121,36290
VultrA100 / 80 GB12.60423289
CoreWeaveA100 / 40 GB12.06017987
CoreWeaveA100 / 40 GB12.06017987
Crusoe CloudA100 / 80 GB815.6001,36287
Oblivus CloudA100 / 80 GB25.140680
Oblivus CloudA100 / 80 GB410.282180
VultrA100 / 80 GB410.41782179
VultrA100 / 80 GB25.20840678
Latitude.shH100 (80 GB)835.22,69377
CoreWeaveH100 / 80 GB14.25032276
FluidStackH100 / 80 GB14.25032276
Latitude.shH100 (80 GB)417.61,32175
Oblivus CloudA100 / 40 GB12.3917975
ACE CloudA100 / 80 GB13.11023274
Latitude.shH100 (80 GB)14.432273
PaperspaceA100 / 80 GB13.1823273
FluidStackH100 / 80 GB14.76032268
CoreWeaveH100 / 80 GB14.78032267
Oblivus CloudA100 / 80 GB820.41,36267
LambdaV100 / 16 GB84.428966
ACE CloudA100 / 80 GB26.20040665
Oblivus CloudV100 / 16 GB10.654265
PaperspaceA100 / 80 GB412.7282165
VultrA100 / 80 GB820.8331,36265
PaperspaceA100 / 80 GB26.3640664
Latitude.shA100 (80 GB)823.21,36259
Oblivus CloudV100 / 16 GB21.37759
Oblivus CloudV100 / 16 GB42.615359
PaperspaceA100 / 40 GB13.0917958
PaperspaceA100 / 80 GB825.441,36254
CoreWeaveV100 / 16 GB10.8004253
CirrascaleA100 / 80 GB826.0301,36252
ExoscaleV100 / 16 GB43.3215346
ACE CloudA100 / 80 GB29.28040644
VultrH100 / 80 GB17.532243
DatacrunchV100 / 16 GB114242
DatacrunchV100 / 16 GB227739
DatacrunchV100 / 16 GB4415338
ExoscaleV100 / 16 GB22.017738
CirrascaleA100 / 80 GB422.96082136
DatacrunchV100 / 16 GB8828936
ExoscaleV100 / 16 GB11.384230
OVHcloudV100 / 16 GB11.974221
OVHcloudV100 / 16 GB23.947720
OVHcloudV100 / 16 GB47.8915319
PaperspaceV100 / 16 GB12.34218

Spot GPUs

CloudGPU Type / Memory*# of GPUsSpotThroughput**Throughput** / $***
AzureA100 / 80 GB10.76232303
AzureA100 / 80 GB43.05821269
AzureA100 / 80 GB21.53406266
JarvislabsA100 / 40 GB10.79179227
GCPA100 / 40 GB11.62179111
AWSV100 / 16 GB10.924246
AWSV100 / 16 GB43.6715342
AzureV100 / 16 GB11.044240
AWSV100 / 16 GB87.3428939
AzureV100 / 16 GB22.087737
AzureV100 / 16 GB44.1615337

In all these throughput per dollar tables:

  • Not all possible configurations are listed, more commonly used, deep learning focused configurations are included.
  • West or Central US regions were used where possible
  • These are the list prices for each category, high volume buyers can possibly get better pricing

Finally, it is good to clarify what “spot” means. Spot resources are:

Interruptible so users need to keep on recording their progress. For example, Amazon EC2 P3, which provides V100 32 GB, is one of the most frequently interrupted Amazon spot services.4

Offered on a dynamic, market-driven basis. The price for these GPU resources can fluctuate based on supply and demand, and users typically bid on the available spot capacity. If a user’s bid is higher than the current spot price, their requested instances will run.

Cloud GPU availability in different clouds

Input the model that you want in the search box to identify all cloud providers that offer it:

ProviderGPUMulti-GPUOn-demand $ per single GPU hour***
AWSM60 8 GB1, 2, 4x1.14
AWST4 16 GB1, 2, 4, 8x1.20
AWSA10G 24 GB1, 4, 8x1.62
AWSV100 16 GB1, 4, 8x3.06
AWSV100 32 GB8x3.90****
AWSA100 40 GB8x4.10****
AWSA100 80 GB8x5.12****
AzureK80 12 GB1, 2, 4x0.90
AzureT4 16 GB1, 4x1.20
AzureP40 24 GB1, 2, 4x2.07
AzureP100 16 GB1, 2, 4x2.07
AzureV100 32 GB8x2.75
AzureV100 16 GB1, 2, 4x3.06
AzureA100 40 GB8x3.40****
AzureA100 80 GB1, 2, 4x3.67
AzureA100 80 GB8x4.10****
GCPT4 16 GB1, 2, 4x0.75
GCPK80 12 GB1, 2, 4, 8x0.85
GCPP4 8 GB1, 2, 4x1.00
GCPP100 16 GB1, 2, 4x1.86
GCPV100 16 GB1, 2, 4, 8x2.88
GCPA100 40 GB1, 2, 4, 8, 16x3.67
OCIA100 40 GB8x4.00
OCIA100 80 GB8x3.05
OCIA10 24 GB1,2,4x2.00
OCIV100 16 GB1,2,4,8x2.95
OCIP100 16 GB1,2x1.275
ACE CloudA2 (16 GB)1, 2x0.59
ACE CloudA30 (32 GB)1, 2x0.95
ACE CloudA100 (80 GB)1, 2x3.11
Alibaba CloudA100 80 GB8x
CirrascaleA100 (80 GB)4, 8x5.74
CirrascaleRTX A6000 (48 GB)8x1.12
CirrascaleRTX A5000 (24 GB)8x0.51
CirrascaleRTX A4000 (16 GB)8x0.34
CirrascaleA40 (48 GB)8x1.44
CirrascaleA30 (24 GB)8x
CirrascaleV100 (32 GB)4, 8x1.92
CirrascaleRTX 6000 (48GB)8x1.18
CoreWeaveH100 (80 GB)1x4.25
CoreWeaveA100 (80 GB)1x2.21
CoreWeaveA100 (40 GB)1x2.06
CoreWeaveV100 (16 GB)1x0.80
CoreWeaveA40 (48 GB)1x1.28
CoreWeaveRTX 6000 (48 GB)1x1.28
CoreWeaveRTX 5000 (24 GB)1x0.77
CoreWeaveRTX 4000 (16 GB)1x0.61
CoreWeaveQuadro RTX 5000 (16 GB)1x0.57
CoreWeaveQuadro RTX 4000 (8 GB)1x0.24
Crusoe CloudA6000 (48 GB)1, 2, 4, 8x0.92
Crusoe CloudA40 (48 GB)1, 2, 4, 8x1.10
Crusoe CloudA100 (80 GB)1, 2, 4, 8x1.45
Crusoe CloudH100 (80 GB)8x
FluidStackH100 (80 GB)1x4.25
FluidStackA100 (80 GB)1x2.21
Jarvis LabsQuadro RTX 5000 16 GB1x0.49
Jarvis LabsQuadro RTX 6000 24 GB1x0.99
Jarvis LabsRTX A5000 24 GB1x1.29
Jarvis LabsRTX A6000 48 GB1x1.79
Jarvis LabsA100 40 GB1x2.39
Lambda LabsQuadro RTX 6000 24 GB1, 2, 4x1.25
Lambda LabsRTX A6000 48 GB1, 2, 4x1.45
Lambda LabsV100 16 GB8x6.8
Latitude.shH100 (80 GB)1, 4, 8x4.40
Latitude.shA100 (80 GB)8x23.2
LeaderGPUA100 (40 GB)
LeaderGPUA10 (24 GB)
LeaderGPUV100 (32 GB)
LinodeQuadro RTX 6000 24 GB1, 2, 4x1.50
OVHV100 32 GB1, 2, 4x1.99
OVHV100 16 GB1, 2, 4x1.79
PaperspaceQuadro M4000 8 GB1x0.45
PaperspaceQuadro P4000 8 GB1, 2, 4x0.51
PaperspaceQuadro RTX 4000 8 GB1, 2, 4x0.56
PaperspaceRTX A4000 16 GB1, 2, 4x0.76
PaperspaceQuadro P5000 16 GB1, 2, 4x0.78
PaperspaceQuadro RTX 5000 16 GB1, 2, 4x0.82
PaperspaceQuadro P6000 24 GB1, 2, 4x1.10
PaperspaceRTX A5000 24 GB1, 2, 4x1.38
PaperspaceRTX A6000 48 GB1, 2, 4x1.89
PaperspaceV100 32 GB1, 2, 4x2.30
PaperspaceV100 16 GB1x2.30
PaperspaceA100 40 GB1x3.09
PaperspaceA100 80 GB1, 2, 4, 8x3.19
SeewebRTXA6000 (48 GB)1, 2, 3, 4, 5x0.74
SeewebRTXA6000 (24 GB)1, 2, 3, 4, 5x0.64
SeewebA30 (24 GB)1, 2, 3, 4, 5x0.64
SeewebL4 (24 GB)1, 2, 3, 4, 5x0.38
SeewebA100 (80 GB)1, 2, 3, 4, 5x2.22
TensorDockA100 (80 GB)1x1.40
TensorDockL40 (40 GB)1x1.05
TensorDockV100 (16 GB)1x0.22
TensorDockA6000 (48 GB)1x0.47
TensorDockA40 (48 GB)1x0.47
TensorDockA5000 (24 GB)1x0.21
TensorDockA4000 (16 GB)1x0.13
TensorDockRTX 4090 (24 GB)1x0.37
TensorDockRTX 3090 (24 GB)1x0.22
TensorDockRTX 3080 Ti (12 GB)1x0.17
TensorDockRTX 3080 (10 GB)1x0.17
TensorDockRTX 3070 Ti (8 GB)1x0.14
TensorDockRTX 3060 Ti (8 GB)1x0.10
TensorDockRTX 3060 (12 GB)1x0.10
Vast.aiL40 (45 GB)1, 2, 4x1.10
Vast.aiA100 (40 GB)1, 2, 4x1.40
Vast.aiA40 (48 GB)1, 2x0.40
Vast.aiA6000 (24 GB)1, 2, 4, 8x0.44
Vast.aiA5000 (24 GB)1, 2, 4, 8x0.20
Vast.aiA4000 (16 GB)1, 2, 4, 5, 8x0.15
Vast.aiV100 (16 GB)2, 5x0.18
Voltage ParkH100 80 GB8x1.89****
VultrL40S 48 GB1, 2, 4, 8x1.75
VultrH100 80 GB1x7.50
VultrA100 80 GB1, 2, 4, 8x2.60
VultrA40 (48 GB)1, 4x1.83
VultrA16 (16 GB)1, 2, 4, 8, 16x0.51

**** Computed values. This was needed when single GPU instances were not available.5 6

Other cloud GPU considerations

Availability: Not all GPUs listed above may be available due to capacity constraints of the cloud providers and increasing demand for generative AI.

Data security: For example, cloud GPU marketplaces like Vast.ai offer significantly lower prices but depending on the specific resource requested, the data security of the workload could be impacted, givings hosts the capability to access workloads. Since we prioritized enterprise GPU needs, Vast.ai wasn’t included in this benchmark.

Ease of use: Documentation quality is a subjective metric but developers prefer some cloud providers’ documentation over others. In this discussion, GCP’s documentation was mentioned as lower quality than other tech giants’.7

Familiarity: Even though cloud providers put significant effort into making their services easy-to-use, there is a learning curve. That is why major cloud providers have certifications systems in place. Therefore, for small workloads, the cost savings of using a low cost provider may be less than the opportunity cost of the time it takes a developer to learn how to use their cloud GPU offering.

Buy GPUs or rent cloud GPUs

Buying makes sense

– If your company has the know-how and preference to host the servers or manage colocated servers.

– For uninterruptible workloads: For the volume of GPUs for which you can ensure a high utilization (e.g. more than 80%) for a year or more.8

– For interruptible workloads: The high utilization period quoted above needs to be a few times longer since on-demand (uninterruptible computing) prices tends to be a few times more expensive than spot (interruptible computing) prices.

Our recommendation for businesses with heavy GPU workloads is a mix of owned and rented GPUs where guaranteed demand runs on owned GPUs and variable demand runs on the cloud. This is why tech giants like Facebook are building their own GPU clusters including hundreds of GPUs.9

Buyers may be tempted to consider consumer GPUs which offer a better price/performance ratio however, the EULA of their software prohibits their use in data centers.10 Therefore, they are not a good fit for machine learning except for minor testing workloads on data scientists’ machines.

What are the top cloud GPU hardware?

Almost all cloud GPUs use Nvidia GPU instances. AMD and other providers also offer GPUs however due to various reasons (e.g. limited developer adoption, lower price per performance etc.), their GPUs are not as widely demanded as Nvidia GPUs.

To see cloud GPU providers that offer non-Nvidia GPUs, please check out the comprehensive list of cloud GPU providers.

Read about all AI chips / hardware.

What are the top cloud GPU platforms?

Top cloud GPU providers are:

  • AWS
  • Microsoft Azure
  • CoreWeave
  • Google Cloud Platform (GCP)
  • IBM Cloud
  • Jarvis Labs
  • Lambda Labs
  • NVIDIA DGX Cloud
  • Oracle Cloud Infrastructure (OCI)
  • Paperspace CORE

For more on these providers, check out cloud gpu providers.

If you are not sure about cloud GPUs, explore other options like serverless GPU.

For tools allow users to collaborate on, check out cloud collaboration tools.

If you are unclear about what cloud GPUs are, here is more context:

What is a cloud GPU?

Unlike a CPU, which may have a relatively small number of cores optimized for sequential serial processing, a GPU can have hundreds or even thousands of smaller cores designed for multi-threading and handling parallel processing workloads.

A cloud GPU refers to a certain way of GPU usage that’s provided as a service through cloud computing platforms. Much like traditional cloud services, a cloud gpu allows you to access high-performance computing resources spot or on-demand, without the need for upfront capital investment in hardware.

What are the functions/application areas of cloud GPUs?

Cloud GPUs are primarily used for processing tasks that require high computational power. Here are some of the primary uses for cloud GPUs:

Machine Learning and AI

GPUs are particularly effective at handling the complex calculations required for machine learning (ML) and artificial intelligence (AI) models. They can process multiple computations in parallel, making them suitable for training large neural networks and algorithms.

Deep learning

Deep learning is a sub-field of machine learning. Deep learning algorithms greatly benefit from the parallel processing capabilities of GPUs, making training and inference faster and more efficient.

Data processing

Data analysis

GPUs are used to accelerate computing and data processing tasks, such as Big Data analysis and real-time analytics. They can handle high-throughput, parallel processing tasks more efficiently than CPUs.

Scientific computing

In scientific research, cloud GPUs can handle computations for simulations, bioinformatics, quantum chemistry, weather modeling, and more.

Simulations

Certain complex simulations can run more efficiently on GPUs.

Gaming & entertainment

Cloud GPUs are used to provide cloud gaming services, such as Google’s Stadia or NVIDIA’s GeForce Now, where the game runs on a server in the cloud, and the rendered frames are streamed to the player’s device. This allows high-quality gaming without the need for a powerful local machine.

Graphics rendering

GPUs were initially designed to handle computer graphics, and they still excel in this area. Cloud GPUs are used for 3D modeling and rendering, 3D visualizations, virtual reality (VR), computer-aided design (CAD), and computer-generated imagery (CGI).

Video processing

They’re used in video encoding and decoding, video editing, color correction, effects rendering, and other video processing tasks.

Cryptocurrency mining

GPUs are also used in tasks like cryptocurrency mining. However application-specific integrated circuits (ASICs) are offering better economics for more commonly mined crypto currencies.

Notes

Cloud providers are constantly updating their offering, part of this research could be outdated.

AIMultiple’s research is sponsored by cloud GPU providers. Sponsors can be identified by links to their websites and they are ranked at the top of lists that they participate in.

Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on
Cem Dilmegani
Principal Analyst

Cem is the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per Similarweb) including 60% of Fortune 500 every month.

Cem's work focuses on how enterprises can leverage new technologies in AI, automation, cybersecurity(including network security, application security), data collection including web data collection and process intelligence.

Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.

Cem's hands-on enterprise software experience contributes to the insights that he generates. He oversees AIMultiple benchmarks in dynamic application security testing (DAST), data loss prevention (DLP), email marketing and web data collection. Other AIMultiple industry analysts and tech team support Cem in designing, running and evaluating benchmarks.

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

Sources:

AIMultiple.com Traffic Analytics, Ranking & Audience, Similarweb.
Why Microsoft, IBM, and Google Are Ramping up Efforts on AI Ethics, Business Insider.
Microsoft invests $1 billion in OpenAI to pursue artificial intelligence that’s smarter than we are, Washington Post.
Data management barriers to AI success, Deloitte.
Empowering AI Leadership: AI C-Suite Toolkit, World Economic Forum.
Science, Research and Innovation Performance of the EU, European Commission.
Public-sector digitization: The trillion-dollar challenge, McKinsey & Company.
Hypatos gets $11.8M for a deep learning approach to document processing, TechCrunch.
We got an exclusive look at the pitch deck AI startup Hypatos used to raise $11 million, Business Insider.

To stay up-to-date on B2B tech & accelerate your enterprise:

Follow on

Next to Read

Comments

Your email address will not be published. All fields are required.

0 Comments