Best Cloud GPUs for Inference & Model Serving

What inference actually demands from a rented GPU

Inference is the serving phase of a model’s life: weights are already trained, and you pay for compute every time a user sends a prompt, an image, or a request. That changes the hardware math completely compared with training. Training is throughput-bound and runs for hours or days on dense clusters; inference is latency-sensitive, bursty, and runs indefinitely. When you rent a GPU to serve a model, you are optimizing for cost-per-token or cost-per-request at an acceptable response time, not for raw FLOPs over a long job.

The single most important constraint is usually memory capacity, not compute. A model has to fit in VRAM alongside its key-value (KV) cache, which grows with batch size and context length. For large language models this is the dominant pressure: a model served in 16-bit needs roughly two bytes per parameter just for weights, so a mid-size model can sit comfortably on a single mid-range card while a frontier-scale model may require multiple GPUs linked together. The comparison above lets you filter by VRAM precisely so you can match a card to the largest model you intend to serve plus headroom for concurrent requests.

The specs that move inference cost and latency

• VRAM capacity determines whether the model and its KV cache fit on one GPU or must be sharded. Spilling to a second GPU or to host memory adds latency and complexity.
• Memory bandwidth is the real bottleneck for token generation. Autoregressive decoding reads the entire weight set for every token produced, so high-bandwidth memory (HBM-class) generates tokens faster than GDDR-class memory at the same compute tier.
• Low-precision support matters enormously. Cards with FP8 or INT8 tensor paths let you quantize models to serve more requests per dollar with little quality loss. Quantization to 8-bit or 4-bit also shrinks the memory footprint, often letting a model fit on a cheaper card.
• Interconnect (NVLink versus PCIe) only matters once a model spans multiple GPUs. For single-GPU serving it is irrelevant; for tensor-parallel serving of very large models it directly affects token latency.

Batch (offline) versus real-time inference

Two very different serving patterns hide under the word “inference,” and they want different rentals.

Real-time inference serves live users: a chatbot, an API endpoint, an image generator behind a web app. Here tail latency rules, GPUs often sit partly idle waiting for traffic, and you cannot tolerate an instance being yanked away mid-request. This pattern favors on-demand, always-available capacity and a card with strong memory bandwidth so per-request latency stays low even at small batch sizes.

Batch or offline inference processes a large backlog: scoring a dataset, generating embeddings for a corpus, captioning a million images. There are no live users, so latency per item barely matters and you can pack large batches to saturate the GPU. This pattern is the ideal candidate for interruptible or spot capacity, because if an instance is reclaimed you simply resume the queue. When you read the list above, decide first which of these two patterns you are in, because it changes which billing model and availability tier is rational.

Why throughput and utilization beat peak FLOPs

A card that looks twice as powerful on paper rarely halves your inference bill. Decoding is memory-bound, so a GPU’s advertised peak compute is often underused during generation. What you actually pay for is effective tokens per second per dollar under your real batch sizes and context lengths. Modern serving stacks recover a lot of wasted capacity through continuous batching, paged KV caches, and quantization. The practical takeaway when renting: a mid-tier GPU running an optimized server can beat a flagship card running an unoptimized one, and a smaller, cheaper card that still fits your model is frequently the cost winner for steady traffic.

Provider features that matter specifically for serving

Inference runs continuously, so the surrounding platform features weigh more heavily than they do for a one-off training job. When comparing the options above, check these dimensions:

• Billing granularity: per-second or per-minute billing rewards bursty, scale-to-zero serving; coarse hourly billing punishes endpoints that idle between traffic spikes.
• Cold-start and provisioning speed: if you scale replicas up and down with demand, how fast a new GPU instance becomes ready directly affects user-facing latency and your ability to autoscale.
• Persistent storage and image caching: pulling large model weights on every start is slow and sometimes metered. Cached images or attached volumes that hold the weights cut cold starts dramatically.
• Egress fees: serving sends results back to users continuously. Per-gigabyte egress that is invisible for training can become a real line item for high-volume APIs.
• On-demand reliability versus spot pricing: real-time endpoints generally need guaranteed on-demand capacity; batch jobs can chase cheaper interruptible instances.
• Autoscaling and serverless options: scale-to-zero matters when traffic is spiky, so you are not paying for an idle GPU overnight.

How to read the comparison above for inference

Work in this order. First, identify the largest model you must serve and confirm a card has enough VRAM for the weights plus KV cache at your expected concurrency. Second, prefer high memory bandwidth and low-precision (FP8/INT8) support to maximize tokens per second per dollar. Third, match the billing and availability model to your pattern: on-demand with fine-grained billing for live endpoints, interruptible capacity for offline batch work. Use the live table for current rates, since per-hour pricing shifts with demand and scarcity and varies by provider; the durable rule is that the cheapest card that comfortably fits your model and traffic almost always wins, not the most powerful one available.

Frequently asked questions

How much GPU memory do I need to serve a model for inference?

Plan for the model weights plus the key-value cache. In 16-bit, weights need roughly two bytes per parameter, and quantizing to 8-bit or 4-bit cuts that substantially. Then add headroom for the KV cache, which grows with batch size and context length. Filter the list above by VRAM and pick a card that fits the model with room to spare so concurrency does not push you out of memory.

Is a cheaper GPU good enough for inference, or do I need a flagship card?

For many serving workloads a mid-range card is the better value. Token generation is bound by memory bandwidth rather than peak compute, so flagship FLOPs are often underused. If your model fits in a smaller card’s VRAM and an optimized serving stack keeps the GPU busy, you usually get better cost-per-token than renting the most expensive option.

Should I use spot or interruptible instances for inference?

It depends on the pattern. Offline batch inference tolerates interruptions well, since you can resume a queue, making cheaper spot capacity attractive. Real-time, user-facing endpoints generally need guaranteed on-demand capacity, because an instance being reclaimed mid-request causes failures and breaks latency guarantees.

What makes inference billing different from training billing?

Training is a finite, throughput-bound job, while inference runs continuously and often idles between traffic spikes. That makes per-second billing, fast provisioning, scale-to-zero, and predictable egress costs far more important for serving than they are for a one-off training run. Weigh those platform features in the comparison above alongside the raw hourly GPU rate.