NVIDIA B200 memory-bound vs compute-bound workloads

Odpověď

NVIDIA B200 delivers 2,250 FP16 TFLOPS and 75 FP32 TFLOPS, backed by 8,000 GB/s of memory bandwidth and 192 GB of VRAM. In mixed-precision fine-tuning, those numbers typically convert to solid throughput on dense models up to several tens of billions of parameters.

For low-latency inference, real-world tokens-per-second on common large language models depends more on memory bandwidth than peak FLOPS — the 8,000 GB/s figure is the relevant ceiling for autoregressive decoding. On batched workloads like diffusion image generation, compute becomes the dominant factor again.

At $1.99 per hour on the budget-friendly cloud provider, performance-per-dollar is competitive for AI-heavy workloads.

Two tracked cloud providers currently offer NVIDIA B200: Vultr and RunPod. Vultr has the cheaper rate at $1.99/hr.