How well does NVIDIA RTX PRO 6000 scale across multiple GPUs?

回答

252 FP16 TFLOPS and 1,792 GB/s of memory bandwidth put NVIDIA RTX PRO 6000 squarely in the class of accelerators targeted at modern transformer workloads. FP32 caps at 125 TFLOPS, which still handles most non-AI scientific compute comfortably.

For training from scratch, token throughput roughly tracks FP16 TFLOPS. For production inference on foundation models, throughput tracks bandwidth. Real-world numbers will depend heavily on the framework stack (PyTorch, TensorRT-LLM, vLLM), and can vary 30-50% depending on how aggressively you quantise.

Two tracked cloud providers currently offer NVIDIA RTX PRO 6000: Latitude.sh and RunPod. Latitude.sh has the cheaper rate at $1.71/hr.