How well does NVIDIA L4 scale across multiple GPUs?

💡 Answer

121 FP16 TFLOPS and 300 GB/s of memory bandwidth put NVIDIA L4 squarely in the class of accelerators targeted at modern transformer workloads. FP32 caps at 30.3 TFLOPS, which still handles most non-AI scientific compute comfortably.

For training from scratch, token throughput roughly tracks FP16 TFLOPS. For production inference on foundation models, throughput tracks bandwidth. Real-world numbers will depend heavily on the framework stack (PyTorch, TensorRT-LLM, vLLM), and can vary 30-50% depending on how aggressively you quantise.

The cheapest NVIDIA L4 cloud access right now is on RunPod at $0.39/hr.