NVIDIA L40S real-world generative AI performance

Ответ

366 FP16 TFLOPS and 864 GB/s of memory bandwidth put NVIDIA L40S squarely in the class of accelerators targeted at modern transformer workloads. FP32 caps at 91.6 TFLOPS, which still handles most non-AI scientific compute comfortably.

For training from scratch, token throughput roughly tracks FP16 TFLOPS. For production inference on foundation models, throughput tracks bandwidth. Real-world numbers will depend heavily on the framework stack (PyTorch, TensorRT-LLM, vLLM), and can vary 30-50% depending on how aggressively you quantise.

The cheapest NVIDIA L40S right now is on Vast.ai at $0.55/hr; Novita AI and Latitude.sh also list it — pick the region and billing that fits your workload.