Throughput
Inference throughput is the number of output tokens (or requests) served per second by the inference system, a function of batch size, model size, and hardware utilisation.
What is Throughput?
Inference throughput is the number of output tokens (or requests) served per second by the inference system, a function of batch size, model size, and hardware utilisation.
Inference throughput is the number of output tokens (or requests) served per second by the inference system, a function of batch size, model size, and hardware utilisation.
Where is it used?
vLLM continuous batching and TensorRT-LLM achieve high throughput on H100s; production serving targets maximise throughput while meeting per-request latency SLAs.
How to build it
Run `benchmark_throughput.py` from vLLM with `--model llama-3-8b --batch-size 32 --input-len 512 --output-len 128` and read the reported `tokens/s` metric.