Throughput Evaluation
Throughput evaluation measures the number of output tokens per second a serving system can sustain under concurrent load, critical for capacity planning and cost estimation.
What is Throughput Evaluation?
Throughput evaluation measures the number of output tokens per second a serving system can sustain under concurrent load, critical for capacity planning and cost estimation.
Throughput evaluation measures the number of output tokens per second a serving system can sustain under concurrent load, critical for capacity planning and cost estimation.
Where is it used?
vLLM `benchmark_throughput.py`, Triton Inference Server metrics, and `locust` load tests measure throughput; production teams balance throughput against latency SLAs.
How to build it
Run `python -m vllm.entrypoints.openai.api_server --model ...` then `benchmark_serving.py --backend vllm --num-prompts 1000 --request-rate 20` to measure sustained tokens/s.