Latency Evaluation
Latency evaluation measures the time-to-first-token and inter-token latency of an LLM under controlled prompts, critical for real-time chat and interactive applications.
What is Latency Evaluation?
Latency evaluation measures the time-to-first-token and inter-token latency of an LLM under controlled prompts, critical for real-time chat and interactive applications.
Latency evaluation measures the time-to-first-token and inter-token latency of an LLM under controlled prompts, critical for real-time chat and interactive applications.
Where is it used?
vLLM benchmarks, TensorRT-LLM `benchmark.py`, and `optimum-benchmark` report latency; production serving teams set latency SLAs and regress-test them on every model update.
How to build it
Use `optimum-benchmark --backend vllm --task text-generation --model meta-llama/Meta-Llama-3-8B` to get TTFT and inter-token latency, or time `model.generate` with `torch.cuda.Event`.