Inference Latency
Inference latency is the time taken to produce a response, split into time-to-first-token (TTFT, prefill-bound) and inter-token latency (decode-bound), both critical for user experience.
What is Inference Latency?
Inference latency is the time taken to produce a response, split into time-to-first-token (TTFT, prefill-bound) and inter-token latency (decode-bound), both critical for user experience.
Inference latency is the time taken to produce a response, split into time-to-first-token (TTFT, prefill-bound) and inter-token latency (decode-bound), both critical for user experience.
Where is it used?
vLLM, TensorRT-LLM, and SGLang are optimised for latency; OpenAI and Anthropic report TTFT and tokens/sec metrics; production SLAs target <500ms TTFT and >50 tokens/sec.
How to build it
Use `time.time()` around `model.generate(...)`, measure TTFT by timing the first-streaming-chunk event and total time / tokens for inter-token latency; log these per request.