Prefill
Prefill is the first inference phase where the model processes the entire prompt in parallel, computing KV caches for all prompt tokens; it is compute-bound and dominates time-to-first-token.
What is Prefill?
Prefill is the first inference phase where the model processes the entire prompt in parallel, computing KV caches for all prompt tokens; it is compute-bound and dominates time-to-first-token.
Prefill is the first inference phase where the model processes the entire prompt in parallel, computing KV caches for all prompt tokens; it is compute-bound and dominates time-to-first-token.
Where is it used?
vLLM, TensorRT-LLM, and HF `generate()` all run prefill before decode; long-context models like Gemini 1.5 spend most of TTFT on prefill of the large prompt.
How to build it
Time the first token by wrapping `model(input_ids)` (prefill only) with `torch.cuda.Event` timers; compare TTFT for prompt lengths 128, 512, and 4096 to see prefill scaling.
Code
A practical example: