Decode
Decode is the autoregressive phase where the model generates one token at a time using the KV cache, feeding each new token back as input; it is memory-bandwidth-bound and dominates total generation time.
What is Decode?
Decode is the autoregressive phase where the model generates one token at a time using the KV cache, feeding each new token back as input; it is memory-bandwidth-bound and dominates total generation time.
Decode is the autoregressive phase where the model generates one token at a time using the KV cache, feeding each new token back as input; it is memory-bandwidth-bound and dominates total generation time.
Where is it used?
All autoregressive LLMs (GPT, Llama, Mistral) decode token-by-token; vLLM continuous batching and speculative decoding target decode-phase bottlenecks.
How to build it
After prefill, loop `next_token = model(past_kv, last_token).logits.argmax(-1)`, append to sequence, and measure per-token latency with `time.time()` to observe decode speed.