Generation Loop
The generation loop is the iterative process that produces text autoregressively: forward pass, sample next token, append, check stop conditions, repeat. It is the runtime engine of every autoregressive LLM.
What is Generation Loop?
The generation loop is the iterative process that produces text autoregressively: forward pass, sample next token, append, check stop conditions, repeat. It is the runtime engine of every autoregressive LLM.
The generation loop is the iterative process that produces text autoregressively: forward pass, sample next token, append, check stop conditions, repeat. It is the runtime engine of every autoregressive LLM.
Where is it used?
This loop runs in every LLM inference engine: HuggingFace `generate()`, vLLM, llama.cpp, and OpenAI's serving infrastructure. Optimizations like KV-caching, speculative decoding, and continuous batching all operate inside this loop.
How to build it
Write: `ids = prompt_ids; for _ in range(max_new): logits = model(ids); next = sample(logits[:, -1]); ids = cat([ids, next]); if stop(next): break`. Enable KV-cache by passing `past_key_values` and only feeding the new token each step.