Top-K
Top-k sampling restricts the next-token choice to the k highest-probability tokens, zeroing out the rest before renormalizing and sampling. This prevents rare, nonsensical tokens from being selected while keeping some diversity.
What is Top-K?
Top-k sampling restricts the next-token choice to the k highest-probability tokens, zeroing out the rest before renormalizing and sampling. This prevents rare, nonsensical tokens from being selected while keeping some diversity.
Top-k sampling restricts the next-token choice to the k highest-probability tokens, zeroing out the rest before renormalizing and sampling. This prevents rare, nonsensical tokens from being selected while keeping some diversity.
Where is it used?
Top-k is supported by HuggingFace `generate()`, OpenAI's API (top-k parameter), and LLaMA inference code. Common values are 40-50. GPT-2 popularized top-k sampling as a balance between greedy and full random sampling.
How to build it
Find the k-th largest logit, mask everything below: `vals, _ = logits.topk(k); logits[logits < vals[-1]] = -inf; probs = F.softmax(logits, dim=-1)`. Then `torch.multinomial(probs, 1)`.
Code
A practical example: