Top-P
Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds p, discarding the long tail. It adapts the candidate pool size dynamically: fewer choices for confident steps, more for uncertain ones.
What is Top-P?
Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds p, discarding the long tail. It adapts the candidate pool size dynamically: fewer choices for confident steps, more for uncertain ones.
Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds p, discarding the long tail. It adapts the candidate pool size dynamically: fewer choices for confident steps, more for uncertain ones.
Where is it used?
Top-p is the default in OpenAI's API (`top_p=1.0` means off, `0.9` is common) and is used in Claude, LLaMA, and Mistral generation. It generally produces higher-quality text than fixed top-k because it adapts to distribution sharpness.
How to build it
Sort logits descending, compute cumulative softmax probabilities, mask tokens where cumulative prob exceeds p: `sorted_probs, idx = sort(probs, desc); cum = cumsum(sorted_probs); mask = cum > p; sorted_probs[mask] = 0; renorm`.