Probability Distribution
A probability distribution over the vocabulary assigns a non-negative value to each token that sums to 1. The softmax function converts logits into this distribution, and the shape is controlled by temperature and top-k/top-p filters.
What is Probability Distribution?
A probability distribution over the vocabulary assigns a non-negative value to each token that sums to 1. The softmax function converts logits into this distribution, and the shape is controlled by temperature and top-k/top-p filters.
A probability distribution over the vocabulary assigns a non-negative value to each token that sums to 1. The softmax function converts logits into this distribution, and the shape is controlled by temperature and top-k/top-p filters.
Where is it used?
The softmax distribution is what LLMs sample from during generation. GPT-4, LLaMA, and DALL-E's text encoder all produce token probability distributions. Cross-entropy loss measures the gap between the predicted distribution and the true (one-hot) target.
How to build it
Apply temperature-scaled softmax: `probs = F.softmax(logits / temperature, dim=-1)`. For top-k: zero out logits below the k-th largest before softmax. For top-p: sort cumulative probabilities and mask the tail.