Sampling
Sampling is the process of drawing a value from a probability distribution. In generative models, sampling turns a probability distribution over tokens into a concrete next-token choice.
What is Sampling?
Sampling is the process of drawing a value from a probability distribution. In generative models, sampling turns a probability distribution over tokens into a concrete next-token choice.
Sampling is the process of drawing a value from a probability distribution. In generative models, sampling turns a probability distribution over tokens into a concrete next-token choice.
Where is it used?
Llama, GPT, and Claude use top-k, top-p (nucleus), and temperature sampling to draw tokens from the softmax distribution during text generation.
How to build it
Use `torch.multinomial(probs, num_samples=1)` to sample from a softmax distribution, and implement top-k filtering by zeroing probabilities below the k-th largest logit.