Random Sampling
Random sampling picks the next token by drawing from the model's probability distribution rather than always taking the argmax. This introduces stochasticity, producing diverse and creative output across multiple runs.
What is Random Sampling?
Random sampling picks the next token by drawing from the model's probability distribution rather than always taking the argmax. This introduces stochasticity, producing diverse and creative output across multiple runs.
Random sampling picks the next token by drawing from the model's probability distribution rather than always taking the argmax. This introduces stochasticity, producing diverse and creative output across multiple runs.
Where is it used?
Sampling is the default generation mode for creative writing and chat in models like GPT-4, LLaMA, and Mistral. Combined with temperature, top-k, and top-p, it gives the controllable variety that production LLM APIs expose to users.
How to build it
Convert logits to probabilities and sample: `probs = F.softmax(logits, dim=-1); next = torch.multinomial(probs, num_samples=1)`. Set a manual seed via `torch.manual_seed(42)` for reproducibility.