Next-Token Prediction
Next-token prediction is the task of estimating the probability distribution over the vocabulary for the token that follows a given context. It is the core training objective and inference mechanism of autoregressive LLMs.
What is Next-Token Prediction?
Next-token prediction is the task of estimating the probability distribution over the vocabulary for the token that follows a given context. It is the core training objective and inference mechanism of autoregressive LLMs.
Next-token prediction is the task of estimating the probability distribution over the vocabulary for the token that follows a given context. It is the core training objective and inference mechanism of autoregressive LLMs.
Where is it used?
This is the pretraining objective for GPT-2, GPT-3, LLaMA, and all decoder-only LLMs. At inference, next-token prediction drives greedy decoding, sampling, and beam search. Code completion in Copilot uses the same mechanism with a code-trained LM.
How to build it
Shift labels by one: `logits = model(input_ids)[:, :-1]` and `labels = input_ids[:, 1:]`, then compute `F.cross_entropy(logits.reshape(-1, V), labels.reshape(-1))`. At inference, take `logits[:, -1]` and sample the next token.
Code
A practical example:
Further reading
Want to go deeper? These resources cover Next-Token Prediction in more detail: