Loss Calculation
Loss calculation measures how wrong the model's predictions are versus the true next token. For language modeling, cross-entropy loss compares the model's probability distribution over the vocabulary to the actual next token (represented as a one-hot vector).
What is Loss Calculation?
Loss calculation measures how wrong the model's predictions are versus the true next token. For language modeling, cross-entropy loss compares the model's probability distribution over the vocabulary to the actual next token (represented as a one-hot vector).
Loss calculation measures how wrong the model's predictions are versus the true next token. For language modeling, cross-entropy loss compares the model's probability distribution over the vocabulary to the actual next token (represented as a one-hot vector).
Where is it used?
Cross-entropy loss is the standard for all autoregressive LLM pretraining (GPT, LLaMA, Mistral). It is also used during supervised fine-tuning. The average loss over a batch, exponentiated, gives the perplexity metric used to compare models.
How to build it
`loss = F.cross_entropy(logits.reshape(-1, vocab_size), targets.reshape(-1))` where `targets = input_ids[:, 1:].contiguous()` and `logits = logits[:, :-1]`. Perplexity: `ppl = torch.exp(loss)`. For label smoothing: `F.cross_entropy(..., label_smoothing=0.1)`.