Cross Entropy
Cross-entropy loss measures the difference between a predicted probability distribution and the true distribution, heavily penalizing confident wrong predictions. It is the standard loss for classification.
What is Cross Entropy?
Cross-entropy loss measures the difference between a predicted probability distribution and the true distribution, heavily penalizing confident wrong predictions. It is the standard loss for classification.
Cross-entropy loss measures the difference between a predicted probability distribution and the true distribution, heavily penalizing confident wrong predictions. It is the standard loss for classification.
Where is it used?
Every autoregressive LLM — GPT, Llama, Claude — trains with cross-entropy loss over the vocabulary at each token position to learn next-token prediction.
How to build it
Use `F.cross_entropy(logits, targets)` where logits are (N, vocab_size) and targets are (N,) integer labels; verify it equals `-log_softmax(logits)[targets]`.