Logits
Logits are the raw, unnormalized scores produced by the model's final linear layer (LM head) for each token in the vocabulary. They are real numbers that can be negative; applying softmax converts them to probabilities.
What is Logits?
Logits are the raw, unnormalized scores produced by the model's final linear layer (LM head) for each token in the vocabulary. They are real numbers that can be negative; applying softmax converts them to probabilities.
Logits are the raw, unnormalized scores produced by the model's final linear layer (LM head) for each token in the vocabulary. They are real numbers that can be negative; applying softmax converts them to probabilities.
Where is it used?
Every LLM outputs logits over the vocabulary before softmax. GPT-4, LLaMA, and Mistral all project the final hidden state through an LM head (`Linear(d_model, vocab_size)`) to produce logits. Logits are used for loss computation and sampling.
How to build it
Apply `self.lm_head = nn.Linear(d_model, vocab_size, bias=False)` to the final hidden state: `logits = self.lm_head(hidden)`. Often tie the LM head weights to the input embedding weights: `self.lm_head.weight = self.token_embedding.weight`.