Language Model
A language model (LM) is a probability distribution over sequences of tokens. It assigns a probability to a sequence and can predict the next token given previous ones, capturing statistical patterns and grammar of natural language.
What is Language Model?
A language model (LM) is a probability distribution over sequences of tokens. It assigns a probability to a sequence and can predict the next token given previous ones, capturing statistical patterns and grammar of natural language.
A language model (LM) is a probability distribution over sequences of tokens. It assigns a probability to a sequence and can predict the next token given previous ones, capturing statistical patterns and grammar of natural language.
Where is it used?
Language models power autocorrect, machine translation (T5), text generation (GPT-4), and code completion (Copilot). Modern LLMs like Claude, LLaMA, and Gemini are large neural language models fine-tuned with RLHF.
How to build it
Train a transformer on next-token prediction: tokenize text, feed sequences into the model, compute cross-entropy loss against shifted targets, and optimize. Use PyTorch `nn.CrossEntropyLoss` with logits of shape `(B, T, vocab)`.