Causal Language Modeling
Causal language modeling (CLM) trains a model to predict each token given only the tokens before it, using a causal mask. This prevents information leakage from future tokens and is the standard objective for GPT-style models.
What is Causal Language Modeling?
Causal language modeling (CLM) trains a model to predict each token given only the tokens before it, using a causal mask. This prevents information leakage from future tokens and is the standard objective for GPT-style models.
Causal language modeling (CLM) trains a model to predict each token given only the tokens before it, using a causal mask. This prevents information leakage from future tokens and is the standard objective for GPT-style models.
Where is it used?
CLM is the training objective for all decoder-only LLMs: GPT-2, GPT-3, LLaMA, Mistral, and Gemma. It contrasts with masked language modeling (MLM) used by BERT, which predicts masked tokens with bidirectional context.
How to build it
Create a lower-triangular mask: `mask = torch.tril(torch.ones(T, T))` and apply it as `attn_scores.masked_fill(mask == 0, float('-inf'))`. Train with shifted labels and cross-entropy loss over the full sequence.