Causal Attention
Causal (masked) attention restricts each token to attending only to itself and previous tokens, preventing information from flowing from the future. This is what makes autoregressive language models possible.
What is Causal Attention?
Causal (masked) attention restricts each token to attending only to itself and previous tokens, preventing information from flowing from the future. This is what makes autoregressive language models possible.
Causal (masked) attention restricts each token to attending only to itself and previous tokens, preventing information from flowing from the future. This is what makes autoregressive language models possible.
Where is it used?
GPT, Llama, and Mistral use causal attention in every layer during training and inference; BERT uses bidirectional (non-causal) attention because it is for masked prediction, not generation.
How to build it
Create a lower-triangular mask with `torch.tril(torch.ones(seq_len, seq_len))`, apply it via `scores.masked_fill(mask == 0, float('-inf'))` before softmax, and verify future positions are zero.
Code
A practical example: