Decoder-Only Transformer
A decoder-only transformer drops the encoder and cross-attention, using only causal self-attention and FFN blocks. It takes a token sequence as input and predicts the next token, making it ideal for autoregressive language modeling.
What is Decoder-Only Transformer?
A decoder-only transformer drops the encoder and cross-attention, using only causal self-attention and FFN blocks. It takes a token sequence as input and predicts the next token, making it ideal for autoregressive language modeling.
A decoder-only transformer drops the encoder and cross-attention, using only causal self-attention and FFN blocks. It takes a token sequence as input and predicts the next token, making it ideal for autoregressive language modeling.
Where is it used?
This is the architecture behind nearly all modern LLMs: GPT-2, GPT-3, GPT-4, LLaMA, Mistral, Gemma, and Qwen. The entire model is a stack of decoder blocks processing the prompt and generating continuations.
How to build it
Stack N blocks of `[causal_self_attn, FFN]` with pre-norm residuals. In PyTorch: `x = x + attn(ln1(x), causal_mask=True)` then `x = x + ffn(ln2(x))`. Apply a final LayerNorm and `Linear(d_model, vocab_size)` to get logits.