Decoder
The decoder generates output tokens autoregressively while attending to its own past outputs (causal self-attention) and the encoder's representations (cross-attention). It uses a causal mask to prevent looking at future tokens.
What is Decoder?
The decoder generates output tokens autoregressively while attending to its own past outputs (causal self-attention) and the encoder's representations (cross-attention). It uses a causal mask to prevent looking at future tokens.
The decoder generates output tokens autoregressively while attending to its own past outputs (causal self-attention) and the encoder's representations (cross-attention). It uses a causal mask to prevent looking at future tokens.
Where is it used?
The decoder is the generative half of encoder-decoder models like T5, BART, and the original transformer for translation. During inference it generates one token at a time; during training it uses teacher forcing with a causal mask.
How to build it
Each decoder block has three sublayers: causal self-attention, cross-attention over encoder output, and FFN. Use `nn.TransformerDecoderLayer` or build manually with `attn_weights.masked_fill_(causal_mask, -inf)`.