Transformer Block
A transformer block is the repeated unit of a transformer, containing a multi-head self-attention sublayer and a feed-forward (MLP) sublayer, each with residual connections and layer normalization.
What is Transformer Block?
A transformer block is the repeated unit of a transformer, containing a multi-head self-attention sublayer and a feed-forward (MLP) sublayer, each with residual connections and layer normalization.
A transformer block is the repeated unit of a transformer, containing a multi-head self-attention sublayer and a feed-forward (MLP) sublayer, each with residual connections and layer normalization.
Where is it used?
GPT-2 small has 12 blocks; Llama-3 70B has 80; each block refines the token representations that flow through the residual stream, progressively building higher-level features.
How to build it
Implement `x = x + attn(layernorm(x))` then `x = x + mlp(layernorm(x))` (pre-norm), and stack N copies; verify gradient flow with `torch.autograd.grad` to confirm residuals help depth.
Code
A practical example: