Layer Normalization
Layer normalization stabilizes training by normalizing activations across the feature dimension of a single token: subtract the mean and divide by the standard deviation of that token's features, then apply a learnable scale and shift.
What is Layer Normalization?
Layer normalization stabilizes training by normalizing activations across the feature dimension of a single token: subtract the mean and divide by the standard deviation of that token's features, then apply a learnable scale and shift.
Layer normalization stabilizes training by normalizing activations across the feature dimension of a single token: subtract the mean and divide by the standard deviation of that token's features, then apply a learnable scale and shift.
Where is it used?
Used in every transformer block in GPT, BERT, and T5. LLaMA and Gemma replaced standard LayerNorm with RMSNorm, which drops the mean-centering step for efficiency. LayerNorm is applied before (pre-norm) or after (post-norm) the attention/FFN sublayers.
How to build it
Use `nn.LayerNorm(d_model)` in PyTorch, or implement RMSNorm: `x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * self.weight` for the modern variant used by LLaMA.