Norms
A norm measures the length or magnitude of a vector. The L2 norm (Euclidean) is most common in ML; L1 norm sums absolute values. Norms are used for regularization and normalization.
What is Norms?
A norm measures the length or magnitude of a vector. The L2 norm (Euclidean) is most common in ML; L1 norm sums absolute values. Norms are used for regularization and normalization.
A norm measures the length or magnitude of a vector. The L2 norm (Euclidean) is most common in ML; L1 norm sums absolute values. Norms are used for regularization and normalization.
Where is it used?
LayerNorm in every transformer block normalizes activations using L2 norm; weight decay applies an L2 penalty to gradients during LLM training.
How to build it
Use `torch.norm(v, p=2)` for L2 and `torch.norm(v, p=1)` for L1, and implement LayerNorm manually by subtracting the mean and dividing by the L2 norm of a random vector.
Code
A practical example: