Multiple Attention Heads
Multi-head attention runs several attention heads in parallel on the same input, each with its own projections, then concatenates and projects their outputs. This lets the model attend to multiple relations simultaneously.
What is Multiple Attention Heads?
Multi-head attention runs several attention heads in parallel on the same input, each with its own projections, then concatenates and projects their outputs. This lets the model attend to multiple relations simultaneously.
Multi-head attention runs several attention heads in parallel on the same input, each with its own projections, then concatenates and projects their outputs. This lets the model attend to multiple relations simultaneously.
Where is it used?
Every transformer block in GPT, Llama, and BERT uses multi-head attention; the heads collectively capture diverse subspaces of the representation that a single head could not.
How to build it
Implement a loop over `num_heads` computing per-head attention, `torch.cat(outputs, dim=-1)`, then apply a final `W_O` projection, and compare with `nn.MultiheadAttention`.
Code
A practical example: