Attention Parameters
Attention parameters are the weight matrices that project tokens into queries (Q), keys (K), values (V), and the output projection (O). Multi-head attention has one set per head, often packed into a single matrix for efficiency.
What is Attention Parameters?
Attention parameters are the weight matrices that project tokens into queries (Q), keys (K), values (V), and the output projection (O). Multi-head attention has one set per head, often packed into a single matrix for efficiency.
Attention parameters are the weight matrices that project tokens into queries (Q), keys (K), values (V), and the output projection (O). Multi-head attention has one set per head, often packed into a single matrix for efficiency.
Where is it used?
Every GPT, LLaMA, and BERT model has these four projections per layer per head. GQA (grouped-query attention) in LLaMA-2 and Mistral shares K/V heads to reduce parameters and KV-cache size. MQA (multi-query) shares one K/V head.
How to build it
Standard: four `nn.Linear(d_model, d_model)` for Q, K, V, O. Multi-head: reshape output to `(B, T, n_heads, head_dim)`. For GQA: `n_kv_heads < n_heads` and broadcast K/V. Total attention params per layer: `4 * d_model * d_model` (standard).