Number of Heads
The number of attention heads splits the model dimension into multiple parallel attention computations, allowing the model to attend to different relationships simultaneously (e.g., one head tracks syntax, another tracks coreference).
What is Number of Heads?
The number of attention heads splits the model dimension into multiple parallel attention computations, allowing the model to attend to different relationships simultaneously (e.g., one head tracks syntax, another tracks coreference).
The number of attention heads splits the model dimension into multiple parallel attention computations, allowing the model to attend to different relationships simultaneously (e.g., one head tracks syntax, another tracks coreference).
Where is it used?
GPT-2 small has 12 heads, LLaMA-2 7B has 32, GPT-3 175B has 96. Grouped-query attention (LLaMA-2 70B: 64 query heads, 8 KV heads) and multi-query attention (Falcon, PaLM) reduce KV-cache memory by sharing K/V heads.
How to build it
Set `n_heads` in config, ensure `d_model % n_heads == 0`. Reshape Q/K/V: `(B, T, n_heads, d_model // n_heads) -> (B, n_heads, T, head_dim)`. For GQA: `n_kv_heads` fewer K/V projections, then `repeat_interleave` to match query heads.