Head Dimension
The head dimension (d_k) is the size of each attention head's query, key, and value vectors. It is typically d_model / num_heads, so total compute stays constant as head count changes.
What is Head Dimension?
The head dimension (d_k) is the size of each attention head's query, key, and value vectors. It is typically d_model / num_heads, so total compute stays constant as head count changes.
The head dimension (d_k) is the size of each attention head's query, key, and value vectors. It is typically d_model / num_heads, so total compute stays constant as head count changes.
Where is it used?
Llama-2 7B uses d_model=4096 with 32 heads → d_k=128; GPT-2 small uses 768/12 = 64 per head; the choice affects per-head expressivity versus number of parallel relations.
How to build it
Set `d_k = d_model // num_heads`, reshape Q/K/V to `(batch, seq, num_heads, d_k)`, and verify that concatenating all heads recovers `d_model` width before the output projection.