Attention Head
An attention head is one independent attention computation with its own Q, K, V projections, allowing the model to focus on different relationships (syntax, coreference, semantics) in parallel.
What is Attention Head?
An attention head is one independent attention computation with its own Q, K, V projections, allowing the model to focus on different relationships (syntax, coreference, semantics) in parallel.
An attention head is one independent attention computation with its own Q, K, V projections, allowing the model to focus on different relationships (syntax, coreference, semantics) in parallel.
Where is it used?
GPT-2 small has 12 heads per layer; Llama-3 70B has 64 heads per layer; each head can specialize in different linguistic phenomena, which interpretability research studies.
How to build it
Create separate `W_Q, W_K, W_V` matrices for one head, compute attention, and stack multiple heads to see how different heads learn different patterns on the same input.