QKV Projection
The QKV projection maps the input hidden state into per-head queries, keys, and values via learned weight matrices, typically computed as a single fused matmul then split and reshaped into heads.
What is QKV Projection?
The QKV projection maps the input hidden state into per-head queries, keys, and values via learned weight matrices, typically computed as a single fused matmul then split and reshaped into heads.
The QKV projection maps the input hidden state into per-head queries, keys, and values via learned weight matrices, typically computed as a single fused matmul then split and reshaped into heads.
Where is it used?
In every multi-head attention layer, the fused `nn.Linear(d_model, 3*d_model)` projection is split into Q/K/V, then reshaped to `(batch, seq, num_heads, d_k)` and transposed for batched matmul.
How to build it
Compute `qkv = linear(x)`, split with `.chunk(3, dim=-1)`, reshape with `.view(batch, seq, num_heads, d_k)`, and `.transpose(1, 2)` to put heads first for batched `torch.matmul`.