Q/K/V Matrices
QKV matrices are the three learned projection matrices W_Q, W_K, W_V that map the input hidden state into queries, keys, and values. They are often fused into a single matrix for efficiency.
What is Q/K/V Matrices?
QKV matrices are the three learned projection matrices W_Q, W_K, W_V that map the input hidden state into queries, keys, and values. They are often fused into a single matrix for efficiency.
QKV matrices are the three learned projection matrices W_Q, W_K, W_V that map the input hidden state into queries, keys, and values. They are often fused into a single matrix for efficiency.
Where is it used?
Every attention layer in GPT and Llama uses `nn.Linear(d_model, 3*d_model)` to compute Q, K, V in one matmul, then splits the output into three chunks.
How to build it
Use `qkv = nn.Linear(d_model, 3*d_model)(x)` then `Q, K, V = qkv.chunk(3, dim=-1)`, and verify each has shape (batch, seq, d_model).