Query
The query is a vector representing what a token is 'looking for' in the rest of the sequence. It is computed by projecting the token's hidden state with a learned weight matrix W_Q.
What is Query?
The query is a vector representing what a token is 'looking for' in the rest of the sequence. It is computed by projecting the token's hidden state with a learned weight matrix W_Q.
The query is a vector representing what a token is 'looking for' in the rest of the sequence. It is computed by projecting the token's hidden state with a learned weight matrix W_Q.
Where is it used?
In every attention layer of an LLM, each token's query is compared against all keys to decide how much to attend to each position; W_Q is a learned parameter.
How to build it
Compute `Q = x @ W_Q` where `W_Q = nn.Linear(d_model, d_k).weight.T`, and inspect Q's shape — one query vector per token, of dimension d_k.