Value
The value is a vector representing the content a token contributes to others, computed by projecting the hidden state with W_V. Attention output is a weighted sum of values using attention weights.
What is Value?
The value is a vector representing the content a token contributes to others, computed by projecting the hidden state with W_V. Attention output is a weighted sum of values using attention weights.
The value is a vector representing the content a token contributes to others, computed by projecting the hidden state with W_V. Attention output is a weighted sum of values using attention weights.
Where is it used?
The value vectors are what actually flow into the next layer; in LLM inference, caching V alongside K avoids recomputing values for past tokens, enabling efficient generation.
How to build it
Compute `V = x @ W_V`, then `output = attention_weights @ V`, and verify the output for a token is a convex combination (weighted average) of all value vectors.