Attention Scores
Attention scores are the raw dot products between each query and all keys, forming a (seq_len × seq_len) matrix. Higher scores indicate stronger relevance between token pairs.
What is Attention Scores?
Attention scores are the raw dot products between each query and all keys, forming a (seq_len × seq_len) matrix. Higher scores indicate stronger relevance between token pairs.
Attention scores are the raw dot products between each query and all keys, forming a (seq_len × seq_len) matrix. Higher scores indicate stronger relevance between token pairs.
Where is it used?
In LLMs, the attention score matrix is scaled and softmaxed to produce weights; causal masking zeros out future-token scores before softmax in autoregressive models.
How to build it
Compute `scores = Q @ K.transpose(-2,-1) / math.sqrt(d_k)`, apply a causal mask with `scores.masked_fill(mask == 0, float('-inf'))`, and print the matrix before and after softmax.