Attention Weights
Attention weights are the softmax-normalized scores that determine how much each value contributes to a token's updated representation. They form a stochastic matrix with rows summing to 1.
What is Attention Weights?
Attention weights are the softmax-normalized scores that determine how much each value contributes to a token's updated representation. They form a stochastic matrix with rows summing to 1.
Attention weights are the softmax-normalized scores that determine how much each value contributes to a token's updated representation. They form a stochastic matrix with rows summing to 1.
Where is it used?
Visualizing attention weights in tools like BertViz shows which tokens GPT attends to; in LLMs, attention weights reveal copying, coreference, and syntactic patterns learned by the model.
How to build it
Compute `weights = F.softmax(scores, dim=-1)`, call `output = weights @ V`, and visualize the weight matrix with `matplotlib.pyplot.imshow` to see attention patterns.