MDSection 9 of 38 · 12 subtopics
Attention
The mechanism that powers Transformers. Scaled dot-product, softmax, masking, and causal attention.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
01
Attention
Token-to-token connection graph.
attention
02
Self-Attention
Attention heatmap.
self-attention
03
Query
Q-vector inspector.
query
04
Key
Key-vector inspector.
key
05
Value
Value-flow visualization.
value
06
Q/K/V Matrices
Three synchronized matrix viewers.
qkv-matrices
07
Attention Scores
Score matrix heatmap.
attention-scores
08
Scaled Dot-Product Attention
Formula pipeline.
scaled-dot-product-attention
09
Softmax
Logit-to-probability animation.
softmax
10
Attention Weights
Token heatmap.
attention-weights
11
Causal Attention
Triangular attention matrix.
causal-attention
12
Attention Mask
Interactive mask grid.
attention-mask