MDSection 8 of 38 · 6 subtopics
Positional Information
Transformers need position info. Sinusoidal encodings, learned embeddings, and RoPE.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
01
Token Position
Token sequence with position labels.
token-position
02
Positional Encoding
Encoding heatmap.
positional-encoding
03
Sinusoidal Encoding
Sine/cosine graph + heatmap.
sinusoidal-encoding
04
Learned Positional Embeddings
Position embedding matrix.
learned-positional-embeddings
05
Relative Position
Token-distance matrix.
relative-position
06
RoPE
Rotating-vector visualization.
rope