Positional Encoding
A positional encoding is a vector added to each token embedding that encodes its position in the sequence, allowing the transformer to use order information. Methods include sinusoidal, learned, and RoPE.
What is Positional Encoding?
A positional encoding is a vector added to each token embedding that encodes its position in the sequence, allowing the transformer to use order information. Methods include sinusoidal, learned, and RoPE.
A positional encoding is a vector added to each token embedding that encodes its position in the sequence, allowing the transformer to use order information. Methods include sinusoidal, learned, and RoPE.
Where is it used?
The original Transformer and GPT-2 use sinusoidal or learned encodings; Llama and Mistral use RoPE; the choice affects context length generalization and extrapolation.
How to build it
Implement `pos_emb = sinusoidal(seq_len, d_model)` and add it to token embeddings `x = x + pos_emb`, then verify that attention scores differ between same-tokens at different positions.