Token Position
Token position is the index of a token within a sequence. Because self-attention is permutation-invariant by default, transformers need explicit position information to know token order.
What is Token Position?
Token position is the index of a token within a sequence. Because self-attention is permutation-invariant by default, transformers need explicit position information to know token order.
Token position is the index of a token within a sequence. Because self-attention is permutation-invariant by default, transformers need explicit position information to know token order.
Where is it used?
Without positional information, GPT would treat 'dog bites man' and 'man bites dog' identically — every transformer injects position via encodings or embeddings before attention.
How to build it
Run a small transformer with and without positional encodings on a word-order-sensitive task and observe that without positions, the model cannot distinguish reversed sequences.