Relative Position
Relative position encoding represents the distance between tokens rather than their absolute indices, so attention depends on how far apart two tokens are, not where they sit in the sequence.
What is Relative Position?
Relative position encoding represents the distance between tokens rather than their absolute indices, so attention depends on how far apart two tokens are, not where they sit in the sequence.
Relative position encoding represents the distance between tokens rather than their absolute indices, so attention depends on how far apart two tokens are, not where they sit in the sequence.
Where is it used?
T5 uses relative bias, and RoPE is a multiplicative form of relative encoding; relative methods generalize better to long sequences than absolute encodings.
How to build it
Implement a relative position bias `bias[i,j] = f(j-i)` added to attention scores, train a small transformer, and test on sequences longer than training length to see extrapolation.