Sinusoidal Encoding
Sinusoidal positional encoding uses sine and cosine functions of different frequencies across dimensions to produce a unique position vector for each index, enabling relative positions to be linear functions of each other.
What is Sinusoidal Encoding?
Sinusoidal positional encoding uses sine and cosine functions of different frequencies across dimensions to produce a unique position vector for each index, enabling relative positions to be linear functions of each other.
Sinusoidal positional encoding uses sine and cosine functions of different frequencies across dimensions to produce a unique position vector for each index, enabling relative positions to be linear functions of each other.
Where is it used?
The original 'Attention Is All You Need' Transformer uses sinusoidal encodings; they allow extrapolation to longer sequences than seen in training, which learned embeddings struggle with.
How to build it
Implement the formula `PE[pos, 2i] = sin(pos / 10000^(2i/d))` and `PE[pos, 2i+1] = cos(...)` in PyTorch, plot the heatmap with matplotlib, and add it to an embedding tensor.
Code
A practical example: