Learned Positional Embeddings
Learned positional embeddings treat each position as a trainable embedding vector, like token embeddings, allowing the model to discover optimal position representations from data rather than a fixed formula.
What is Learned Positional Embeddings?
Learned positional embeddings treat each position as a trainable embedding vector, like token embeddings, allowing the model to discover optimal position representations from data rather than a fixed formula.
Learned positional embeddings treat each position as a trainable embedding vector, like token embeddings, allowing the model to discover optimal position representations from data rather than a fixed formula.
Where is it used?
GPT-2, BERT, and ViT use learned positional embeddings; they perform well within the trained context length but do not extrapolate to longer sequences without interpolation tricks.
How to build it
Use `nn.Embedding(max_seq_len, d_model)`, add `pos_emb(torch.arange(seq_len))` to token embeddings, and observe that positions beyond `max_seq_len` fail at inference.