RoPE
Rotary Position Embedding (RoPE) encodes position by rotating the query and key vectors by an angle proportional to their position, so the dot product naturally depends on relative position. It is the dominant modern method.
What is RoPE?
Rotary Position Embedding (RoPE) encodes position by rotating the query and key vectors by an angle proportional to their position, so the dot product naturally depends on relative position. It is the dominant modern method.
Rotary Position Embedding (RoPE) encodes position by rotating the query and key vectors by an angle proportional to their position, so the dot product naturally depends on relative position. It is the dominant modern method.
Where is it used?
RoPE is used by Llama 2, Llama 3, Mistral, Qwen, and PaLM; its extrapolation properties enable context extension via techniques like NTK-aware scaling and YaRN.
How to build it
Implement RoPE by applying `rotate_half` with `torch.cos` and `torch.sin` of position-scaled angles to Q and K, then verify that attention scores shift correctly when sequences are offset.