rotary position embeddings (RoPE)

Attention by itself is blind to order: shuffle the words and the raw math gives the same result, yet language obviously depends on position. RoPE fixes this by rotating each query and key vector by an angle proportional to its position in the sequence. Picture each pair of coordinates as a tiny clock hand; a token at position five spins five steps, a token at position fifty spins fifty. Because the dot product between two rotated vectors depends only on the difference of their angles, attention automatically sees relative distance.

Concretely, the model splits each head's vector into 2D blocks and applies a rotation matrix whose angle is position times a fixed per-block frequency. No extra parameters are learned and nothing is added to the embeddings; the rotation happens inside attention, on queries and keys only. This relative, multiplicative encoding extrapolates more gracefully than the old additive sinusoids, and its frequency knob is exactly what people tweak to stretch a model to longer contexts.