rotary position embedding (RoPE)

Instead of adding a position vector to a token, RoPE rotates the query and key vectors by an angle proportional to the token's position. Picture each pair of feature dimensions as a clock hand; a token at position m turns the hand by m steps. When you later take a dot product between a query at position m and a key at position n, the two rotations combine so the resulting score depends only on the relative offset n minus m, not on either absolute position.

Concretely, RoPE partitions the d-dimensional vector into d/2 two-dimensional blocks and applies a 2x2 rotation matrix to block k with angle m times theta_k, where theta_k = base^(-2k/d) and base is typically 10000. Because rotation matrices satisfy R_m transpose times R_n = R_(n-m), the inner product of R_m q with R_n k equals q transpose R_(n-m) k, encoding relative position exactly while leaving vector norms unchanged. It is applied to queries and keys only, never to values, and adds essentially no parameters.

RoPE has become the default in modern LLMs such as LLaMA, GPT-NeoX, and PaLM because it is parameter-free, composes cleanly with KV caching, and degrades gracefully. Its main limitation is length extrapolation: beyond the trained context the high-frequency components alias, which is precisely what NTK-aware scaling and YaRN repair by rescaling the theta_k frequencies.