Where am I? Position in modern transformers

Attention is order-blind by construction

A pure self-attention block is a set operation: permute the input tokens and the outputs permute the same way. That symmetry is elegant but useless for language, where the cat sat and sat the cat mean different things. Every transformer therefore injects position somehow. Vol I's sinusoidal positional encoding added a fixed table to the input embeddings; it works, but it ties the model to absolute indices it saw during training and tends to fall apart past that length.

Two observations reshaped the field. First, what attention actually needs is relative position — how far apart two tokens are — not their absolute index. Second, position is best applied where the comparison happens: inside the query–key dot product of scaled dot-product attention, not bolted onto the input. Both RoPE and ALiBi follow from taking these seriously.

RoPE: rotate, don't add

Rotary position embedding (RoPE) takes each query and key vector, splits it into 2-D pairs, and rotates each pair by an angle proportional to the token's position. The magic is in the dot product: when you compare a query at position m with a key at position n, the rotations combine so that the score depends only on the difference m − n. Absolute rotations in, relative position out — for free, with no extra parameters.

Different coordinate pairs rotate at different frequencies, from very fast (high-frequency pairs flip within a few tokens) to very slow (low-frequency pairs barely move across thousands of tokens). The fast dimensions resolve nearby word order; the slow dimensions carry coarse long-range position. This frequency ladder is exactly the lever YaRN will pull later to extend context — so hold the picture.

\langle R_m\mathbf{q},\,R_n\mathbf{k}\rangle=\mathbf{q}^{\top}R_{n-m}\,\mathbf{k},\qquad R_m=\begin{pmatrix}\cos m\theta & -\sin m\theta\\ \sin m\theta & \cos m\theta\end{pmatrix}

RoPE's core identity: rotating query and key makes their dot product depend only on the relative offset n−m.

# RoPE applied to one query vector q at position m
for pair i in pairs(q):
    theta = m * base ** (-2*i / d)   # angle grows with position
    q[i] = rotate_2d(q[i], theta)     # rotate this (x, y) pair
# the same rule applies to keys; the q.k score then depends on (m - n)

RoPE rotates coordinate pairs; the dot product turns absolute angles into relative position.

ALiBi: just penalize distance

ALiBi (Attention with Linear Biases) is almost aggressively simple: add nothing to embeddings and rotate nothing. Instead, before the softmax, subtract a penalty proportional to the distance between query and key — the farther back a token is, the lower its score. Each attention head gets its own penalty slope, so some heads look sharply local while others keep a long, gentle reach.

a_{ij}=\mathrm{softmax}\!\left(\frac{\mathbf{q}_i^{\top}\mathbf{k}_j}{\sqrt{d}}-m\,(i-j)\right),\qquad i\ge j

ALiBi in one line: before softmax, subtract a head-specific slope m times the token distance from each score.

Because the penalty is a fixed linear function of distance, ALiBi extrapolates gracefully: a model trained at 2k tokens still behaves sensibly at 4k, since the bias formula is defined for any distance. RoPE, by contrast, was not originally built to extrapolate — push it well past its training length untouched and accuracy collapses, which is the problem the next-to-last guide exists to solve.

Which one, and why it matters downstream

Most frontier open models (the Llama and Qwen families) chose RoPE for its strong in-distribution quality and its clean, tunable frequency structure. A few long-context-first designs leaned on ALiBi for its built-in extrapolation. The decision ripples: RoPE's frequencies interact with context length extension tricks, and the choice constrains which long-context method you can bolt on later.