YaRN context extension

YaRN is a recipe for stretching a model's usable context window far beyond its trained length by rescaling RoPE's rotation frequencies, with only a tiny amount of fine-tuning. The problem it solves: a model trained at, say, 4k tokens has never seen the rotation angles that arise at position 32k, so naive extrapolation produces garbage. YaRN reshapes those angles so longer positions fall back into the range the model already understands.

It builds on NTK-aware interpolation but treats RoPE's frequency bands differently by wavelength. High-frequency dimensions with short wavelength encode fine local order and are left almost untouched to preserve precise nearby relations; low-frequency dimensions with long wavelength are interpolated and scaled so distant positions stay in-distribution; a middle band is smoothly ramped between the two. YaRN adds one more trick, a slight temperature adjustment to the attention softmax that scales the logits by a constant, compensating for the entropy change of longer contexts and noticeably improving perplexity.

The payoff is large: extended LLaMA-style models reach 64k to 128k contexts after fine-tuning on only a fraction of a percent of the original tokens, far cheaper than training long from scratch. The limitation is that extension stretches an existing positional scheme rather than adding new long-range information, so very-long-context retrieval quality still depends on data and attention design.