context window extension

A model pretrained to handle, say, four thousand tokens does not gracefully read forty thousand: feed it positions it never saw during training and its rotary position encodings land out of distribution, attention degrades, and quality falls off a cliff. Context window extension is the set of techniques that push a pretrained model's usable context far beyond its original limit while paying only for a little extra tuning rather than a full retrain from scratch.

Most modern methods work on the position encoding rather than the weights. Position interpolation rescales the position indices so that long sequences fall back inside the range the model was trained on, trading angular resolution for reach. NTK-aware scaling and YaRN refine this by adjusting the rotary base frequencies non-uniformly — stretching low frequencies that carry long-range information while sparing high frequencies that encode local order — and a brief continued-training phase on long sequences lets the model settle into the new geometry. The result is coherent attention over contexts many times the pretraining length.

Extension makes long-document reasoning and large retrieval contexts possible, but it is not free. Attention and the KV cache still cost roughly linearly to quadratically more as length grows, so memory and latency climb; and a model that can technically attend over a long context may still use the middle of it poorly, the lost-in-the-middle effect, so a larger window is necessary but not sufficient for genuinely long reasoning.