Why context doesn't just stretch
Tell a RoPE model trained at 4k tokens to read 16k and quality falls off a cliff. The reason traces back to guide 1: the slow RoPE frequencies now rotate to angles the model never saw in training, so distant positions look like gibberish it has no representation for. Extending context length is therefore not a config flag — it is the problem of keeping positional signals in a range the model can interpret.
There are three distinct sub-problems, and confusing them wastes weeks. Extension is teaching a model to handle a larger context window than it was trained on. Streaming is generating indefinitely without the cache or quality blowing up. Distribution is fitting one enormous sequence across many GPUs at all. The next three sections take them one at a time.
YaRN: rescale the frequencies
YaRN (Yet another RoPE extensioN) extends context by selectively rescaling RoPE's rotation frequencies. The insight is that not all frequencies should be treated alike: the slow, long-range frequencies are the ones running out of range, so interpolate those toward angles the model already understands, while leaving the fast, local frequencies — which resolve nearby word order — almost untouched. This per-frequency, wavelength-aware treatment is what separates YaRN from naive linear interpolation.
RoPE's per-dimension rotation angle and wavelength — YaRN rescales the position by a factor s, applying it more to high-frequency dimensions than to low-frequency ones.
The payoff is efficiency: YaRN can push context far beyond training length with only a brief fine-tune on a small amount of long data — orders of magnitude cheaper than retraining. It is the standard recipe behind many '32k' and '128k' open variants, which are typically a base model plus a YaRN-style rescale and a short adaptation run.
Attention sinks: the secret to endless streaming
Suppose you want a chatbot that never stops, so you keep only a sliding window of recent tokens in the KV cache and evict the oldest. Surprisingly, quality collapses the moment the very first tokens get evicted — even though they are far away and seemingly irrelevant. This is the attention sink phenomenon: models learn to dump excess attention probability onto the first few positions as a kind of no-op, because softmax forces the weights to sum to one and something has to absorb the slack.
Interactive self-attention: clicking a word highlights the tokens it attends to.
The fix is delightfully cheap: always keep those first few 'sink' tokens in the cache alongside the sliding window. With the sink preserved, a model trained on a few thousand tokens can stream millions without degradation and without any fine-tuning. It pairs naturally with sliding-window attention — the window handles local content, the sinks stabilize the softmax.
Ring attention: one sequence, many devices
Even with a small cache, a million-token sequence's activations can exceed any single GPU's memory. Ring attention solves this by splitting the sequence into blocks, one per device, arranged in a logical ring. Each device computes attention for its own block of queries while key/value blocks are passed device-to-device around the ring. Because the communication of the next block overlaps with the computation on the current one, the network transfer is largely hidden behind useful work.
Attention is a sum over key blocks, so each device can accumulate its block's contribution as blocks rotate around the ring — an exact softmax with no single GPU holding the whole sequence.
The beautiful property is that the maximum context scales with the number of devices, not the memory of any one — add nodes to the ring and the addressable sequence grows linearly, in principle without bound. Combined with FlashAttention-style local kernels on each device, ring attention is how teams reach contexts of hundreds of thousands to millions of tokens for training and long-document inference.