sliding-window attention

Full attention lets every token look at every earlier token, and that all-to-all comparison costs grow with the square of the sequence length, which becomes brutal at thousands of tokens. Sliding-window attention caps the view: each token may only attend to a fixed number of nearby predecessors, say the last few thousand, like reading with a flashlight that only lights up the recent page rather than the whole book.

This makes cost grow linearly with length and keeps the KV cache bounded, because old tokens fall out of the window. Long-range information is not lost outright: stacking many windowed layers lets influence travel further with each layer, the way a rumor spreads neighbor to neighbor, so a token can indirectly reach context far outside its own window. Many models interleave windowed layers with occasional full-attention layers to recover true long-distance links while keeping most layers cheap.