shifted window attention

Shifted window attention is the clever fix that lets a windowed transformer stay cheap while still letting information travel across the whole image. The problem it solves is a dilemma. Computing attention only inside fixed local windows makes cost linear and is exactly why Swin scales, but if the windows never move, the patches in window A can never directly influence the patches in window B — information is sealed inside each box forever. Shifting the window grid every other layer is the escape hatch.

The recipe alternates two layer types. A regular window layer partitions the feature map into a grid of non-overlapping windows and runs full self-attention inside each. The next layer shifts that whole grid diagonally by half a window (for a 7x7 window, by 3 patches down and 3 right) before partitioning again. After the shift, the new windows straddle the old boundaries: patches that were on the edges of two adjacent windows now sit together in one window and attend to each other. Stack these alternating regular and shifted layers and, over a few blocks, every patch can reach every other — global connectivity emerges from purely local computation.

Shifting naively would create partial windows at the image borders and increase the number of windows, hurting efficiency. Swin avoids this with an elegant trick: a cyclic shift rolls the feature map (the slice that falls off one edge wraps to the opposite edge), so the windows stay a uniform size and count. Because the cyclic shift puts non-adjacent regions into the same window, an attention mask is applied so that patches only attend within their true sub-region, then the map is rolled back. The net effect is cross-window communication at essentially no extra cost over plain windowed attention.

Conceptually, shifted windows are how Swin recovers the global receptive field that plain windowing throws away, restoring a key strength of full ViT attention while keeping linear cost. It is the single mechanism that makes the hierarchical, window-based design viable rather than a collection of isolated local transformers.