swin transformer

The Swin Transformer is a redesign of the ViT that fixes two of its practical weaknesses for general vision: its quadratic cost on high-resolution images, and its single flat scale. The name is short for Shifted WINdow. Instead of letting every patch attend to every other patch across the whole image, Swin computes attention only within small local windows, and instead of keeping one fixed token resolution it builds a pyramid of progressively coarser, wider tokens — much like the stages of a convolutional network (think a ResNet's C2–C5 feature maps). The result is a backbone that drops into detection and segmentation pipelines where plain ViT struggles.

Two ideas carry the design. First, window-based multi-head self-attention partitions the feature map into non-overlapping windows of, say, 7x7 patches and runs full attention only inside each window; because window size is fixed, cost grows linearly with image area rather than quadratically — the crucial change that makes megapixel inputs tractable. Second, since pure windowing would trap information inside each window forever, alternate layers use shifted windows: the window grid is offset by half a window, so a patch that was on the edge of one window now shares a window with its former neighbours, letting information flow across window boundaries over successive layers.

The hierarchy is built by patch merging between stages: groups of 2x2 neighbouring tokens are concatenated and linearly projected, halving the spatial resolution and (typically) doubling the channel width, exactly like the downsampling in a CNN. Starting from a fine grid, Swin produces feature maps at multiple scales (commonly 1/4, 1/8, 1/16, 1/32 of the input), which is precisely the multi-scale input that feature-pyramid detectors and segmentation heads expect.

These choices reintroduce convolution-like inductive biases — locality and a coarse-to-fine hierarchy — into a transformer, which is why Swin trains well on ImageNet-1k alone without the giant pretraining ViT demands, and why it became a dominant general-purpose backbone for dense prediction (object detection, semantic and instance segmentation) when it appeared in 2021, with later versions scaling to very large models and high resolutions.