Scaling to Real Images: Swin Transformer and Hierarchical Attention

The quadratic wall: why attention can't scale

Back in guide 2 we saw the engine of every Transformer: self-attention lets each patch look at every other patch and decide who to listen to, using scaled dot-product attention. That all-to-all conversation is exactly what gives ViT its power — but it is also a trap. If every token must compare itself with every other token, then doubling the number of tokens roughly quadruples the work. This section makes that statement precise, because understanding this single bottleneck explains every design choice the Swin Transformer makes.

Here is the counting argument in plain words. If there are N tokens, then self-attention forms a pairwise comparison between token i and token j for every choice of i and j. The number of such ordered pairs is N times N, i.e. N². Each comparison is not free either — every token is a vector of length D (the model dimension), so a single dot product costs about D multiply-adds. Multiply the number of pairs by the cost per pair and you get the total below.

Global attention is quadratic in the token count N; windowed attention (next sections) is linear.

Read the left side first. The cost of global attention is O(N²·D): N is the total number of tokens (one per image patch), D is the dimension of each token vector, and the N² comes from the all-pairs comparison we just counted. The big-O simply means 'grows in proportion to' — we ignore constant factors and focus on how cost scales as the image gets bigger. The right side, O(N·M²·D), is what Swin will achieve: M is the window size measured in tokens (say 7, for a 7×7 window). Notice the crucial difference — once we fix M to a small constant, M² is just a constant too, so the only thing that still grows with image size is the lone factor of N. That turns a quadratic curve into a straight line.

Let's put real numbers on it. A typical detection backbone might process an image as a 56×56 grid of tokens. That is N = 56 × 56 = 3,136 tokens. The number of pairwise comparisons is therefore N² = 3,136² ≈ 9.8 million — call it ten million dot products, per attention layer, per head. Now imagine going to a 112×112 grid (a higher-resolution image): N jumps to 12,544, and N² explodes to about 157 million — sixteen times more work for only twice the resolution per side. This is the 'quadratic wall': the cost outruns the hardware long before you reach the fine spatial detail that detection and segmentation demand.

Windowed attention: think locally

The Swin Transformer's first big idea is almost embarrassingly simple: don't let every token talk to every other token. Instead, chop the grid of tokens into non-overlapping square windows — Swin uses 7×7 patches per window — and run ordinary self-attention strictly inside each window. A token in the top-left window attends to the other 48 tokens in its own window and to nobody else. The expensive all-pairs comparison still happens, but only over the 49 tokens of a single window, never over all 3,136 tokens at once.

Here is the analogy that makes it stick. Picture a packed stadium where everyone wants to share news. If every single person had to speak with every other person, the chatter would be impossible — that is global attention. Windowed attention says instead: only talk to the people in your own row. Each row buzzes with its own quick conversation, all rows talk at the same time, and the total amount of talking grows only with the number of rows — that is, linearly with the size of the crowd. The price of admission for cheapness is that, for now, your row has no idea what the row behind it is saying.

Notice what just happened philosophically. In guide 4 we learned that ViT's data hunger comes from throwing away the locality bias that CNNs bake in — a CNN assumes nearby pixels matter most, while a pure ViT must learn that from scratch with mountains of data. Windowed attention deliberately reintroduces that locality inductive bias: by construction, a token can only attend to its near neighbours. Swin is, in this sense, a partial homecoming to the CNN's wisdom, achieved without giving up the attention mechanism. That locality is not just a speed hack; it is also a useful prior that helps Swin learn from less data.

# Why windowed attention is LINEAR in the number of tokens.
# x is a feature map of shape (H, W, D); M is the window side (e.g. 7).

def window_partition(x, M):
    # Cut the H x W grid into non-overlapping M x M windows.
    windows = []
    for i in range(0, H, M):
        for j in range(0, W, M):
            windows.append(x[i:i+M, j:j+M])      # each window: (M, M, D)
    return windows                                # there are (H/M)*(W/M) of them

def windowed_attention(x, M):
    out = []
    for w in window_partition(x, M):
        tokens = w.reshape(M*M, D)                # M^2 tokens in this window
        out.append(self_attention(tokens))       # cost ~ O((M^2)^2 * D) per window
    return reassemble(out)

# Total cost = (#windows) * (cost per window)
#            = (H*W / M^2) * O(M^4 * D)
#            = O(H*W * M^2 * D)
#            = O(N * M^2 * D)        <-- linear in N because M is fixed.

Shifted windows: let neighbors connect

If windows never see each other, we lose the global view. Swin's masterstroke — and the single idea the architecture is named after — is shifted window attention. The trick: alternate two kinds of layers. The first layer uses the regular window grid we just described. The next layer shifts the entire window grid diagonally by half a window (for a 7×7 window, that's a shift of about 3 patches down and 3 patches right). Because the grid moved, the new windows now straddle the boundaries of the old ones — so tokens that lived in different windows last layer are suddenly grouped together and finally get to talk.

Back to the stadium. Imagine that after each round of row-only chatter, we ask everyone to shuffle their seats so the dividing lines between rows fall in new places. People who were at opposite ends of two different rows now sit together and swap what they each heard. Do this every other round and, layer by layer, a rumour started in one corner of the stadium gradually ripples across the entire crowd — even though no single conversation ever involved more than one small group. Shifted windows give Swin a growing 'effective receptive field': after enough layers, any token can have indirectly influenced any other token, restoring global context.

There is one practical wrinkle. After shifting, the windows at the edges of the image are incomplete — partial windows of odd sizes poke out at the borders. Running attention on a patchwork of irregular window sizes would be slow and messy. Swin solves this with a clever cyclic shift: it wraps the image around like a torus, sliding the tokens that fell off the top and left edges back in at the bottom and right. Now every window is a full, regular M×M block again, so the fast batched computation still works. The catch is that wrapping puts non-adjacent tokens (say, the top edge and the bottom edge of the image) into the same window — and those should not actually attend to each other.

The fix is an attention mask. Inside each wrapped-around window, Swin marks which token-pairs are genuine neighbours and which are only neighbours by accident of the wrap. For the fake pairs it adds a large negative number to their attention score before the softmax, so after the softmax their weight is essentially zero — they are computed but contribute nothing. (Recall from guide 2 that softmax turns scores into weights, and a hugely negative score becomes a near-zero weight.) After attention, the cyclic shift is reversed to put every token back where it belongs. The net effect is beautiful: full, regular, fast windows everywhere, yet each token only truly attends to its real spatial neighbours.

Patch merging: building a feature pyramid

So far every Swin layer works on the same grid of tokens. But real vision needs more than one scale: a fine grid is great for spotting tiny details, while a coarse grid is great for understanding whole objects and scene layout. Swin builds this multi-scale view with patch merging. Every so often, between stages, it takes each 2×2 block of neighbouring tokens and fuses them into a single token. The grid shrinks, each surviving token now summarises a larger region, and — crucially — the model keeps enough representational capacity by widening each token vector at the same time.

One patch-merging step halves each spatial side and doubles the channel depth.

Let's unpack the arrow carefully. On the left, the feature map has height H, width W, and C channels per token (the channel count is the length of each token's vector). Patch merging looks at every non-overlapping 2×2 neighbourhood — that's 4 tokens, each of length C — and concatenates them into one long vector of length 4C. Stacking four vectors of length C end to end gives 4C, naturally. Then a single linear layer projects that 4C-vector down to 2C. The result, on the right, is a grid with half the height (H/2), half the width (W/2), and double the channels (2C). So the number of tokens drops by 4× (since H/2 × W/2 = HW/4) while each token gets twice as rich.

def patch_merging(x):
    # x: a feature map of shape (H, W, C)
    # Pick out the four positions of every 2x2 block.
    x0 = x[0::2, 0::2, :]   # top-left      -> (H/2, W/2, C)
    x1 = x[1::2, 0::2, :]   # bottom-left   -> (H/2, W/2, C)
    x2 = x[0::2, 1::2, :]   # top-right     -> (H/2, W/2, C)
    x3 = x[1::2, 1::2, :]   # bottom-right  -> (H/2, W/2, C)

    x = concat([x0, x1, x2, x3], axis=-1)   # glue the 4 neighbours -> (H/2, W/2, 4C)
    x = linear(in=4*C, out=2*C)(x)          # learnable squeeze 4C -> 2C
    return x                                # (H/2, W/2, 2C): half the grid, double the depth

Let's walk a concrete Swin-T through its four stages, starting from a 224×224 image cut into 4×4 patches, giving a 56×56 token grid with C channels. Stage 1 attends at 56×56×C. A patch-merging step then produces Stage 2 at 28×28×2C; another gives Stage 3 at 14×14×4C; a final one gives Stage 4 at 7×7×8C. Four feature maps, four scales — fine detail at the bottom of the pyramid, whole-object semantics at the top. This deliberately mirrors a CNN: pooling layers in a ResNet do the same halving-of-resolution, widening-of-channels dance to build their classic feature pyramid.

Swin as a universal backbone

Now the payoff. In computer vision a 'backbone' is the feature-extractor stem that you bolt task-specific heads onto: a classification head, a detection head, a segmentation head. For years that backbone was a CNN like ResNet, precisely because detection and segmentation frameworks expect a pyramid of multi-scale feature maps to feed structures like a Feature Pyramid Network. Because the Swin Transformer emits exactly that pyramid — its four stages at 1/4, 1/8, 1/16 and 1/32 of the input resolution — it slots straight into those existing frameworks as a drop-in replacement for the CNN. No re-engineering of the heads required.

It helps to see how the three ideas snap together into one coherent machine. Windowed attention gives Swin a locality inductive bias and a cost that is linear in image size. Shifted windows let information leak across window boundaries layer by layer, restoring the global context that made attention worth using. Patch merging stacks the result into a multi-scale hierarchy. Locality + global flow + hierarchy: that is precisely the trio of properties that made CNNs such good vision backbones — now reconstructed inside a Transformer that can still learn rich, content-dependent attention.

This is exactly where plain ViT struggles as a backbone. With a single low-resolution grid and no native pyramid, a vanilla ViT cannot easily hand a detection or segmentation head the range of scales it needs — you have to graft extra machinery on to fake one. Swin makes that grafting unnecessary, which is a big part of why it caught on so fast.

The real-world verdict matches the theory. When Swin appeared it set new state-of-the-art numbers on the flagship detection and segmentation benchmarks (COCO object detection, ADE20K semantic segmentation) while staying efficient, and it was rapidly adopted as a general-purpose vision backbone across industry and research. In the lineage of this track, Swin is the architecture that finally made the Vision Transformer practical for dense, high-resolution, real-world images — not just for classifying tidy 224×224 thumbnails. It is, in essence, a successful hybrid of CNN and Transformer thinking: CNN-style hierarchy and locality, Transformer-style attention.

Where Vision Transformers are headed

Step back and look at the arc this track has traced. The original Vision Transformer proved that pure attention, fed image patches, could match CNNs — but only when trained on enormous labelled datasets. The Data-efficient image Transformer (DeiT) then showed how to train a strong ViT on ordinary ImageNet alone, using distillation and careful recipes. The Swin Transformer re-engineered the architecture itself so attention scales to real, high-resolution images and serves as a universal backbone. Three moves: prove the idea, make it data-efficient, make it scale.

The deepest answer to ViT's data hunger, though, is to stop relying on labels at all. Self-supervised pretraining learns from raw, unlabelled images by inventing a task that requires no human annotation. The leading example is masked image modeling, as in MAE (Masked Autoencoder): randomly hide most of an image's patches — often 75% — and train the Transformer to reconstruct the missing pixels from the few that remain. To fill in a hidden patch correctly, the model is forced to understand objects, textures and context, so it learns rich features for free. This is the direct visual cousin of how large language models pretrain by predicting masked or next words.

A second, complementary family is self-distillation, the idea behind DINO. Here two copies of the network — a 'student' and a slowly-updated 'teacher' — each see different random crops and augmentations of the same image, and the student is trained to match the teacher's output. With no labels at all, the network learns features so well-organised that, remarkably, its attention maps light up on object boundaries on their own, and its features cluster by semantic category. Together, masked image modeling and self-distillation are the real path to data efficiency: pretrain once on a sea of unlabelled images, then fine-tune on whatever small labelled task you actually care about.

Zoom out one more level and the biggest trend is unification. The exact same Transformer block you met in guide 3 now underlies state-of-the-art models for text, images, audio and video alike. Because an image-as-patches is just a sequence and text-as-tokens is just a sequence, a single architecture can ingest both at once — this is what powers modern multimodal models that caption images, answer questions about a photo, or generate pictures from a sentence. Vision and language, once separate research worlds, increasingly speak the same architectural language: attention over sequences of tokens.