patch merging

Patch merging is how a hierarchical vision transformer zooms out between stages, turning many fine tokens into fewer, coarser, information-richer ones. A plain ViT keeps the same number of tokens at the same resolution from start to finish — it is flat. But good vision usually needs a pyramid: fine detail early, broad context late, the way a CNN halves spatial size and doubles channels stage by stage. Patch merging is the transformer's equivalent of that downsampling step.

Mechanically it groups each 2x2 block of neighbouring tokens and fuses them into one. The four tokens, each of width C, are concatenated into a single vector of width 4C, then passed through a linear layer that projects 4C down to 2C. The outcome: the spatial token count drops by a factor of four (height and width each halved), while the channel dimension grows from C to 2C. So a 56x56 grid of C-dimensional tokens becomes a 28x28 grid of 2C-dimensional tokens — fewer, fatter tokens covering larger image regions.

Repeating this between stages builds the multi-scale hierarchy that makes models like Swin work for dense prediction. Starting at a fine resolution and merging several times yields feature maps at a cascade of scales (for example 1/4, 1/8, 1/16, 1/32 of the input), which is exactly what feature-pyramid networks, detection heads and segmentation decoders consume. It also keeps compute in check: because attention cost grows with the square of the token count, quartering the tokens at each stage sharply reduces the work in the deeper, wider stages.

Note the contrast with the very first patch embedding. Patch embedding is a one-time pixels-to-tokens conversion at the input; patch merging operates later, token-to-token, repeatedly, fusing already-computed representations rather than raw pixels. It is the learned, transformer-native counterpart of strided convolution or pooling, with the concatenate-then-project design preserving more information than a simple average would.