token merging (ToMe)

A transformer's cost grows with the number of tokens, yet many tokens are redundant — neighboring image patches that look alike, or repetitive text. Token merging combines similar tokens into one as they pass through the network, shortening the sequence and cutting both attention and MLP cost; in its inference-only form it needs no retraining at all.

ToMe inserts a step between attention and the MLP in each block: it measures token similarity using the cosine similarity of the attention keys, applies a fast bipartite soft matching to pair up the most similar tokens, and averages each pair into a single merged token, reducing the count by a fixed amount r per layer. A size weighting keeps each merged token proportionally influential in later attention. For dense outputs like segmentation the merges can be unmerged at the end.

Token merging gives the largest gains in vision transformers and diffusion models, where it buys big speedups for small quality loss. In language it overlaps with pruning and KV-cache compression and must respect causal masking and positional structure, so it is applied more carefully there.