mixture-of-depths (MoD)

A standard transformer spends the same compute on every token at every layer, even though many tokens are easy glue words. Mixture-of-depths lets each layer process only a chosen subset of tokens and lets the rest skip straight past it through the residual connection, so the model spends compute where it is actually needed.

Each block has a lightweight router that scores tokens and, under a fixed capacity — say the top 12.5 percent of the sequence — selects which tokens enter the attention-plus-MLP computation; the unselected tokens are routed around it unchanged. Because the capacity is static, the FLOPs per layer are known in advance, unlike data-dependent early exit. The router is trained jointly, with an auxiliary predictor or an expert-choice-style selection that keeps the top-k decision usable causally at inference time.

MoD borrows the routing idea from mixture-of-experts but routes across depth rather than across parallel experts. The result is a model that reaches the same loss with fewer FLOPs, or runs faster at equal parameter count, by simply not computing layers for tokens that do not need them.