Switch Transformer

The Switch Transformer is a mixture-of-experts model that pushes routing to its simplest extreme: each token goes to exactly one expert, top-1, rather than two or more. The authors showed this top-1 switch routing is enough to train stable, very large sparse models, scaling to over a trillion parameters, while keeping the per-token compute of a single feed-forward block.

Routing top-1 halves the routing compute and communication versus top-2 and simplifies the gating, but it makes load balance fragile, so the design leans on three ingredients: an auxiliary load-balancing loss, an expert capacity factor that caps how many tokens each expert accepts with excess tokens dropped through the residual, and selective bfloat16/float32 precision to keep the router numerically stable. It also reframed MoE as a way to trade memory for sample efficiency, reporting large pretraining wall-clock speedups at equal quality.

Switch Transformer is the work that made sparse MoE practical and popular for language models, establishing top-1 routing, capacity factors, and balancing losses as the standard toolkit later refined by GShard, Mixtral, and expert-choice routing. Its broader claim is a scaling argument: when you are memory-rich but compute-bound, adding sparse parameters buys quality more cheaply than adding depth or width, because most of those parameters sit idle for any given token. That reframing, parameters are cheap and selectively activated, is the conceptual seed of every large MoE language model that followed.