sparse mixture of experts (sparse MoE)

A sparse mixture of experts replaces a transformer's single feed-forward block with many parallel feed-forward experts plus a small router that sends each token to just a few of them. The model holds a large pool of parameters, but any given token activates only a thin slice. This decouples parameter count from compute: you can grow capacity tenfold while keeping the per-token floating-point operations nearly fixed.

A gating network produces scores over E experts; the top-k, often k equals 2, are selected and their outputs combined with weights from a softmax over the chosen scores. Because only k of the E feed-forwards run per token, compute scales with k rather than E. Training needs a load-balancing auxiliary loss so the router does not collapse onto a few experts, plus capacity limits that drop or reroute tokens when an expert's buffer overflows. Sparsity routes gradients only through the selected experts, which keeps cost down but also complicates optimization.

Sparse MoE underlies models like Mixtral and, reportedly, GPT-4-class systems, delivering strong quality per training floating-point operation. The practical costs are memory, since all experts must be stored and sharded across devices, the all-to-all communication of token routing, and sensitivity to load imbalance.