Mixture of Experts: Bigger but Cheaper

Dense models do all the work, every time

In a normal — *dense* — model, every token passes through every parameter on every step. Doubling the parameters doubles the compute per token. That linear coupling is the wall: more parameters help, but you pay for all of them on every single token, forever, at training and at inference.

Mixture of Experts (MoE) breaks that coupling. Replace the single feed-forward network in a block with many parallel feed-forward networks — the experts — but route each token through only a few of them. A model can hold 600 billion parameters of knowledge while any given token only touches 30 billion. You buy capacity without buying compute.

The router decides who works

The brains of an MoE layer is a tiny network called the router or gate. For each token, MoE routing scores all the experts, picks the top few — often top-2 of, say, 64 — sends the token only to those, and blends their outputs weighted by the router's scores. The router is learned alongside everything else: over training, experts quietly specialize, and the router learns which token goes where.

y=\sum_{i\in\text{top-}k} g_i(x)\,E_i(x),\qquad g(x)=\mathrm{softmax}\!\left(W_g\,x\right)

Top-k gating: the router softmax-scores every expert but mixes only the few it selects.

scores  = router(token)              # one score per expert, e.g. 64 scores
top     = top_k(scores, k=2)         # pick the 2 highest-scoring experts
out     = sum(softmax(top.scores)[i] * expert[top.idx[i]](token)
              for i in range(2))      # blend just those two

Top-2 routing: 64 experts exist, but only 2 run for this token.

Load balancing is where it breaks

MoE has one nasty failure mode. Early in training the router finds a couple of experts that happen to be slightly good, sends them more tokens, which trains them more, which makes them better — a rich-get-richer spiral. Left alone, a handful of experts do all the work and the rest are dead weight, wasting most of your capacity. Fixing this is the whole craft of expert load balancing.

Auxiliary load-balancing loss — add a penalty that pushes the router toward spreading tokens evenly across experts, fighting the rich-get-richer spiral directly.
Expert capacity limits — cap how many tokens any one expert may accept per batch; overflow tokens are dropped or passed through unchanged so no expert hogs the queue.
Noisy or bias-corrected routing — perturb the router's scores during training so a slightly-better expert can't instantly monopolize, giving cold experts a chance to warm up.

\mathcal{L}_{\text{aux}} = \alpha \, N \sum_{i=1}^{N} f_i \, P_i

The auxiliary load-balancing loss penalizes each expert's token share f_i times its mean router probability P_i, pushing routing toward an even spread.

Why labs put up with the headache

MoE adds real complexity: an extra loss to tune, tokens that scatter unevenly across GPUs, and a router that can quietly collapse. Labs accept all of it because the scaling math is too good to refuse — for a fixed compute budget per token, a sparse model reaches lower loss than the dense model of the same active size. DeepSeek, Mixtral, and Qwen's largest open models are all MoE. The trade you are making is plain: spend memory to hold all the experts, save compute by only running a few.