mixture-of-experts
/ MIKS-cher uv EK-sperts /
Mixture-of-experts is a way to make a model much bigger in total knowledge while keeping the cost of using it roughly flat. Instead of one large feed-forward network that every token must pass through, the layer holds many smaller "expert" networks side by side, plus a small "router" that, for each token, picks just a couple of experts to actually run. Picture a hospital with dozens of specialists: a triage nurse glances at each patient and sends them to the two most relevant doctors, rather than making everyone see every doctor.
The beauty is the split between total size and active size. A mixture-of-experts model might contain, say, eight times more parameters than a comparable dense model, but because each token only fires two of eight experts, the computation per token barely grows. You get the storage capacity of a giant model at the running cost of a much smaller one. This is why several of the largest, most capable models are built this way — it is one of the few levers for adding capacity without a proportional explosion in compute.
The catch is that the savings are real for computation but not for memory: all those experts must still be loaded and held, so a mixture-of-experts model is large and demanding to deploy even if it is cheap per token to run. The router is also delicate — if it lazily sends most tokens to the same few experts, the others sit idle and the capacity is wasted, so training needs extra tricks to keep the load balanced. And the experts rarely become clean human-interpretable specialists; the routing is learned and often opaque, so "experts" is a useful metaphor more than a literal description.
A model has 8 experts per layer but routes each token to only 2. Its total parameter count rivals a model four times larger, yet each token's compute matches a much smaller dense model — capacity of the giant, running cost closer to the dwarf.
Many experts on the books, only a few on the clock per token.
Mixture-of-experts saves compute, not memory — every expert must still be stored and loaded, so these models are big and demanding to host. And the "experts" rarely line up with human-meaningful topics; routing is learned and largely opaque, so don't picture a tidy panel of named specialists.