mixture of experts (MoE)

A dense model runs every token through the same giant feed-forward layer, so making the model smarter means making that layer bigger and every token pays the full bill. A mixture of experts replaces one big feed-forward block with many smaller expert blocks plus a router that, for each token, picks just a couple of them to run. Picture a hospital where a receptionist sends each patient to two relevant specialists instead of making everyone see all of them.

This lets total parameters grow enormously while the work done per token stays small, because only the chosen experts activate. A model might hold sixty-four experts yet route each token to only two, so it carries the knowledge of a huge network at the compute cost of a small one. The catch is that all experts still live in memory, the router must be trained to make good choices, and keeping experts evenly used takes special care, which the next few terms unpack.