expert load balancing

Left alone, a router tends to play favorites: a few experts get most of the tokens and grow strong, while the rest starve, never improve, and waste their parameters. This collapse is the central failure mode of mixture-of-experts training. Load balancing is the set of pressures that keep traffic spread across all experts, so every specialist gets enough practice to become useful.

The usual tool is an auxiliary loss added during training that penalizes uneven routing, nudging the fraction of tokens each expert receives toward equal. Hardware adds a second reason to balance: experts are spread across many devices, and the slowest, most overloaded device sets the pace, so a token capacity cap drops or reroutes overflow once an expert is full. Newer schemes lean on expert-choice routing or bias terms instead of a heavy auxiliary loss, since that loss, pushed too hard, can itself hurt quality.