expert parallelism

A mixture-of-experts layer holds many parallel feed-forward sub-networks — the experts — but routes each token to only a few of them. Because the experts are independent and only a subset fires per token, the natural way to scale them is to place different experts on different devices. That is expert parallelism: each device stores and computes a disjoint group of experts, so the total parameter count grows with the number of devices while the compute per token stays fixed at the few experts it actually visits.

The defining mechanism is a pair of all-to-all collectives. After the router scores each token, a first all-to-all dispatches every token to whichever device hosts its chosen expert; the experts run locally; a second all-to-all returns the processed tokens to their original positions. The all-to-all volume depends on how tokens distribute over experts, so load imbalance is the central enemy — a popular expert overloads its device while others idle, and capacity factors, auxiliary load-balancing losses, and token-dropping policies all exist to tame it. Expert parallelism is normally combined with data parallelism over the non-expert (attention) weights.

Expert parallelism is how trillion-parameter sparse models are trained without trillion-parameter compute: it decouples capacity from FLOPs. Its scaling ceiling is the all-to-all — a global, latency-sensitive exchange that stresses the interconnect topology far more than the all-reduce of dense training.