MoE routing

The router is the small brain that decides which experts each token visits. For a token's hidden vector it computes one score per expert, usually with a single linear layer followed by a softmax, then keeps the top few experts, often the top one or two. Those experts run, and their outputs are blended in proportion to the router's scores. So routing is just a learned, per-token lookup that says where to send this particular word right now.

The tricky part is that the choice is discrete, you either pick an expert or you do not, which is not smoothly differentiable. Training sidesteps this by passing gradients through the soft scores of the experts actually chosen, so the router gradually learns to send, say, code-like tokens to one expert and punctuation to another. Two common styles are token-choice, where each token grabs its favorite experts, and expert-choice, where each expert grabs its favorite tokens; the latter helps keep load even.