sparse vs. dense models

Dense and sparse describe how much of the network each token actually wakes up. In a dense model every parameter participates in every token, so the parameters you store and the compute you spend are the same number. In a sparse model, almost always a mixture of experts, only a small slice of the parameters fire per token, so a model can hold far more total parameters than it ever runs at once.

The practical split is between two budgets: total parameters, which roughly tracks how much the model can know, and active parameters, which tracks the cost per token. A sparse model decouples them, buying a knowledge capacity well beyond its compute. The price is memory, since every expert must be stored and moved even though most sit idle, plus the routing and balancing machinery. Choosing between them is really choosing whether your bottleneck is compute or memory.