model capacity
/ MOD-ul kuh-PASS-it-ee /
Model capacity is a measure of how complicated the patterns are that a model can possibly express — its raw flexibility. A low-capacity model is like a single straight ruler: useful, but it can only draw lines. A high-capacity model is like a sheet of soft clay that can be molded into almost any shape. Capacity grows with things like the number of parameters and the richness of the model's form. It sets the ceiling on what the model could learn, before any training has happened.
Capacity is the dial behind the two classic failure modes. If capacity is too low for the problem, the model cannot bend itself enough to match the real pattern and underfits — the ruler can't trace a curve. If capacity is too high relative to how much data you have, the model has enough flexibility to memorize the training examples, noise and all, and overfits — the clay takes the imprint of every fingerprint. The right capacity is matched to both the complexity of the problem and the amount of data on hand.
Two honest cautions. First, capacity is potential, not achievement: a high-capacity model could fit a complex pattern, but only good training and enough data make it actually do so. Second, more capacity is not simply better. Bigger models cost more to train and run, are easier to overfit on small data, and harder to interpret. The goal is enough capacity to capture the real signal and no more — though in the era of huge models trained on huge data, where exactly that line sits has become genuinely subtle.
Try to learn a curvy pattern with only 20 data points. A straight line (tiny capacity) misses the curve entirely — underfitting. A 50-parameter wiggle (huge capacity) snakes through all 20 points perfectly but predicts garbage in between — overfitting. A gentle curve with just enough capacity captures the trend and generalizes.
Capacity sets the ceiling; data and training decide whether you hit it well.
Capacity is about what a model could represent, not how well it actually performs. A model can have huge capacity and still perform poorly if it is badly trained or starved of data — capacity is the size of the toolbox, not the quality of the work.