regularization

Regularization is the umbrella name for any technique that constrains or biases a model so that it generalizes better to unseen data, usually by discouraging it from fitting the noise in its training set. The unifying intuition is Occam's razor: among all the ways to fit the training data, prefer the simpler, smoother, more stable explanation, because that is the one most likely to keep working on new data. Every method in this toolbox trades a little training-set fit for a lot of test-set reliability — they make the model do slightly worse on data it has seen in exchange for doing better on data it has not.

It is useful to split the toolbox into explicit and implicit regularizers. Explicit regularizers add a deliberate term or operation aimed at simplicity: weight decay / L2 (penalize large weights for smoother functions), L1 (push weights to exactly zero for sparsity), and dropout (randomly disable neurons to prevent co-adaptation and approximate an ensemble). These are things you switch on by choice and tune with a strength knob.

Implicit regularizers are effects that improve generalization as a side benefit of how you train, even though regularization was not their stated purpose. Data augmentation injects label-preserving variety that fights memorization; early stopping halts training before the model specializes to noise; the gradient noise of small-batch SGD biases the optimizer toward flatter, more robust minima; and batch and layer normalization smooth the landscape and add a touch of noise. In practice modern vision models stack several of these at once — for example a ViT trained with AdamW (weight decay), RandAugment and Mixup (augmentation), dropout and stochastic depth, and a cosine schedule.

The deeper framing is the bias-variance tradeoff and capacity control. A model with too much effective capacity for the data will overfit (high variance); regularization reduces that effective capacity — not by removing parameters, but by restricting the space of functions the model is likely to settle on — moving it toward the sweet spot where it is flexible enough to capture the real pattern but not so flexible that it chases noise. How much to apply is itself a hyperparameter to tune against a validation set: too little and you overfit, too much and you underfit.