weight decay

Large weights let a network draw wild, wiggly decision boundaries that snake through every training point and so overfit; small weights force smoother, simpler functions that generalize better. Weight decay encodes the preference for small weights directly into training: at every step it nudges each weight a little toward zero, in proportion to its current size, so weights are only allowed to grow large if the data's gradient signal keeps pushing them there strongly enough to overcome the steady pull back. It is one of the oldest and most dependable regularizers, used in essentially every serious vision model.

In its classic form, weight decay is L2 regularization: you add a penalty (λ/2)‖θ‖² to the loss, where θ are the weights, ‖θ‖² is the sum of their squares, and λ (lambda) is the regularization strength. Taking the gradient of this penalty gives λθ, so the update gains a term that subtracts a fraction λ of each weight every step — literally decaying it toward zero. With plain SGD, 'add an L2 penalty to the loss' and 'decay the weights directly' are mathematically the same operation, which is why the two names are used interchangeably.

That equivalence breaks for adaptive optimizers, and this is the single most important practical point. In Adam, every gradient component — including the λθ from the L2 penalty — is divided by the per-parameter scale √v̂. So weights that happen to have large, frequent gradients get their decay shrunk, and the regularization no longer acts uniformly. AdamW (Loshchilov and Hutter, 'Decoupled Weight Decay') fixes this by removing the penalty from the loss and instead subtracting λθ from the weights directly, after the adaptive step: θ ← θ − η(adaptive step) − η λ θ. This decoupled form restores the intended uniform shrinkage and consistently generalizes better, which is why AdamW — not Adam-plus-L2 — is the standard for training ViT, DETR, ConvNeXt, and most large vision models.

Two practical notes. The strength λ trades off fit against simplicity: too large and the model underfits because the pull toward zero overwhelms the data; too small and it has no regularizing effect. And it is common to exclude certain parameters from weight decay — particularly the bias terms and the scale/shift parameters of normalization layers — because shrinking those toward zero hurts rather than helps.