weight decay
/ wayt dih-KAY /
Weight decay is a gentle pressure that keeps a model's internal numbers (its weights) from growing too large. The intuition: a model with huge weights is one that has latched onto sharp, extreme rules tailored to its training data — often a sign it's memorizing rather than understanding. Weight decay adds a constant nudge toward smaller weights, encouraging simpler, smoother solutions that tend to generalize better to new data.
It works by adding a penalty to the loss function: along with "how wrong are your predictions," the model is also charged for "how big are your weights." The most common form, L2 regularization, penalizes the sum of the squared weights, which gently shrinks all of them toward zero. A related form, L1 regularization, penalizes the sum of absolute weights and tends to push some weights exactly to zero — effectively switching features off, which can make the model simpler and more interpretable. A single strength knob controls how hard this pressure pushes.
Why bother? Because flexibility is a double-edged sword: a model powerful enough to learn the real pattern is also powerful enough to memorize noise, and weight decay tilts it toward the former. It directly fights overfitting, alongside tools like dropout and early stopping. One honest wrinkle: in adaptive optimizers like Adam, the naive way of adding this penalty interacts badly with the per-weight step sizes, which is precisely why AdamW was created to "decouple" weight decay and apply it cleanly.
Fitting a wiggly curve through 10 noisy points, an unrestrained model produces a wild, spiky line that hits every point exactly but swings absurdly between them. Add weight decay and the same model settles for a smooth, gentle curve that misses each point slightly but captures the real trend — and predicts new points far better.
A penalty on big weights buys smoother, more generalizable curves.
"Weight decay" and "L2 regularization" are the same thing for plain SGD but subtly differ inside adaptive optimizers. For transformers, use AdamW, which applies decoupled weight decay correctly.