AdamW and decoupled weight decay

For plain SGD, adding an L2 penalty to the loss and 'decaying the weights' a little each step are exactly the same operation. People assumed this equivalence carried over to Adam, but it does not. Because Adam divides each gradient by a per-parameter estimate of its recent magnitude, folding the L2 term into the loss means that term is also divided by the same denominator, so heavily updated weights get decayed far less than lightly updated ones. The regularization becomes inconsistent and entangled with the optimizer state.

AdamW decouples the two. Instead of putting the penalty into the loss where it flows through the adaptive denominator, it subtracts the decay directly from the weights at the end of each step, alongside the adaptive update: theta <- theta - eta (m-hat / (sqrt(v-hat) + epsilon) + lambda theta). Now every weight shrinks in true proportion to its own magnitude, independent of how the adaptive scaling treated its gradient, which is what weight decay was always meant to do.

This small fix matters a great deal in practice: AdamW is now the default optimizer for Transformer and large-language-model training. Decoupling restores honest, parameter-independent regularization and lets you tune the learning rate and the decay strength more or less independently instead of having them interfere through the second-moment term.