AdamW for LLMs

AdamW is the workhorse optimizer behind almost every large language model. Plain gradient descent moves every weight by the same-sized step; Adam instead gives each weight its own adaptive step, scaling by running estimates of the gradient's mean and variance. Parameters with small, noisy gradients get larger effective steps, and parameters with large gradients get reined in — which makes training far more stable across the wildly different scales inside a transformer.

The W marks the one crucial fix over plain Adam: weight decay is decoupled from the gradient update, applied as a separate gentle pull of every weight toward zero. In ordinary Adam, naively folding weight decay into the gradient interacts badly with the per-parameter scaling and weakens regularisation; decoupling restores it cleanly. For huge models this matters, helping keep weights well-conditioned and improving generalisation.

AdamW's cost is memory: it stores two extra values (the running mean and variance) per parameter, so the optimizer state can be twice the size of the model itself. That overhead is a major reason large-scale training leans on memory-saving tricks like sharded optimizer states.