adam optimizer

A single global learning rate is a blunt instrument: some weights want big steps because their gradients are tiny and consistent, while others want tiny steps because their gradients are large and erratic. Adam (Adaptive Moment Estimation, Kingma and Ba, 2015) gives every parameter its own automatically tuned step size. The idea fuses two tricks. First, momentum: keep a running average of the gradient so updates are smooth. Second, per-parameter scaling: keep a running average of the squared gradient as a measure of how large and noisy each coordinate's gradient typically is, and divide each update by that, so consistently-small-gradient weights get amplified and consistently-large-gradient weights get tamped down. The result is an optimizer that 'just works' across a huge range of problems with little tuning, which is why it is the default for training transformers, CLIP, diffusion models, and most modern vision-language systems.

The update keeps two exponential moving averages. The first moment m ← β₁ m + (1−β₁) g estimates the mean gradient (momentum), and the second moment v ← β₂ v + (1−β₂) g² estimates the mean squared gradient elementwise, where g is the current gradient and g² is squared component by component. Because both start at zero they are biased toward zero early in training, so Adam applies bias correction m̂ = m/(1−β₁ᵗ) and v̂ = v/(1−β₂ᵗ) at step t. The parameters then update as θ ← θ − η · m̂ / (√v̂ + ε). Dividing by √v̂ is the adaptive part: it is like normalizing each coordinate's step by an estimate of its gradient's root-mean-square magnitude. The tiny constant ε (epsilon) only prevents division by zero.

The default hyperparameters are remarkably robust: β₁ = 0.9, β₂ = 0.999, ε = 1e-8, and learning rate η around 1e-3 (the often-cited 3e-4 is a popular safe choice for deep nets). β₁ controls the momentum timescale and β₂ the much longer timescale over which the per-parameter scale is estimated. Because the early estimate of v̂ is unreliable, Adam is almost always paired with a learning-rate warmup that ramps η up over the first few hundred to few thousand steps, especially for transformers and large-batch training.

The most important pitfall is weight decay. Adding an L2 penalty to the loss does NOT give clean weight decay under Adam, because the penalty's gradient gets divided by √v̂ along with everything else, so heavily-updated weights are decayed less. AdamW (Loshchilov and Hutter) fixes this by decoupling weight decay from the gradient step — subtracting λθ directly from the parameters — and AdamW is now the standard for training ViT, DETR, and most large vision models. Other caveats: vanilla Adam can fail to converge on some convex problems (motivating the AMSGrad fix), and SGD with momentum sometimes generalizes better than Adam on pure CNN image classifiers, so the 'best' optimizer is genuinely architecture-dependent.