the Adam optimizer
/ AD-um /
Imagine descending a landscape where some directions are steep cliffs and others are nearly flat plains, and you only ever feel a noisy, jittery sense of the slope. You would want to take confident strides along the flat directions but cautious small steps down the cliffs, and to smooth out the jitter by remembering recent slopes. Adam is the optimizer that does all three at once: per-direction adaptive step sizes plus momentum smoothing — which is why it became the default workhorse for training neural networks.
Adam (Adaptive Moment Estimation) keeps two running averages of the mini-batch gradient g_k. The first moment m_k = beta_1 * m_{k-1} + (1 - beta_1) * g_k is a smoothed average of the gradient (momentum, typically beta_1 = 0.9). The second moment v_k = beta_2 * v_{k-1} + (1 - beta_2) * g_k^2 is a smoothed average of the squared gradient, per coordinate (typically beta_2 = 0.999). After a bias correction that fixes their cold start (m-hat = m_k / (1 - beta_1^k), and likewise v-hat), the update is x_{k+1} = x_k - alpha * m-hat / (sqrt(v-hat) + epsilon). The division by sqrt(v-hat) is the key: each coordinate is scaled DOWN if its gradients have been large and erratic, and effectively scaled up where they have been small and steady — an automatic per-parameter learning rate. The epsilon (around 1e-8) just prevents dividing by zero.
Adam matters because it works well out of the box on the messy, non-convex, high-dimensional, noisy objectives of deep learning, needing far less learning-rate tuning than plain SGD and coping gracefully with gradients that vary wildly in scale across parameters. The honest caveats are real: Adam's adaptivity does NOT make it provably better than well-tuned SGD — on many vision tasks SGD-with-momentum still generalizes as well or better, and Adam's original convergence proof had a flaw later patched (AMSGrad), with the AdamW variant fixing how weight decay interacts with the adaptive scaling. It is also not a curvature (second-order) method despite the per-coordinate scaling — it has no real Hessian information — and on smooth deterministic problems a true quasi-Newton method like L-BFGS converges far faster. Adam's home turf is large-scale stochastic training, not classical smooth optimization.
Training a network where one weight sees gradients around 100 and another around 0.001, plain SGD with a single learning rate either explodes the first or crawls on the second. Adam divides each by the root-mean-square of its own recent gradients, so both move at a sensible per-coordinate pace — the adaptivity that lets a single alpha work across wildly different parameter scales.
Momentum plus per-coordinate scaling by recent gradient size.
Adam is a convenient default, not a universally superior method: well-tuned SGD-with-momentum often generalizes as well or better, especially in vision. Despite per-coordinate scaling it carries NO true curvature information, so on smooth deterministic problems L-BFGS beats it; use AdamW (decoupled weight decay) rather than naive L2-with-Adam.