Why a first-order step can be a bad step
Picture a loss surface that is a long, thin ravine — gentle along its length, steep across its width. The gradient points mostly across, so a fixed step size either crawls down the length or bounces off the walls. The mismatch is captured by the condition number of the curvature matrix: the ratio of its largest to smallest eigenvalue. When that ratio is large, first-order methods slow to a crawl, and no single learning rate is right for all directions.
A gradient-descent path stepping down a loss surface toward the minimum.
The exact cure is Newton's method: precondition by the inverse Hessian, `w ← w − η H⁻¹ g`, which makes the ravine locally round so every direction is equally easy. For a network with billions of parameters this is fantasy — `H` is a square matrix in the number of parameters, far too large to form, store, or invert. Curvature-aware optimization is the engineering of useful approximations to `H⁻¹ g` that fit in memory and run in a fraction of the step budget.
Natural gradient: curvature from probability, not loss
Natural gradient descent preconditions not by the Hessian of the loss but by the Fisher information matrix — the curvature of the model's output distribution as you move the parameters. The key property is invariance: a natural-gradient step is the same physical move whether you parameterize a Gaussian by its variance or its log-variance, because the Fisher measures change in the predicted distribution itself, using the KL divergence as the local yardstick.
Natural gradient rescales the loss gradient by the inverse Fisher information of the model's output distribution.
That invariance is the deep reason natural gradient is attractive: ordinary gradient descent's path depends on arbitrary choices of units, while the natural gradient follows the same trajectory through distribution space. For exponential-family losses the Fisher coincides with a positive-semidefinite curvature called the Generalized Gauss-Newton matrix, which is why natural gradient and Gauss-Newton methods so often meet in practice.
K-FAC: a Kronecker factorization of curvature
K-FAC (Kronecker-Factored Approximate Curvature) makes the Fisher tractable with one structural assumption: for a single layer, the curvature factorizes as a Kronecker product of two small matrices — one built from the layer's input activations, the other from the backpropagated output gradients. Instead of inverting one enormous block, you invert two modest ones whose dimensions are the layer's input and output widths, and the inverse of a Kronecker product is the Kronecker product of the inverses.
K-FAC's key trick: a Kronecker product factorizes a layer's Fisher, so its inverse is just the Kronecker product of two small inverses.
# K-FAC sketch for one linear layer (a = input, g = output-grad) A = ema(A, a.T @ a) # input covariance (in x in) G = ema(G, dg.T @ dg) # grad covariance (out x out) # preconditioned update reshapes the gradient matrix dW: dW_nat = inv(G) @ dW @ inv(A) # = (A kron G)^-1 applied to vec(dW)
The factors are stale-tolerant — you can recompute and re-invert them only every few hundred steps — so K-FAC's amortized overhead is modest. In practice it can reach a target loss in noticeably fewer steps than tuned SGD on some vision and reinforcement-learning workloads. The price is real engineering complexity and extra memory for the factors, which is why it never became the universal default.
Hessian-free: curvature without the matrix
Hessian-free optimization takes a different route: never form any curvature matrix at all. The trick is that you can compute a Hessian-vector product `Hv` at the cost of roughly one extra backward pass, using automatic differentiation, even though `H` itself is never materialized. With cheap `Hv` products in hand, you solve the Newton system `H δ = −g` approximately with conjugate gradient, an iterative solver that only ever needs to multiply `H` by vectors.
Each conjugate-gradient iteration is one Hessian-vector product, and you truncate after a handful of them, trading exactness for speed. This is elegant because it inherits curvature information directly from backprop with no factorization assumption — but every outer step now contains an inner loop, so wall-clock cost per update is high. Hessian-free shines when each step is precious (small, expensive-to-evaluate models) and fades when steps are cheap and plentiful.
Shampoo: full-matrix adaptivity, factored per axis
Adam keeps a diagonal preconditioner — one scalar per parameter — which ignores all correlations between coordinates. Shampoo keeps a full-matrix preconditioner but factors it along each axis of the parameter tensor. For a weight matrix it maintains a left preconditioner over rows and a right preconditioner over columns, each accumulated from outer products of gradients and raised to a fractional inverse power. The update multiplies the gradient on both sides, capturing within-row and within-column correlations that a diagonal method cannot.
Shampoo preconditions a weight matrix by the inverse fourth-roots of per-axis gradient covariances — full-matrix adaptivity, factored per axis.
Shampoo sits between Adam and full Newton: richer than diagonal, far cheaper than the true curvature. Distributed implementations amortize the costly matrix-root computations across devices and steps, and large-scale results — including competitive optimizer benchmarks — have made it one of the most credible second-order-flavored methods for real training. It is also the conceptual parent of the orthogonalizing optimizers you will meet in Guide 5.