K-FAC optimizer
Natural gradient descent is wonderful in theory but the Fisher information matrix for a deep network has millions of rows and columns, so storing or inverting it is out of the question. K-FAC is the trick that makes approximate natural gradient affordable: it observes that, layer by layer, the Fisher block for a layer's weights factorizes approximately as a Kronecker product of two much smaller matrices, one built from the layer's inputs and one from the gradients flowing back into it.
For a layer whose weights are W, K-FAC approximates the curvature block as F_layer ~ A (kron) G, where A is the second-moment matrix of the input activations and G is the second-moment matrix of the pre-activation gradients. The Kronecker identity (A kron G)^{-1} = A^{-1} kron G^{-1} means you only ever invert the two small factors, whose sizes are the layer's input and output widths rather than the full parameter count. The preconditioned update multiplies the layer's gradient by these two inverses, optionally with a damping term added to keep things well conditioned.
In practice K-FAC converges in markedly fewer steps than first-order methods on CNNs and Transformers, at the cost of heavier per-step work: it must periodically re-estimate and re-invert the factors, and it relies on damping and a running average of the statistics. It is the bridge between elegant second-order theory and networks too large for the exact Fisher.
The Kronecker structure turns one huge inverse into two small ones.
K-FAC's Kronecker factorization is an approximation: it assumes activations and back-propagated gradients are statistically independent and ignores cross-layer correlations, so it captures most but not all of the true curvature.