natural gradient descent
Ordinary gradient descent treats the parameter space as flat: it penalizes moving far in plain Euclidean distance between coordinate vectors. But the quantity we actually care about is how much the model's predictive distribution changes, and that is measured by the Kullback-Leibler divergence, not by the raw distance between weights. Natural gradient descent fixes the mismatch by asking for the steepest descent direction with respect to that statistical distance, so the step you take does not depend on how you happened to parameterize the model.
Concretely, the natural gradient preconditions the ordinary gradient by the inverse of the Fisher information matrix, giving the update theta <- theta - eta F^{-1} grad L. The Fisher F is the expected outer product of the log-likelihood gradient, which is also the local second-order curvature of the KL divergence between the model and a slightly perturbed copy of itself. Multiplying by F^{-1} rescales each direction by how statistically sensitive the output is to it, producing steps that are invariant to smooth reparameterizations.
This invariance is the deep payoff: progress no longer depends on arbitrary coordinate choices, and the method is the theoretical ancestor of K-FAC and of trust-region policy methods. The catch is scale: F is parameter-count by parameter-count, so forming or inverting it exactly is hopeless for deep nets, and every practical natural-gradient method is really an approximation to F^{-1}.
The natural gradient is the ordinary gradient preconditioned by the inverse Fisher information.
The Fisher equals the Gauss-Newton matrix for common losses (squared error, cross-entropy), which is why practical natural-gradient methods often precondition with the positive-semidefinite Gauss-Newton matrix instead of the true Hessian.