JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The Geometry of Policy Space: Natural Gradients

A step that is small in parameters can be huge in behavior. The natural gradient measures distance where it actually matters — in the space of distributions.

Why Euclidean steps mislead

Ordinary gradient ascent treats every parameter direction as equally costly — a fixed step in parameter space. But what we care about is the policy's behavior, and the map from parameters to behavior is wildly uneven. Near a saturated softmax, a large parameter change barely shifts the action distribution; elsewhere a tiny tweak flips it entirely. Measuring step size in raw parameters is like measuring travel distance in degrees of latitude: the same number means different things in different places.

Ordinary gradient ascent takes equal-sized steps in parameter space, sliding down the loss surface blind to how much the policy's behavior actually shifts.

A loss curve with arrows stepping downhill toward the minimum.

The fix is to measure a step by how much the policy distribution actually changes, not by how much the numbers changed. The natural distance between two probability distributions is the KL divergence, and that is exactly the distance the monotonic-improvement penalty from Guide 1 used.

The Fisher information metric

For tiny moves, the KL divergence between the old and new policy is well approximated by a quadratic form whose matrix is the Fisher information matrix of the policy. Intuitively, the Fisher matrix is the local curvature of behavior: directions where the policy is very sensitive get large entries (so a step there is expensive), and insensitive directions get small entries (cheap). It turns the flat parameter space into a curved manifold where distance means behavioral change.

D_{\mathrm{KL}}\!\left(\pi_\theta \,\|\, \pi_{\theta+\Delta\theta}\right) \approx \tfrac{1}{2}\,\Delta\theta^{\top} F(\theta)\,\Delta\theta

For tiny moves, the KL change between the old and new policy is a quadratic form whose matrix is the Fisher information of the policy.

The natural policy gradient

The natural policy gradient preconditions the ordinary gradient by the inverse Fisher matrix. The result is the steepest-ascent direction in distribution space: the update that improves the surrogate objective the most per unit of KL change. It is invariant to how you parameterize the policy — reparameterize the network and the natural gradient points the same behavioral way, which is why it converges so much more gracefully than the raw gradient.

\tilde{\nabla}_{\theta} J(\theta) = F(\theta)^{-1}\,\nabla_{\theta} J(\theta)

The natural policy gradient preconditions the ordinary gradient by the inverse Fisher matrix — the steepest-ascent direction in distribution space.

In practice you feed it a low-variance advantage estimate. The standard choice is the generalized advantage estimator (GAE), which trades a little bias for a big variance cut and pairs naturally with every method in this track.

A unifying lens: mirror descent

There is an elegant way to see all of this at once. Mirror descent policy optimization frames each update as: maximize the surrogate minus a divergence-to-the-old-policy penalty. Choose KL as the divergence and you recover the natural-gradient / trust-region family; the KL-constrained update is just the constrained twin of the penalized form. This single template will reappear in TRPO, in PPO's penalty variant, and in RLHF.