natural policy gradient (NPG)

The ordinary policy gradient measures distance in parameter space, but parameters are a poor ruler: two very different parameter vectors can give nearly identical policies, while a tiny parameter change can swing the policy completely. The natural policy gradient measures distance in the space of distributions instead, so its steps are invariant to how the policy happens to be parameterized.

It preconditions the gradient by the inverse of the Fisher information matrix, which is the second-order metric the KL divergence induces on policy space. The update moves the parameters along the inverse-Fisher-times-gradient direction, the direction of steepest ascent under that natural geometry. Trust region policy optimization is essentially this step with a KL-constrained line search, and in practice the Fisher matrix is never inverted explicitly: Fisher-vector products with conjugate gradient compute the direction cheaply.

Because it follows the geometry of the policy distribution, the natural gradient is invariant to reparameterization and underlies TRPO, ACKTR, and related methods.