natural policy gradient

The ordinary policy gradient points uphill in parameter space, but parameter space is a misleading map: a tiny tweak to one weight can swing the policy wildly while a big tweak to another barely changes behaviour. The natural policy gradient fixes the map. It measures distance by how much the policy's action distribution actually changes, not by how much the numbers in the weight vector change, and then takes the steepest-ascent step under that honest, behaviour-aware notion of distance.

Mathematically, you precondition the vanilla gradient by the inverse of the Fisher information matrix, which is the local curvature of the KL divergence between nearby policies. This makes the update invariant to how you happened to parameterize the policy, so progress no longer stalls in flat or skewed regions of parameter space. TRPO is essentially a natural policy gradient with an added trust-region step-size rule; computing the Fisher inverse exactly is expensive, so practical methods approximate it with conjugate gradient.