Fisher information of a policy

The Fisher information matrix is the curvature that tells the natural policy gradient which directions are sensitive and which are inert. Picture standing on the policy's loss surface: in some directions a small parameter move barely changes how the policy acts, in others it lurches the action probabilities sharply. The Fisher matrix is exactly the average outer product of the score — the gradient of the log-probability of chosen actions — so it captures how strongly each parameter direction reshapes the policy's behaviour.

Crucially, this matrix is the second-order, or local quadratic, approximation of the KL divergence between a policy and a nearby one. That is why preconditioning by its inverse turns a step in parameter space into a controlled step in behaviour space, and why a quadratic KL constraint reduces to one expressed through the Fisher matrix. For deep policies the full matrix is far too large to form, so methods never build it; they only need Fisher-vector products, which conjugate gradient can supply efficiently.