conjugate gradient in TRPO

To take a natural-gradient step, TRPO must solve a linear system: the Fisher matrix times the search direction equals the policy gradient. For a modern neural policy that matrix has millions of rows and columns, so forming it, let alone inverting it, is out of the question. Conjugate gradient is the escape hatch. It is an iterative solver that finds the direction using only the ability to multiply the Fisher matrix by a vector, never the matrix itself.

Those Fisher-vector products can be computed cheaply with automatic differentiation — roughly the cost of a couple of extra backward passes — by differentiating the KL divergence twice in a particular order. Ten or so conjugate-gradient iterations usually give a good enough direction, turning an impossible exact inversion into a handful of gradient-sized operations. This Hessian-free trick is what makes second-order policy optimization tractable at deep-network scale, and the same machinery appears in natural-gradient and K-FAC optimizers elsewhere.