JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

TRPO: Making the Trust Region Practical

Trust Region Policy Optimization turns the monotonic-improvement theory into an algorithm you can run — using conjugate gradient and a backtracking line search.

The KL-constrained problem

Trust Region Policy Optimization (TRPO) makes the trade from Guide 1 explicit. Instead of penalizing the KL divergence with a fragile coefficient, it imposes a hard cap: maximize the surrogate objective subject to the average KL between old and new policy staying below a fixed budget. That budget is the radius of a trust region — the neighborhood in distribution space where the surrogate is trustworthy.

\max_{\theta}\ \mathbb{E}\!\left[\frac{\pi_\theta(a\mid s)}{\pi_{\theta_{\text{old}}}(a\mid s)}\,\hat{A}(s,a)\right]\quad \text{s.t.}\quad \mathbb{E}\big[D_{\text{KL}}(\pi_{\theta_{\text{old}}}\,\|\,\pi_\theta)\big]\le \delta

TRPO's core problem: maximize the importance-sampled surrogate advantage subject to a hard KL budget δ.

Choosing a constraint rather than a penalty is what gives TRPO its practical stability. A fixed KL budget keeps every update at the same behavioral size regardless of where you are on the loss surface, which is far easier to tune than a penalty weight that means different things at different points.

Solving it without inverting the Fisher matrix

Linearize the surrogate and quadratically approximate the KL constraint (its matrix is the Fisher information matrix) and the optimal direction is exactly the natural gradient. But forming and inverting the Fisher matrix is infeasible for a deep network. TRPO's trick is that it never needs the matrix itself — only its product with a vector, which can be computed cheaply.

Enter conjugate gradient. It solves the linear system that defines the natural-gradient direction using only Fisher-vector products, converging in a handful of iterations without ever materializing the matrix. Each Fisher-vector product is a single extra backprop, so the whole solve costs a small constant number of gradient passes.

F\,x=\hat{g}\ \Rightarrow\ x=F^{-1}\hat{g},\qquad \beta=\sqrt{\dfrac{2\delta}{x^{\top}F\,x}}

Conjugate gradient solves Fx = g for the natural-gradient direction using only Fisher–vector products, then β scales it to the KL-budget edge δ.

The line-search safety net

The conjugate-gradient direction comes from a quadratic approximation of the real KL constraint, so the full step it suggests might overshoot — the true KL could exceed the budget, or the true surrogate could actually drop. TRPO guards against this with a backtracking line search: it tries the full step, and if the true KL is too large or the surrogate did not improve, it halves the step and checks again, until both conditions hold.

Because the quadratic model can overshoot, TRPO backtracks along the proposed step — shrinking it like a careful descent — until the measured KL stays within budget and the surrogate truly improves.

Interactive gradient descent rolling down a loss surface, illustrating the backtracking line search that shrinks the step until it is safe.

TRPO end to end

  1. Run the current policy to collect a batch of trajectories and estimate advantages (GAE).
  2. Compute the surrogate gradient with importance sampling against the old policy.
  3. Use conjugate gradient with Fisher-vector products to get the natural-gradient direction.
  4. Scale that direction to the largest step inside the KL budget.
  5. Backtrack with the line search until the KL constraint holds and the surrogate improves; apply it.