trust region policy optimization (TRPO)

Trust region policy optimization is policy gradient on a leash. The danger with raw policy gradients is that one overlarge step can collapse a working policy beyond recovery. TRPO limits how far each update may move the policy, measured by the KL divergence between the old and new action distributions, so every step stays inside a trust region where the local approximation can still be believed.

It maximizes an importance-weighted surrogate advantage subject to a constraint that the mean KL from the old policy stays below a small bound. The constrained step is solved along the natural-gradient direction: Fisher-vector products plus conjugate gradient approximate the search direction without ever forming the Fisher matrix, followed by a backtracking line search that enforces the KL constraint and a genuine improvement. A penalty version comes with a monotonic-improvement lower bound.

TRPO is principled but second-order and expensive. Proximal policy optimization is the cheaper first-order successor that replaces the hard KL constraint with a clipped surrogate, which is why PPO is far more common in practice.