trust region policy optimization (TRPO)

Imagine adjusting a recipe you can only taste once: change too much at a time and you might ruin it, with no way to recover. TRPO treats a policy update the same way. After collecting experience under the current policy, it asks how far it can push the policy before its estimate of improvement stops being reliable. It then takes the largest step that stays inside this trustworthy neighbourhood — the trust region — rather than a fixed-size gradient step that might overshoot and collapse performance.

Concretely, TRPO maximizes a surrogate objective — the expected advantage weighted by the ratio of new to old action probabilities — subject to a hard constraint that the average KL divergence between the old and new policy stays below a small threshold. It approximates the objective linearly and the constraint quadratically, computes a natural-gradient direction with conjugate gradient, then runs a backtracking line search to confirm the real KL constraint and improvement actually hold. This buys stable, near-monotonic progress at the price of heavy second-order machinery.