JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why Policy Gradients Need a Leash

Vanilla policy gradients can wreck a good policy in a single step. Meet the surrogate objective and the promise of monotonic improvement.

The one-bad-step problem

Plain policy-gradient methods like the REINFORCE algorithm follow the policy-gradient theorem: nudge the policy parameters in whatever direction raises the expected return. That direction is locally correct, but the gradient says nothing about how far you may safely move. Take too large a step and the new policy can land in a region the old data never described — and a single update can collapse a policy that took hours to train.

This is worse in RL than in supervised learning for two reasons. First, the data distribution itself depends on the policy: change the policy and you change which states you visit, so a misstep poisons the very samples you learn from next. Second, gradient estimates are noisy — the high policy-gradient variance means the direction you trusted may have been mostly noise.

The agent–environment loop: because the policy chooses actions, changing it changes which states the agent even visits.

Diagram of an agent taking an action in an environment and receiving a new state and reward, forming a closed loop.

The surrogate objective

The key idea is to optimize a stand-in for the true return that we can evaluate using the data we already collected from the old policy. This surrogate objective reweights each sampled action by an importance-sampling ratio — the new policy's probability of that action divided by the old policy's — and multiplies it by the action's advantage, i.e. how much better than average that action turned out to be.

L(\theta) = \mathbb{E}_{s,a\sim\pi_{\text{old}}}\!\left[\frac{\pi_\theta(a\mid s)}{\pi_{\text{old}}(a\mid s)}\,\hat{A}(s,a)\right]

The surrogate objective: reweight old-policy data by the importance ratio times the advantage.

Maximizing the surrogate pushes probability mass toward actions with positive advantage and away from negative ones — exactly what we want. The catch: importance ratios are only trustworthy while the new policy stays close to the old one. Push the policy far and those ratios explode, the surrogate stops resembling the true return, and the guarantee evaporates. That closeness constraint is the seed of every method in this track.

The monotonic improvement guarantee

Here is the beautiful result that started the modern era. One can prove that the true return of the new policy is at least the surrogate value minus a penalty proportional to how far the new policy strays from the old (measured by a distribution distance). If you maximize the surrogate while keeping that distance small, you obtain a monotonic improvement guarantee: each update provably never decreases performance.

J(\pi_{\text{new}}) \;\ge\; L_{\pi_{\text{old}}}(\pi_{\text{new}}) \;-\; C\,\max_{s} D_{\mathrm{KL}}\!\big(\pi_{\text{old}}(\cdot\mid s)\,\|\,\pi_{\text{new}}(\cdot\mid s)\big)

Monotonic improvement: the new policy's true return is at least the surrogate value minus a KL-divergence penalty.

Where this ladder goes

  1. Guide 2 — fix the geometry: measure policy distance the right way with the natural gradient.
  2. Guide 3 — TRPO turns the guarantee into a hard trust-region constraint you can actually solve.
  3. Guide 4 — PPO throws out the hard solver and clips the ratio instead; it became the default.
  4. Guide 5 — scaling, off-policy correction, and how PPO ended up at the heart of RLHF.