KL-constrained policy update

When you improve a policy from data the old policy collected, your estimate is only trustworthy near that old policy. A KL-constrained update enforces this directly: change the policy as much as you like to raise return, but keep the Kullback–Leibler divergence between the old and new action distributions below a budget. KL is a natural ruler here because it measures how differently the two policies behave on the states you actually visited, not just how far their parameters moved.

There are two ways to enforce the budget. A hard constraint, as in TRPO, solves a constrained optimization so the average KL never exceeds the threshold. A soft constraint adds a penalty term, subtracting a coefficient times the KL from the objective, so larger divergences are discouraged but not forbidden. Either way, the KL leash is what turns a raw, high-variance policy gradient into stable, sample-efficient progress, and the same idea reappears in RLHF, where staying close to the reference model prevents reward hacking.