monotonic improvement guarantee

The dream of policy optimization is an update rule that never makes things worse — each new policy is provably at least as good as the last. The key result behind TRPO supplies exactly this in principle. It proves that the true return of the new policy is at least the surrogate objective minus a penalty proportional to the maximum KL divergence between the old and new policy. The surrogate is a lower bound that touches the true objective at the current policy.

Because that bound lies below the truth and meets it at the current point, maximizing the bound can only push the real return up: this is a minorization-maximization scheme, the same logic as the EM algorithm. Improve the lower bound, and the actual performance follows. In practice the theoretical penalty coefficient is so conservative that strictly following it would take microscopic steps, so TRPO replaces the penalty with a KL trust region and PPO with a clip — trading the airtight guarantee for steps large enough to learn in reasonable time.