surrogate objective

What we truly want to maximize — the expected return of the new policy — is hard to optimize directly, because evaluating it would require running the new policy to collect fresh data on every candidate update. A surrogate objective is a stand-in we can actually compute from the data we already have. It uses importance sampling to estimate the new policy's expected advantage under the old policy's state visits, giving a function whose gradient at the current policy matches the true policy gradient.

The catch is that this stand-in is only faithful nearby; the farther the new policy drifts from the one that gathered the data, the more the importance weights distort and the estimate diverges from real return. So every advanced policy method pairs the surrogate with a leash — a KL constraint, a clip, or a penalty — that keeps the update inside the region where the surrogate is trustworthy. Optimize the surrogate locally, re-collect data, repeat: that loop is the backbone of TRPO and PPO.