importance ratio clipping

When you reuse data collected by one policy to evaluate or improve a different policy, you must reweight each sample by the importance ratio: how much more or less likely the new policy is to have taken that action. The trouble is that this ratio can explode. If the new policy strongly prefers an action the behaviour policy rarely tried, a single rare sample gets multiplied by a huge weight, and its noise swamps everything else, making updates wildly unstable.

Clipping caps the ratio at a ceiling, trading a little bias for a large reduction in variance. The capped estimator no longer corrects perfectly for the off-policy mismatch, but it stops any one transition from dominating the batch, which keeps learning stable far from the data distribution. This single idea underlies PPO's clipped objective, V-trace's truncated correction in distributed RL, and many off-policy actor-critic methods that must learn from stale or shared experience.