PPO clip vs. penalty

PPO was introduced in two flavours that chase the same goal — keep the new policy close to the old — by different means. PPO-Clip enforces the leash implicitly: it clips the probability ratio so the surrogate stops rewarding moves beyond a band around one, and there is no explicit KL term at all. PPO-Penalty enforces it explicitly: it subtracts a coefficient times the KL divergence from the objective and then adapts that coefficient up or down to hit a target KL.

In the original experiments the clip variant generally matched or beat the penalty variant while being simpler — one fewer thing to tune, no running KL coefficient — so it became the default everyone means by PPO. The penalty variant remains useful when you want to control the divergence precisely rather than just bound it, for example in RLHF, where holding a chosen KL to the reference model trades reward against staying faithful to the base policy. Some systems even combine a clip with an explicit KL penalty for extra stability.