From TRPO's pain to PPO's simplicity
Proximal Policy Optimization (PPO) asks: can we get TRPO's stability with nothing but first-order optimization — plain SGD on minibatches, no conjugate gradient, no line search? The answer is a clever reshaping of the surrogate objective so that it self-limits the step size. You optimize it with the same Adam you use for everything else, and you can run multiple epochs over each batch of data.
Gradient descent stepping down a loss curve toward the minimum, standing in for PPO's plain first-order optimization.
The clipped surrogate
Let r be the importance ratio — new-policy probability over old — for a sampled action, and A its advantage. The clipped surrogate objective takes the minimum of two terms: the usual r·A, and a version where r is clipped to the band [1−ε, 1+ε] before multiplying by A. The min picks whichever is smaller (more pessimistic).
The clipped surrogate objective: a min over the raw and clipped ratio-times-advantage terms.
The genius is in the asymmetry. When an action is good (A positive) and the ratio rises past 1+ε, clipping flattens the objective — there is no more reward for pushing that action's probability even higher, so the gradient vanishes and the policy stops moving. When an action is bad (A negative), the min lets the unclipped term dominate so the policy can still strongly suppress it. The effect mimics a trust region: the update is free to fix mistakes but reined in from over-committing to a single good sample.
Clip versus penalty
PPO actually shipped in two flavors, and knowing the clip-versus-penalty distinction matters. The clip variant is the one above. The penalty variant instead adds a KL term to the objective and scales it with an adaptive KL penalty: after each update, if the measured KL overshot a target, raise the coefficient; if it undershot, lower it. This is the direct softened cousin of TRPO's hard constraint and the mirror-descent template from Guide 2.
The clip variant won the popularity contest for control tasks because it needs no KL target tuning and is dead simple. The penalty variant, however, is the one that resurfaced in language-model alignment, where a controlled KL to a reference policy is the whole point — you will meet it again in Guide 5.
RLHF pipeline: human preferences train a reward model that tunes the policy.
The details that actually matter
PPO's reputation hides an uncomfortable truth: a huge share of its measured performance comes not from the clipped objective but from a stack of unglamorous implementation details. Studies that ablate these found they can matter as much as the core algorithm. Treat them as part of the method, not optional polish.
- Normalize advantages per minibatch (zero mean, unit variance) to stabilize the gradient scale.
- Clip the value-function loss the same way, and weight it into a shared actor-critic loss.
- Add an entropy bonus to keep exploration alive and prevent premature collapse.
- Anneal the learning rate, clip the global gradient norm, and run several epochs per batch.
- Use orthogonal weight initialization and reward/observation normalization.
# One PPO update over a collected batch
for epoch in range(K):
for mb in minibatches(batch):
ratio = exp(logp_new(mb.a, mb.s) - mb.logp_old)
adv = normalize(mb.advantage)
unclipped = ratio * adv
clipped = clip(ratio, 1 - eps, 1 + eps) * adv
policy_loss = -mean(min(unclipped, clipped))
value_loss = mse(value(mb.s), mb.returns)
loss = policy_loss + c1 * value_loss - c2 * entropy(mb.s)
adam_step(loss) # plain first-order; no CG, no line search