proximal policy optimization (PPO)

PPO is TRPO's pragmatic younger sibling: it wants the same safety — never stray too far from the policy that produced your data — but without solving a constrained second-order problem every step. Instead it bakes the caution directly into the loss. You compute the same probability ratio of new to old action probabilities, multiply by the advantage, but then clip the ratio so that pushing it far past one no longer helps the objective. The optimizer simply has no incentive to take a reckless step.

Because the clipped loss is a plain function you can optimize with ordinary stochastic gradient descent, PPO runs several epochs of minibatch updates on each batch of collected experience, then throws the data away and collects fresh rollouts. This combination — a clipped surrogate, multiple cheap update passes, and generalized advantage estimation for the targets — makes PPO simple, robust to hyperparameters, and parallel-friendly. It became the default on-policy algorithm in deep RL and the workhorse behind RLHF for language models.