PPO for preference optimization

Once you have a reward model that scores behaviours, you still need to improve the policy so it earns high scores — and that is a reinforcement learning problem. Proximal policy optimisation (PPO) is the algorithm most often chosen for the job. The agent generates a behaviour, the reward model scores it, and PPO nudges the policy to make high-scoring behaviours more likely, taking small, careful steps so it never lurches off into nonsense.

In the language-model setting the 'environment' is text generation: the policy produces a response token by token, the reward model gives a scalar reward (usually only at the end), and PPO performs a clipped policy-gradient update using that reward plus a per-token KL penalty against the frozen starting model. The clipping keeps each update inside a trust region, so the policy cannot move too far on a single noisy batch — crucial when the reward signal is itself only an imperfect model.

PPO-based RLHF is powerful but finicky. It has many moving parts — a value head, the KL coefficient, reward normalisation, the clipping range — and is sensitive to all of them, so reproducing a good run takes care. That very complexity is why simpler alternatives like direct preference optimisation were invented; yet PPO remains the most flexible recipe, and the only one of the common choices that can genuinely explore for new high-reward behaviour.