PPO for RLHF

PPO is the reinforcement-learning algorithm most associated with RLHF. After you have a reward model, you need to actually change the language model so it produces responses the reward model likes. PPO does this by sampling responses from the current model, scoring them, and nudging the model to make high-scoring responses more likely and low-scoring ones less likely.

The proximal part is the key safety feature. Each update is clipped so the new policy cannot move too far from the old one in a single step, which keeps a noisy reward signal from blowing up the model. On top of that, RLHF adds a KL penalty to the original fine-tuned model, so the policy is pulled toward reward but tethered to a sensible starting point. In effect the model treats its own previous answers as a trust region.

PPO works but is heavy: it juggles four models at once (policy, reference, reward, and value) and is finicky to tune. That cost is a big reason simpler alternatives like DPO became popular.