RL fine-tuning

A model that has already learned language and been taught to follow instructions is competent but unpolished, like a new hire who knows the job but not yet the house style. RL fine-tuning is the coaching phase that polishes it: instead of showing the model correct answers to imitate, you let it generate its own answers, score them, and adjust it to make high-scoring answers more likely. It is the 'RL' in RLHF, the stage where a reward model's scores are actually turned into changes in the model.

Here is the mechanism in this setting. The language model is treated as a policy: given a prompt, producing each word is an action, and a finished answer earns a reward from the reward model. A reinforcement-learning algorithm, most commonly PPO (proximal policy optimization), then updates the model's weights to increase expected reward, the policy-gradient machinery itself belongs to the reinforcement learning domain, so cross-link there rather than re-deriving it. Crucially, a KL-divergence penalty is added so the policy cannot stray too far from the original base model, keeping the text fluent and resisting reward hacking.

RL fine-tuning is what gives modern assistants their characteristic helpful, well-formatted behaviour, but its honest limits are exactly those of RLHF as a whole. Because it optimises a proxy reward, it is prone to reward over-optimization and reward hacking, which the KL penalty only contains; and it shapes the model's outward behaviour rather than installing genuine goals. It is also not the only option, simpler alternatives like direct preference optimization (no explicit RL loop) and best-of-n sampling (no weight updates at all) target the same preferences by different means.