KL penalty to the reference policy

A freshly pretrained model already knows how to write fluent, sensible text. The moment you start optimising it for reward, there is a real danger it forgets all of that and contorts itself into whatever strange output the reward model happens to score highly. The KL penalty is a leash: it measures how far the policy has drifted from the original model and charges a price for drifting, keeping the new behaviour anchored to the old competence.

At each step you add to the reward a penalty proportional to the Kullback-Leibler divergence between the current policy and a frozen reference — usually the supervised-fine-tuned model you started from. The combined objective is 'maximise reward minus β times KL from the reference'. A large β keeps the policy timid and close to the reference; a small β lets it chase reward aggressively. In token-level RLHF this often appears folded directly into a per-token reward.

The KL penalty is the main defence against reward overoptimisation: it physically limits how far the policy can travel to exploit the reward model's blind spots. But it is a blunt instrument. Set β too strong and the model barely improves; too weak and it hacks the reward and degenerates into fluent-sounding junk. Tuning that one coefficient is among the central balancing acts of the whole method.