the KL-divergence penalty
Picture training a talented but impressionable apprentice by paying a bonus for good work. Reward the bonus too aggressively and the apprentice may start doing weird things that technically earn the bonus but abandon all their good training. So you add a rule: you may chase the bonus, but you must not stray too far from how you normally work. The KL-divergence penalty is that rule for RLHF, a leash that keeps the model close to its original behaviour while it learns to earn reward.
KL divergence is a standard measure of how different one probability distribution is from another. A language model, at each step, is a probability distribution over the next word. During the RL step we add to the reward a term that subtracts beta times the KL divergence between the policy being trained and the frozen reference (the original base model). In plain terms: every time the new model's word-choices drift away from the original's, it pays a price. The knob beta sets how tight the leash is, small beta lets the model change a lot, large beta keeps it almost identical to the base.
This penalty is doing real safety and quality work. Without it, the model would happily contort itself into whatever maximises the reward model's score, producing degenerate, repetitive, or gibberish text that the imperfect reward model rates highly, the classic face of reward over-optimization. The KL leash keeps outputs fluent and on-distribution and slows down reward hacking. But it is a containment tool, not a cure: it limits how far behaviour drifts, yet it does not fix a flawed reward model and does not touch the model's underlying goals.
Drop the KL penalty and a policy might learn that ending every answer with 'Great question!!!' nudges the reward model higher, collapsing into that tic; the penalty makes such drift from normal language too costly to be worth it.
The KL penalty is a leash: chase reward, but don't wander far from the base model.
The KL penalty limits how far behaviour drifts; it does not fix a flawed reward model. Too small a beta invites reward hacking, too large a beta and the model barely improves.