KL penalty (in alignment)

The KL penalty is the leash that keeps an aligned model from running off. When you optimize a model to chase reward, it can find weird, degenerate ways to score high — repeating flattering phrases, adopting a strange style, or collapsing its diversity. The KL penalty measures how far the new model's output distribution has drifted from a trusted reference, usually the fine-tuned starting model, and adds a cost proportional to that drift.

Mechanically it appears as a term subtracted from the reward, scaled by a coefficient beta. A small beta lets the model move freely and chase reward hard, at the risk of going off the rails; a large beta keeps it close to the reference and well-behaved, at the cost of learning less. Tuning beta is one of the central knobs of RLHF, and the same anchoring idea is built directly into DPO-style losses.

The deeper reason it works: the reference model encodes a huge amount of fluent, sensible behavior, and staying near it preserves that competence while the reward gently reshapes the parts that matter for alignment.