adaptive KL penalty

When you enforce the trust region with a penalty rather than a hard constraint, you face a tuning headache: how big should the KL coefficient be? Set it too high and the policy barely moves; too low and it lunges past the trustworthy region. The trouble is that the right value drifts during training. An adaptive KL penalty sidesteps this by not fixing the coefficient at all — instead it picks a target KL per update and lets a simple controller adjust the coefficient to hit it.

The rule is a gentle feedback loop. After each update, measure the realized KL between old and new policy. If it came out well above the target, the steps are too aggressive, so raise the coefficient to clamp down next time; if it came out well below, you are being timid, so lower the coefficient to allow larger moves. This keeps the divergence near the desired budget despite changing loss landscapes, and it is the standard way RLHF holds a language model a controlled distance from its reference model.