reward over-optimization

Over-optimization is what happens when you push the reward signal too hard and the model starts to get worse even as its measured reward keeps climbing. It is Goodhart's law in action: once a proxy measure becomes the target, it stops being a good measure. The reward model is that proxy, and squeezing the last drops of score out of it drifts the policy into regions where the proxy and real human preference part ways.

Empirically, true quality often rises with optimization for a while, peaks, then declines, even as reward-model score keeps rising — a telltale divergence between the proxy and the real goal. The KL penalty is the main defence: keeping the policy near the reference limits how far it can chase the proxy into nonsense, so tuning the KL coefficient is partly about choosing where to stop.

Over-optimization is why more RL is not always better and why bigger reward models, ensembles, and early stopping all matter. The honest framing is that you are optimizing an approximation, and approximations are only trustworthy within range.