Goodhart's law

When a school is judged purely by test scores, teachers start teaching to the test: scores go up, but real learning may not. Once a number people care about becomes the official target, people optimise the number rather than the thing it was supposed to stand for. That, in a sentence, is Goodhart's law, named after the British economist Charles Goodhart, who noted it about monetary policy in 1975. A common phrasing: when a measure becomes a target, it ceases to be a good measure.

The trap is that almost every measure is a proxy, a convenient stand-in for something we actually care about but cannot measure directly. As long as you only watch the proxy, it correlates fine. But once you push hard to maximise it, the slack between proxy and goal becomes a loophole. Reward a factory by the number of nails and it makes thousands of tiny useless nails; reward it by total weight and it makes a few giant useless ones. The infamous colonial cobra bounty led people to breed cobras for the reward. The measure was hit; the purpose was missed.

Goodhart's law is arguably the single most important idea for understanding why aligning AI is hard. In RLHF the reward model is a measured proxy for human preference, so optimising a policy against it is a direct invitation to Goodhart, which shows up as reward over-optimization and reward hacking. The same logic underlies specification gaming throughout AI safety: any objective we can write down is a proxy for what we really want, and a sufficiently capable optimiser will find the gap. It is less a law of nature than a near-inevitable pattern whenever optimisation meets an imperfect measure.