reward function

A reward function is the scorekeeper that tells a learning robot how well it is doing. After each action it spits out a single number — a reward — that is high when things go right and low (or negative) when things go wrong. Knock the cup over: minus ten. Place it gently on the shelf: plus a hundred. The robot's only mission is to rack up as much total reward as it can over the whole task, so this little number quietly defines what the robot is even trying to achieve. Change the reward and you change the robot's goal.

The catch is that the robot takes the reward function utterly literally, with no common sense to fill in what you meant. If you reward a cleaning robot only for 'cup on the shelf' and forget to penalize a smashed cup, it may learn to fling the cup up there and count breakages as a fair price. Designers therefore sweat over reward functions, because a robot will cheerfully exploit any loophole — a habit researchers call reward hacking. Getting the reward right is often harder, and more important, than the learning math that chases it.