Rewards, Goals, and the Long Game

One bold claim: every goal is a reward

Reinforcement learning rests on a strikingly simple bet, the reward hypothesis: everything we mean by "a goal" can be captured as maximising the expected total of a single scalar reward. Win the game, keep the robot upright, write a helpful answer — each becomes a number to be made as large as possible. If true, it means we don't need a separate theory for every task; we need one theory of reward maximisation.

It's the total that matters, not the next step

Here is the idea that separates RL from short-sighted reflexes. The agent does not chase the biggest reward right now; it chases the biggest reward added up over the whole future. This is the cumulative reward intuition. A chess player happily sacrifices a piece — a small negative now — to win the game later — a large positive. A studying student forgoes an evening of fun for a result that pays off for years.

The return G_t adds up every reward from the next step to the end of the episode — the total the agent truly maximises.

So when we say an action is "good," we never mean it in isolation. We mean it leads to a high total down the line. This is exactly what gives rise to genuine goal-directed behaviour: the agent learns to accept small costs in service of a larger payoff, because the thing it optimises is the sum, not the moment.

When the reward comes late

Caring about the total raises a hard practical problem: many rewards arrive long after the actions that earned them. This is the challenge of the delayed reward. In a board game the only clear reward — win or lose — comes at the very end, after dozens of moves. A robot's reward for assembling a part comes only when the part is finished. The decisive choices and the payoff can be separated by a huge gap of time.

That gap is what makes RL genuinely hard. If reward followed instantly after every action, learning would be almost trivial — just keep whatever felt good. The art of RL is propagating a reward that lands at step 100 back to the quiet, unglamorous step 7 that actually made it possible.

The credit assignment problem

That "propagating reward backwards" puzzle has a name: the credit assignment problem. When a long sequence of actions ends in success or failure, which actions deserve the credit (or the blame)? The last move before winning? The clever setup ten moves earlier? Some of each? Untangling this is one of the central problems the entire field exists to solve.

1. Picture a 40-move game that you win. The final +1 reward is real, but it doesn't say which move won it.
2. Naively crediting only the last move is wrong — the win was set up much earlier.
3. Naively crediting every move equally is also wrong — some moves were neutral or even mildly bad.
4. Good RL methods spread credit cleverly across time, leaning on the moves that genuinely shifted the odds.