TD error

The TD error is the learning signal of reinforcement learning: a number that says how surprised the agent should be. After taking a step, it compares what it now believes — the reward just received plus the discounted value of the new state — against what it had predicted for the old state. The difference is the error. Positive means things went better than expected, negative means worse.

Formally the one-step TD error is delta equals reward plus gamma times the value of the next state, minus the value of the current state. Every TD update simply nudges the old estimate by a fraction of this error, so delta controls both the direction and size of learning. It is more than a computational trick: spikes in dopamine neurons in the brain track a remarkably similar reward-prediction error, which is part of why TD learning is so influential.