Learning a Guess from a Guess: Temporal-Difference Learning

The best of both worlds

Temporal-difference (TD) learning is, in Richard Sutton's words, the central and novel idea of reinforcement learning. It sits exactly between the two methods you've met. Like Monte Carlo, it learns directly from raw experience with no model of the environment. Like dynamic programming, it updates an estimate using other estimates rather than waiting for a final outcome — it bootstraps.

That combination is what lets TD learn online, after every single step, without ever finishing an episode. For long or continuing tasks this is transformative.

The TD(0) update

Recall the Monte Carlo update moves the estimate toward the full return G: V(s) ← V(s) + α·(G − V(s)). The problem is that G is only known at the end of the episode. The simplest temporal-difference method, TD(0), replaces that full return with a one-step estimate of it: the reward you just received plus your current guess for the next state's value.

# TD(0) update, applied immediately after each transition (s, r, s')
target   = r + gamma * V[s_next]      # the TD target (a guess!)
td_error = target - V[s]              # the TD error, delta
V[s]    += alpha * td_error           # nudge V[s] toward the target

The quantity r + γ·V(s′) is the TD target, and the difference between it and the current estimate is the TD error δ = r + γ·V(s′) − V(s). The TD error is the workhorse of this whole field: it is a moment-to-moment measure of surprise — how much better or worse things turned out than you expected one step ago.

The TD error δ and the one-step, online TD(0) update it drives.

Sampling versus bootstrapping

It helps to name the two independent design choices behind every value-learning method, the theme of sampling versus bootstrapping. Sampling means you use actual experienced transitions instead of summing over all possible next states with a model. Bootstrapping means your update target contains your own current estimate of a future value rather than a real measured return.

With this vocabulary the methods line up cleanly. Dynamic programming bootstraps but does not sample (it needs the model). Monte Carlo samples but does not bootstrap (it waits for the real return). TD does both at once — it samples the next transition and bootstraps off V(s′). That double move is what makes it powerful and, as the next section warns, slightly delicate.

TD vs MC: the bias–variance trade-off

Because the MC target is a full return — a sum of many random rewards along a whole trajectory — it has high variance but no bias. The TD(0) target depends on only one random reward plus a deterministic lookup V(s′), so it has much lower variance, but it is biased: V(s′) is currently wrong, and you are learning toward a wrong number. As learning proceeds and V(s′) improves, the bias shrinks.

Why the TD error is such a useful signal

The TD error δ does more than drive one update. It is a local, online learning signal that can be computed the instant a transition happens, propagated to many states (you'll see this with eligibility traces), and even used to adjust policies, not just values. Strikingly, the firing of dopamine neurons in the brain closely matches a TD-error signal — one of the most celebrated bridges between machine learning and neuroscience.

So far MC uses the whole return (n steps to the end) and TD(0) uses exactly one step. Those are two ends of a spectrum. The next guide fills in everything between them with n-step returns and the elegant TD(λ).