temporal-difference learning

Picture driving to an unfamiliar place. You do not wait until you arrive to judge your earlier guesses about the trip; every block, you nudge your estimated arrival time using what just happened. Temporal-difference learning works the same way: it updates the value of a state immediately, moving the old estimate a little toward a fresher one made one step later, without waiting for the episode to end.

TD is the celebrated blend of two worlds. Like Monte Carlo, it learns directly from experience with no model of the environment; like dynamic programming, it bootstraps — it builds new estimates on top of its own current estimates rather than on full returns. This lets it learn online, step by step, and on tasks that never terminate. The cost is that the target it chases is itself only a guess, so TD estimates are biased, though usually far less noisy than Monte Carlo's.