gradient temporal-difference learning

Ordinary TD with function approximation can diverge off-policy (estimates blow up). Gradient TD methods fix this by being honest about what they optimize: they perform true gradient descent on a well-defined error objective, so they are guaranteed to converge even when linear approximation, off-policy data, and bootstrapping are combined — the very setting that breaks plain TD.

They minimize the mean-squared projected Bellman error (MSPBE) by stochastic gradient descent. The catch is that its gradient contains a product of two expectations over the same next state, which a single sample cannot estimate without bias. GTD2 and TDC solve this with a second set of weights that tracks one of those expectations (a two-timescale trick), giving a sound stochastic gradient.

The price is a second learning rate to tune and somewhat slower learning than plain TD when plain TD happens to work. They are the principled answer to the deadly triad, but in deep RL practitioners often prefer cheaper stabilizers (target networks, replay) that work well enough in practice.