residual gradient methods

The honest alternative to semi-gradient TD. Define the Bellman error as the mismatch between a value and its own one-step backup, then do true gradient descent to minimize the squared error — differentiating through both the prediction and the bootstrapped target. Because it is a genuine gradient on a real objective, it is guaranteed to converge, even off-policy.

It minimizes the mean-squared Bellman error, the expectation of (r + gamma v(s') - v(s))^2, taking the gradient through the gamma v(s') term that semi-gradient ignores. Stability comes at a cost: the objective requires the gradient of two independent samples of the next state (the double-sampling problem), and minimizing Bellman error tends to converge to a worse, blurrier solution than the projected (TD) fixed point.

So residual gradient is stable but often slow and biased toward smearing values; semi-gradient is fast but can diverge; gradient TD (MSPBE) was invented to get the best of both — convergence guarantees while still targeting the better TD fixed point. Knowing which error you minimize (Bellman versus projected Bellman) is the key distinction.