semi-gradient methods

The standard way TD learning is actually done with function approximation — and the reason it is only semi-gradient. When you update toward a bootstrapped target like r + gamma v(s'; w), that target itself depends on the weights w. A true gradient method would differentiate through it too. Semi-gradient methods deliberately ignore that dependence: they treat the target as a fixed number and take only the gradient of the prediction.

The update is w <- w + alpha[r + gamma v(s'; w) - v(s; w)] grad v(s; w), with no gradient through the target. Skipping the target's gradient is what keeps each step a single cheap feature-vector update, so this is cheaper, learns faster, and is what semi-gradient TD(0), Sarsa, and DQN all use. On-policy with linear features it converges to the TD fixed point; off-policy it can diverge — and that missing gradient is exactly what the deadly triad exploits.

Semi-gradient is not a bug to be fixed casually: the full-gradient alternative (residual gradient) is provably stable but in practice often learns slowly and converges to a worse, blurrier solution, so the cure can be worse than the disease. Semi-gradient remains the pragmatic default that powers most working systems; gradient TD is the principled repair you reach for only when the deadly triad actually makes your values diverge.