Gradient descent on the value error
Once value is a parameterized function, learning is optimization: shrink the value error by nudging w downhill along its gradient. For one state with target U, the squared error is (U − v̂(s,w))², and stochastic gradient descent updates w by a step proportional to (U − v̂(s,w)) times the gradient of v̂. For linear methods the gradient of v̂ is just the feature vector x(s), so the update is wonderfully simple: w ← w + α·(U − wᵀx(s))·x(s).
If the target U is a Monte Carlo return — an actual sampled total reward — then U does not depend on w, this is true gradient descent, and it converges to the best least-squares fit. The trouble starts the moment we make U cheaper by bootstrapping.
An interactive ball rolling down a curved loss surface to its minimum as gradient-descent steps are applied.
Bootstrapping breaks the gradient
Temporal-difference learning uses a cheaper, lower-variance target: the immediate reward plus the discounted estimate of the next state, U = r + γ·v̂(s′,w). This is bootstrapping — the target is built from our own current estimate. But now U contains w. The honest gradient of the squared error would have to differentiate through both v̂(s,w) and v̂(s′,w).
Semi-gradient methods cheat — deliberately. They treat the bootstrapped target as a fixed constant and ignore its dependence on w, differentiating only through v̂(s,w). The update keeps the simple form above, with U = r + γ·v̂(s′,w). It is not a real gradient of any objective, which is why it is 'semi' — but on-policy with linear features it converges, it is fast, and it is what plain TD and Sarsa actually do.
The semi-gradient TD(0) update: the bracketed TD error scales the gradient of v̂ alone, ignoring the target's dependence on w.
Where semi-gradient TD lands: the TD fixed point
Semi-gradient TD does not minimize the value error. It converges to a different solution called the TD fixed point, best understood through projection onto value space. Picture all value functions your features can represent as a flat plane inside the huge space of all possible value functions. The true value usually sits off that plane. A Bellman backup pushes a value off the plane (toward truth); projection snaps it back onto the nearest representable point. The TD fixed point is where backing-up-then-projecting leaves the value unchanged.
Solving for the fixed point directly: LSTD
If the TD fixed point of a linear system is what we want, why crawl there by small gradient steps? Least-squares temporal difference (LSTD) computes it in closed form. It accumulates two statistics over the data — a matrix and a vector summarizing the features and TD targets — then solves a single linear system for the weights that satisfy the fixed-point equation. No step size, no thousands of passes; it is the most data-efficient linear TD method.
The price is compute and memory: building and inverting that matrix costs on the order of the square (and the solve, the cube) of the number of features, so LSTD shines with hundreds of features but not millions. It is the batch counterpart to semi-gradient TD and the springboard for the fitted methods in guide 5.
The honest alternative: residual gradients
If semi-gradient methods bother you for ignoring part of the gradient, residual gradient methods do the honest thing: differentiate through both v̂(s,w) and the bootstrapped v̂(s′,w), giving a true gradient of the mean squared Bellman error. Because it is a genuine gradient, it is guaranteed to converge even off-policy — no cheating, no divergence.
The residual-gradient update keeps the extra −γ∇v̂(s′,w) term, honestly differentiating through the bootstrapped target too.
So why isn't everyone using it? It is often slower and biased in stochastic environments — minimizing Bellman error tends to learn an over-smoothed value, and it needs two independent samples of the next state to be unbiased. Residual gradients trade speed and accuracy for safety. Holding all three — speed, off-policy safety, and a true objective — together is the hard problem we attack next via the deadly triad.