projection onto value space

A geometric way to see what approximate TD really computes. The true value function lives in a huge space (one dimension per state); your approximator can only represent a small slice of it — the set of value functions expressible by your features, a low-dimensional subspace. Whenever the Bellman backup pushes you out of that subspace, you must come back by projecting onto the closest representable function.

Let the projection operator (in the on-policy state-weighted L2 norm) onto the representable subspace be written as a map, and let T applied to a policy be the Bellman operator. Semi-gradient TD converges to the fixed point of the composed map project-then-back-up — the value that is unchanged by back up, then project. Gradient TD methods minimize the distance between the backed-up value and its projection, the projected Bellman error, directly. The picture explains why TD's solution differs from the best possible fit.

The subtlety is that the Bellman operator contracts in the max-norm while the projection contracts in a weighted L2 norm; their composition is a contraction on-policy (so TD converges) but not necessarily off-policy (so it can diverge). This single mismatch underlies the deadly triad and most approximate-RL theory.