contraction property of the Bellman operator

Why does hammering the same backup over and over actually settle on an answer instead of wandering forever? Because the Bellman operator is a contraction: every time you apply it, the gap between any two value estimates shrinks by at least a fixed factor. Errors can only get smaller, so the process is pulled inexorably toward a single resting point.

Measured in the max-norm — the largest per-state difference — applying the operator multiplies the distance between any two value functions by at most the discount factor, which is below one. By the Banach fixed-point theorem this guarantees a unique fixed point and geometric convergence to it: the error after k sweeps is at most that factor raised to k, times the starting error. This one fact underpins the correctness of all DP.