iterative policy evaluation

Solving the value equations exactly means inverting a large matrix, which is impractical when there are many states. Iterative policy evaluation reaches the same answer cheaply by starting from a rough guess and refining it. You repeatedly replace each state's value with a better estimate built from its neighbours' current values — like word of "how good things are" spreading one step at a time across the state space.

Each sweep applies the Bellman expectation backup to every state: the new value is the expected immediate reward plus the discounted expected value of the next state under the current estimates. Because this backup is a contraction, every sweep shrinks the error toward zero and the iterates converge to the policy's true value, whatever the starting guess. You stop when the largest change in a sweep falls below a small threshold.