policy iteration

Policy iteration alternates the two moves you now have — measure, then improve — until they stop changing anything. Evaluate the current policy fully, make it greedy with respect to those values, evaluate the new policy fully, and repeat. Each round yields a strictly better policy until no improvement is possible, at which point you hold the optimal one.

It is a two-loop algorithm: an inner loop runs policy evaluation to convergence, an outer loop runs one policy-improvement step. Because there are finitely many deterministic policies and each iteration is strictly better (or equal, which signals termination), policy iteration reaches the optimal policy in a finite number of outer steps — often surprisingly few, even on large state spaces.