generalized policy iteration

Policy iteration and value iteration look like different algorithms, but they share one idea: keep a value estimate and a policy, and let each nudge the other toward consistency. Generalized policy iteration is the name for that idea in its loosest form — you need not finish evaluation before improving, nor improve everywhere at once. Almost every RL method is some flavour of this dance.

Two processes run together. Evaluation makes the value function more accurate for the current policy; improvement makes the policy greedier for the current value function. They pull in tension — turning the policy greedy spoils the value estimate, and re-evaluating un-greedies the policy — yet their only shared resting point is optimality, where the policy is greedy with respect to its own correct values. As long as both processes keep making progress, the pair converges there.