policy improvement

Once you know how good each state is under your current policy, you can ask a sharper question at every state: is there a single action that looks better than what my policy does there? If so, switch to it. Acting greedily on your own value estimates is guaranteed to give a policy at least as good — and usually strictly better.

For each state you compute the action-value, the expected reward plus discounted next-state value, and make the new policy pick the action with the highest one. The policy improvement theorem proves this greedy policy is never worse than the old one; and if it turns out identical, you have already found an optimal policy. This single step is the engine that turns measurement into progress.