value iteration

Policy iteration fully evaluates each policy before improving it, which can be wasteful. Value iteration takes a shortcut: instead of running evaluation to convergence, it does just one backup per state and folds the improvement step right into it. During the loop you never store a policy at all — you simply keep pushing the values toward their optimal level.

Each sweep applies the Bellman optimality backup: a state's new value is the best, over actions, of expected reward plus discounted next-state value. That single max is policy evaluation and improvement merged into one. The iterates converge to the optimal value function, after which a single greedy pass extracts the optimal policy. It is simpler to code than policy iteration and often a good default.