prioritized sweeping

Not every state is equally worth updating. If a value barely changed last time, backing it up again is almost pointless; if a big surprise just rippled in, its neighbours urgently need refreshing. Prioritized sweeping spends effort where it pays off most, always updating the state whose value is currently most wrong before any other.

You keep a priority queue keyed by each state's pending Bellman error — how much a backup would change it. Pop the top state, back it up, then look at its predecessors, the states that lead into it, and push them onto the queue with priority set by how much this change affects them. Effort concentrates around recent changes, so prioritized sweeping can reach a good value function with far fewer backups than uniform sweeps, especially in large or sparsely connected problems.