asynchronous dynamic programming

Classic DP finishes a full pass over every state before reusing any new value — a "synchronous" sweep. But if a million states are irrelevant to where you actually are, that is wasteful. Asynchronous DP drops the requirement of full, ordered sweeps: you may update states in any order you like, one at a time, revisiting important ones often and ignoring others for a while.

You keep a single value table and back up individual states as you choose, immediately using whatever values are current. Convergence to the optimal value function still holds, as long as every state keeps getting selected infinitely often in the limit. This freedom lets you focus computation where it matters — for instance along the trajectories the agent actually visits — and is what makes DP ideas usable inside online, real-time learning.