background vs decision-time planning

There are two moments at which a model can help you, and they feel quite different. Background planning happens in the agent's spare time, away from any particular decision: it uses the model to keep improving a value function or policy, so that when a real situation arrives the answer is already baked in. Decision-time planning happens right when you must act: facing this exact state, you fire up the model and search for the best move here and now.

Dyna is the archetype of background planning — between real steps it replays model-generated transitions to polish its Q-values, and at action time it just looks up the policy, fast. Monte Carlo tree search and MPC are archetypes of decision-time planning — they do little or nothing in advance but compute hard at the moment of choice, building a fresh local lookahead for the state in front of them. Many strong agents blend both: a learned policy gives a good default that a decision-time search then sharpens.

The trade-off is when you pay the compute. Background planning makes acting cheap and reactions instant but bakes in whatever the policy generalised; decision-time planning adapts precisely to the current state and a changing model, at the cost of thinking time before every move.