Dyna architecture

Dyna is the classic recipe for getting the best of both worlds: learn from real experience and from imagined experience at the same time. Every time the agent takes a real step, it does three things — it updates its value estimates from that real transition, it updates its model of the world from it, and then it spends spare moments dreaming, generating fake transitions from the model and learning from those too.

Sutton's original Dyna-Q wraps ordinary Q-learning in this loop. After each genuine interaction it runs several planning steps, each picking a previously seen state-action pair, asking the model what reward and next state to expect, and applying the same Q-learning update as if that imagined experience were real. The model and the value function improve together, and the planning updates propagate information far faster than waiting for real steps alone.

Dyna's beauty is its simplicity and its clean message: planning, acting, and model learning are the same kind of value update fed by different sources of experience. Its weakness is inherited from its model — replaying transitions from a model that has drifted out of date can teach the agent things that are no longer true.