JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Rainbow and Recurrent Replay: Assembling the Foundations

Six improvements, one agent — then add memory for partial observability. This is the state-of-practice value-based foundation, and a launchpad to what's next.

Rainbow: combine the improvements

By now you have met a pile of independent upgrades to plain DQN. Rainbow is the observation that they are mostly orthogonal and can be combined into a single agent — and that doing so beats every one of them alone.

  1. Double Q-learning — removes overestimation bias.
  2. Dueling architecture — separate value and advantage streams.
  3. Prioritized replay — rehearse high-TD-error transitions.
  4. Multi-step (n-step) returns — bootstrap from several steps ahead.
  5. Distributional RL (C51) — learn the return distribution.
  6. Noisy nets — learned, state-dependent exploration.
Q(s,a) \leftarrow Q(s,a) + \alpha\,\underbrace{\big( r + \gamma\,\max_{a'} Q(s',a') - Q(s,a) \big)}_{\text{TD error }\delta}

Every Rainbow ingredient reshapes this one temporal-difference update — the max, the return, or the bootstrap target.

What each ingredient really contributes

The Rainbow paper ran an ablation: remove one component at a time and watch the score drop. The biggest hits came from prioritized replay, multi-step returns, and the distributional head; Double Q-learning and dueling helped less on top of the others (their job partly overlaps with the distributional view).

Partial observability & recurrent replay

Frame stacking handles short-term memory, but four frames can't remember an event from ten seconds ago. Many real tasks are truly a partially observable MDP (POMDP): the optimal action depends on history, not just the current observation. The principled fix is to maintain a belief state — a running summary of everything seen so far.

Recurrent replay (the R2D2 agent) gives the network an LSTM so it carries a hidden state across time, and stores sequences of transitions in the buffer rather than single steps. A wrinkle: a hidden state recorded long ago is stale by the time you replay it, so R2D2 stores the hidden state and 'burns in' a few warm-up steps to re-sync the recurrent memory before computing the loss.

Recurrent replay unrolls an LSTM across time steps so the agent carries a hidden state and learns from whole sequences.

A recurrent network unrolled across several time steps, passing a hidden state forward.

Practical training notes

  1. Budget for scale: Atari results assume tens of millions of frames. If you're learning slowly, suspect too-small a buffer or too-rare target syncs before blaming the algorithm.
  2. Watch the Q-values, not just the score. Steadily climbing-then-exploding Q-values is the classic divergence signature.
  3. Add components one at a time and ablate. A Rainbow that mysteriously underperforms is usually one mis-wired ingredient.
  4. Match the buffer to the task: prioritized for sparse rewards, recurrent (sequence) replay for partial observability.

Where to go next

You now hold the modern value-based foundation: deep RL on pixels, stabilized by replay and target networks, sharpened by Double/Dueling/PER, enriched by distributional learning and noisy exploration, and unified in Rainbow with recurrent memory for POMDPs.

Whatever comes next — value-based or policy-gradient — every agent still lives inside this action–reward loop with its environment.

The reinforcement-learning loop: an agent takes an action, the environment returns a next state and reward.