Rainbow DQN

By 2017 the DQN family had sprouted a half-dozen separate improvements, each published on its own and each helping. Rainbow asked the obvious question: do they stack? It combines six of them into a single agent — Double DQN, prioritized replay, the dueling architecture, multi-step returns, distributional value learning, and noisy networks for exploration — and the answer was yes: the combination beat every individual ingredient and set a new state of the art on Atari.

Beyond a strong agent, Rainbow is a lesson in how the pieces interact, because the authors also ran ablations removing one component at a time. Distributional learning and prioritized replay turned out to contribute the most; removing noisy nets or dueling hurt less. That kind of careful study of which parts actually matter is rarer than it should be. Rainbow remains a standard strong baseline, the thing a new value-based method is expected to be compared against.