deep Q-network (DQN)
/ deep kyoo NET-werk /
The deep Q-network, or DQN, was the breakthrough that fused two ideas — Q-learning and deep neural networks — into something that could learn to play video games straight from the screen. The problem it solved is plain: classic Q-learning keeps a separate number for every situation, but a game has more possible screens than you could ever list. DQN replaces that impossible notebook with a neural network that looks at the raw pixels and estimates a Q-value for each button — generalizing to screens it has never seen.
Naively bolting a neural network onto Q-learning, though, blows up: training becomes unstable and often diverges. DQN's real contribution was two tricks that tamed it. Experience replay stores past moments in a memory buffer and trains on random samples of them, breaking the harmful correlations between consecutive frames. A separate, slowly-updated target network provides a steady reference for the Bellman update, so the network isn't chasing a target that jerks around every step. Together these made deep Q-learning actually work.
In 2013–2015, DeepMind showed a single DQN — same algorithm, same architecture, same settings — learning dozens of Atari games from nothing but pixels and score (a fresh copy trained per game), beating human experts on many. It was a landmark for general learning from raw perception. But be honest about the limits: it needed enormous amounts of play, far more than any human; it floundered on games requiring long-term planning (Montezuma's Revenge was notorious); and it works only when actions are a small discrete set, which is why continuous control needs different methods.
Playing Atari Breakout, DQN sees only the screen pixels and the score. With no idea what a 'ball' or 'paddle' is, it learns over millions of frames to move the paddle to keep the ball alive — and even discovers the expert trick of tunneling through one side to bounce the ball behind the bricks.
Learning Atari from pixels alone — propped up by experience replay and a frozen target network.
DQN's headline 'one algorithm, many games' is true but easily oversold: a fresh network was trained from scratch per game, not one network mastering all at once. And it stumbled badly on games needing long-horizon planning or sparse rewards — proof that 'learns from pixels' is not the same as 'understands the game.'