JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why Deep RL? From Q-tables to Q-networks

When the state space is a screen full of pixels, a lookup table can't help you. Here's what changes when a neural network takes over.

The wall tabular methods hit

Classic tabular Q-learning keeps one number for every (state, action) pair. That is wonderful when there are a few hundred states. But point a camera at an Atari screen and a single frame is 210×160 pixels — billions upon billions of possible images. No table on Earth could store a row for each one, and you would never visit the same state twice anyway.

Interactive tabular Q-learning on a gridworld: one stored value per state-action cell — fine here, hopeless once a state becomes a screen of pixels.

A small grid where each cell shows learned action-values that update as the agent explores.

The fix is generalization: instead of memorizing values, learn a function that maps a state to action-values and shares what it learns across similar states. This is function approximation, and when the function is a deep neural network we call the whole enterprise deep reinforcement learning.

What 'deep' actually buys you

A deep network is a learned feature extractor. Early convolutional layers discover edges, then moving sprites, then game-relevant concepts like 'the ball is heading left' — all without anyone hand-coding them. The same Q-learning update you already know then rides on top of those features.

A convolutional pipeline turns raw pixels into game-relevant features — exactly the learned feature extractor deep RL leans on.

An image flows through convolution and pooling layers into a compact feature vector, then a classifier.

The Atari challenge

The benchmark that launched deep RL was the Atari benchmark built on the Arcade Learning Environment: ~57 games, one network architecture, only the pixels and the score as input. No game-specific tuning allowed — the same agent has to learn Breakout, Pong, and Space Invaders.

One frame is not enough state: from a still image you can't tell which way the ball is moving. The standard trick is frame stacking — feed the last 4 frames together so velocity and direction become visible to the network. It is a cheap, practical way to make a near-partially observable problem look Markovian.

The catch: naive deep Q-learning falls apart

If you simply swap the Q-table for a network and run Q-learning, it usually diverges — values explode, the agent forgets, training oscillates. The culprit is the deadly triad: combine function approximation, bootstrapping (learning from your own predictions), and off-policy updates, and stability is no longer guaranteed.

Q(s,a) \leftarrow Q(s,a) + \alpha\,\big[\, r + \gamma \max_{a'} Q(s',a') - Q(s,a) \,\big]

The bootstrapped Q-learning target reuses the network's own estimate — one corner of the deadly triad that makes naive deep Q-learning diverge.

The road ahead

  1. Guide 2 — the DQN recipe: experience replay + a target network, the two ideas that made deep Q-learning stable at all.
  2. Guide 3 — fixing overestimation: Double DQN, the Dueling architecture, and prioritized replay.
  3. Guide 4 — beyond the mean: distributional RL (C51, QR-DQN) and noisy-network exploration.
  4. Guide 5 — Rainbow and recurrent replay: combine everything, then handle memory and partial observability.