The DQN Recipe: Replay + Target Network

The DQN loss

A Deep Q-Network (DQN) outputs one value per action for the given state: Q(s, ·). We want those values to satisfy the Bellman optimality equation, so we regress each Q(s, a) toward a target y = r + γ·maxₐ′ Q(s′, a′). The gap y − Q(s, a) is exactly the TD error.

If you train this directly, two problems bite. First, consecutive frames are highly correlated, so each minibatch looks almost identical and the network overfits the last few seconds of play. Second, the target y is computed from the same network you are updating, so the target moves every step — you are chasing a moving goalpost. The two stabilizers below fix exactly these two problems.

The DQN loss carries both fixes at once: the target network θ⁻ and the replay buffer D sit right inside the squared TD error.

Experience replay & the replay buffer

Experience replay stores every transition (s, a, r, s′, done) in a large replay buffer (typically ~1 million transitions), then trains on random minibatches sampled from it. Random sampling shatters the temporal correlation, and each experience can be reused many times — a big win in sample efficiency.

The target network

To stop chasing a moving goalpost, DQN keeps a frozen copy of the network — the target network — and uses it to compute y. Every C steps (say, 10,000) you copy the live network's weights into the target. Between copies the target is stationary, so the regression problem stays well-defined for a stretch and learning stops oscillating.

Replay tackles correlation; the target network tackles the moving target. Together they convert the unstable triad into something that reliably trains. Neither alone is enough — ablate either one and Atari scores collapse.

Reward clipping & Huber loss

One network, 57 games, wildly different score scales (Pong gives ±1, Pinball gives thousands). Reward clipping squashes every reward to {−1, 0, +1} so a single learning rate works everywhere. The cost: the agent can no longer tell a small win from a huge one — a known limitation, not a free lunch.

For the loss itself, DQN uses the Huber loss rather than plain squared error. It behaves like MSE for small TD errors but switches to absolute (linear) error for large ones, so a few outlier transitions can't produce gigantic gradients that destabilize training.

The Huber loss on the TD error e: quadratic for small errors, linear beyond δ — robust to the big errors clipping leaves behind.

Putting it together

Here is the whole loop in pseudocode. Note how preprocessing (grayscale + frame stacking), the buffer, the frozen target, clipping and Huber loss all slot into one tidy training step.

buffer = ReplayBuffer(capacity=1_000_000)
Q_target = copy(Q)

for step in range(total_steps):
    a = epsilon_greedy(Q(s))                 # act
    s2, r, done = env.step(a)                # interact
    buffer.add(s, a, clip(r, -1, 1), s2, done)
    s = env.reset() if done else s2

    batch = buffer.sample(32)               # learn from the past
    y = r + gamma * max_a2 Q_target(s2, a2)  # frozen target net
    y = r if done else y                     # no bootstrap past terminal
    loss = huber(Q(s, a) - y)
    Q.update(loss)

    if step % C == 0:                        # periodic sync
        Q_target = copy(Q)