JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Fixing Overestimation: Double, Dueling, and Smarter Replay

Plain DQN is biased to be over-optimistic. Three targeted upgrades make its value estimates honest and its learning sharper.

Why vanilla DQN is over-optimistic

The target y = r + γ·maxₐ′ Q(s′, a′) takes a max over noisy estimates. Because the same noisy Q both picks the best action and reports its value, the max systematically lands on whichever action happened to be over-estimated. This is maximization bias, and with a function approximator it compounds into chronic overestimation of action-values.

Interactive Q-learning on a gridworld: watch values spread from the max-over-actions target that DQN inherits — the very step that turns over-optimistic.

A gridworld where each cell shows a learned Q-value that updates as the agent explores.

Double DQN: decouple selection from evaluation

The cure, borrowed from tabular Double Q-learning, is to use two networks for the two jobs. Double DQN lets the online network select the best next action, but asks the target network to evaluate it. Since the two networks have uncorrelated errors, the selected action is no longer guaranteed to be the over-estimated one.

# vanilla DQN target (over-estimates):
y = r + gamma * max_a2 Q_target(s2, a2)

# Double DQN target (decoupled):
a_star = argmax_a2 Q_online(s2, a2)   # ONLINE net selects
y = r + gamma * Q_target(s2, a_star)  # TARGET net evaluates
One line of change: separate who picks the action from who scores it.
y = r + \gamma\, Q\!\left(s',\; \operatorname*{arg\,max}_{a'} Q(s', a'; \theta);\; \theta^{-}\right)

Double DQN target: the online network picks the next action while the target network scores it, breaking the self-confirming max.

Dueling architecture: value + advantage

Often the exact action barely matters — when you're falling off a cliff, every action is bad. The dueling network architecture splits the head into two streams: a single state-value V(s) and an advantage A(s, a) for each action, then recombines them as Q(s, a) = V(s) + (A(s, a) − meanₐ A(s, a)).

The subtracted mean is what keeps the decomposition identifiable. The payoff: the network can learn that a state is good or bad without having to nail down every action's value first, which speeds learning in states where actions don't differ much.

Prioritized experience replay

Uniform replay treats every transition as equally informative — but a transition with a big surprise (large TD error) teaches more than one the agent already predicts perfectly. Prioritized experience replay (PER) samples transitions in proportion to their TD-error magnitude, so the agent rehearses its biggest mistakes more often.

\delta = r + \gamma \max_{a'} Q(s', a'; \theta^{-}) - Q(s, a; \theta)

The TD error δ measures how surprising a transition is; prioritized replay samples large-|δ| transitions more often.

When to reach for each

  1. Seeing inflated, drifting Q-values? Add Double DQN first — it's nearly free and almost always helps.
  2. Many actions, but the state's overall quality dominates? The dueling head accelerates value learning.
  3. Wasting compute replaying already-mastered transitions? Switch to prioritized replay.
  4. All three are orthogonal and stack cleanly — which is exactly the setup for Guide 5's Rainbow.