JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The Core Problem: Distributional Shift

Why running ordinary off-policy Q-learning on a fixed dataset quietly blows up — and why the cure is a single word: pessimism.

Recap: the dangerous max

Most value-based RL updates a Q-value toward `r + γ · maxₐ′ Q(s′, a′)` — the Bellman optimality target. Online, this works because when the `max` picks an over-optimistic action, the agent eventually goes there, sees reality, and corrects the estimate. The correction loop depends on being able to collect new data. Offline, that loop is cut.

Q(s,a) \leftarrow Q(s,a) + \alpha\left[\, r + \gamma \max_{a'} Q(s',a') - Q(s,a)\,\right]

The Q-learning update: the bootstrapped target hinges on maxₐ′ Q(s′,a′) — the very 'max' that turns dangerous offline.

Extrapolation error

A function approximator does not return `0` or `unknown` for an action it has never seen — it confidently extrapolates a number, often a wildly wrong one. When the `max` ranges over all actions, it will happily seize on whichever out-of-distribution action happens to have the largest error, not the largest true value. This is extrapolation error, and offline it is never washed out.

Like an overfit curve far from its data, a Q-network extrapolates a wildly wrong number for actions it never saw.

Underfit, good-fit, and overfit curves; the overfit curve swings wildly outside the training points.

# Naive off-policy Q-learning, run OFFLINE on a fixed dataset D
for (s, a, r, s_next) in D:
    # The target maxes over ALL actions a' -- including ones never in D
    target = r + gamma * max_over_a_prime(Q(s_next, a_prime))
    Q(s, a) += alpha * (target - Q(s, a))

# The max keeps selecting the action with the LARGEST estimation error,
# not the action with the largest true value. There is no new data in D
# that can ever push those phantom over-estimates back down.
The failure in four lines: the max feeds on errors the dataset cannot correct.

Distributional shift, precisely

Here is the chain. Your learned policy is trained to prefer high-Q actions. The over-estimated actions are exactly the unseen ones. So your policy drifts toward states and actions the behavior policy rarely or never visited. But the Q-values there were never grounded in data — so they are wrong — so the policy chases them harder. That self-reinforcing divergence between the data distribution and the policy's induced distribution is distributional shift, the defining hazard of offline RL.

Interactive Q-learning on a gridworld: the policy is pulled toward high-Q cells — offline, that pull drifts into states the data never covered.

A gridworld where Q-learning assigns values to cells and the agent moves toward higher-value states.

The cure in one word: pessimism

Every offline RL method is, at heart, an answer to one question: *what should we believe about actions the data never showed us?* The universally winning answer is pessimism — assume the unknown is bad until proven otherwise. Concretely, this means either (a) forbidding the policy from straying far from the behavior policy, or (b) deliberately under-estimating the value of unseen actions. The next guide turns these two instincts into named algorithms.

\max_{\pi}\ \mathbb{E}_{(s,a)\sim\pi}\!\left[\,\hat{Q}(s,a)-\lambda\,U(s,a)\,\right]

Pessimism made precise: maximize value minus a penalty λ·U(s,a) for poorly-supported actions, so uncertainty pulls the policy back toward what the data covers.