Learning Without Fresh Interaction: Offline RL

The offline setting and its core pathology

Offline RL (also called batch RL) asks for a policy learned purely from a fixed dataset of past transitions — no environment interaction at all. This is the setting that matters in medicine, robotics with expensive hardware, and anywhere exploration is dangerous or costly. It looks like supervised learning on logged data, and that resemblance is exactly the trap.

The pathology is distributional shift. A value-based learner like Q-learning bootstraps from Q(s', a') where a' is the action the learned policy would take — but that action may never appear in the data. The network extrapolates wildly on these out-of-distribution actions, and because the max in the Bellman target actively seeks out the highest Q-value, it systematically latches onto over-optimistic errors. The policy then chases actions that look great only because the critic has never been corrected on them. This is distribution shift turned adversarial.

y = r + \gamma\,\max_{a'} Q_{\bar\theta}(s',a')

The value-based bootstrap maximizes over a', so out-of-support actions with spuriously high Q get selected — with no environment to correct them, the error compounds.

Conservatism — CQL

Conservative Q-learning (CQL) attacks the overestimation directly. On top of the usual Bellman error it adds a regularizer that pushes down Q-values on the actions the current policy proposes, while pulling up Q-values on the actions actually present in the dataset. The net effect: Q can be trusted on in-distribution actions, but is deliberately pessimistic about anything unfamiliar, so the policy stops being seduced by extrapolated fantasies. Under the right weighting CQL provably learns a lower bound on the true value.

\min_{Q}\;\alpha\,\mathbb{E}_{s\sim\mathcal{D}}\!\Big[\log\textstyle\sum_a e^{Q(s,a)}-\mathbb{E}_{a\sim\hat\pi_\beta}\!\big[Q(s,a)\big]\Big]+\tfrac12\,\mathbb{E}_{\mathcal{D}}\!\big[(Q-\mathcal{B}^{\pi}\hat Q)^2\big]

CQL's objective: a conservative term that pushes Q down on the policy's own actions and up on the dataset's, added to the usual Bellman error.

loss = bellman_error(Q)
     + beta * ( logsumexp_a Q(s,a)        # push DOWN over all actions
               - mean_{a~data} Q(s,a) )    # pull UP on dataset actions

CQL's conservatism penalty bolted onto a standard Bellman loss.

Two families, one principle

CQL is the value-regularization family. A sibling family does policy constraint: keep the learned policy close to the behavior policy that generated the data (by an explicit divergence penalty, by a learned generative model of dataset actions, or by clipping how far the policy may stray). Both express the same offline principle — stay where the data can vouch for you — and both face the same dilemma: too much conservatism and the policy can never improve beyond the dataset; too little and overestimation returns.

RL as sequence modeling — Decision Transformer

A radically different offline approach sidesteps value functions entirely. Decision Transformer treats a trajectory as a sequence of (return-to-go, state, action) tokens and trains a causal transformer to predict the next action — pure supervised sequence modeling, no Bellman backup, no bootstrapping. At test time you condition on a desired return: prompt the model with a high target return and it autoregressively produces the actions that, in the data, tended to achieve it.

Because it never bootstraps, Decision Transformer simply cannot suffer the offline overestimation pathology — a real strength. Its honest weakness is the flip side: with no Bellman backup it cannot stitch together good sub-segments from different sub-optimal trajectories the way value-based methods can, so on datasets that demand such recombination it can underperform CQL. The two approaches are best read as complementary points on a bias–capability spectrum, not rivals.

Sparse rewards and hindsight

A close cousin of the offline challenge is the sparse-reward goal-reaching problem: the agent almost never stumbles onto the goal, so almost every trajectory has zero reward and there is nothing to learn from. Hindsight experience replay (HER) turns failure into a curriculum: after an episode that missed the intended goal, relabel it as if the state the agent actually reached had been the goal all along. Suddenly that trajectory is a successful demonstration — of reaching a different goal — and a goal-conditioned policy can learn from it.

g' = \phi(s_T),\qquad r'_t = r\big(s_t,a_t,g'\big)

Hindsight relabeling: swap the intended goal for the state actually reached, so a failed rollout yields a usable success signal.