JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Algorithms: Constrain the Policy or Distrust the Value

BCQ, CQL, IQL, AWR — the working algorithm families of offline RL, and a clear rule for which to reach for.

Two ways to be pessimistic

Almost every practical offline algorithm picks one of two strategies. Policy constraint: keep the learned policy close to the behavior policy, so it only ever proposes actions the data supports — this family is policy-constraint methods. Value pessimism: let the policy be whatever, but train the value function to actively distrust out-of-distribution actions, so they never look attractive. Some modern methods quietly do both.

BCQ — only consider plausible actions

Batch-Constrained Q-learning (BCQ) was the method that put distributional shift on the map. Its trick is to train a generative model of the data, propose only actions that model deems likely, and take the `max` over that restricted set instead of all actions. By construction the dangerous unseen actions are never even candidates, so extrapolation error is cut off at the source.

BCQ trains a generative model (a conditional VAE) of the data and only proposes actions it deems plausible — never querying the unseen.

A variational autoencoder: an encoder maps inputs to a latent space and a decoder reconstructs them, used here to generate in-distribution actions.

CQL — push the phantom values down

Conservative Q-Learning (CQL) attacks the value side. Alongside the normal Bellman loss it adds a regularizer that lowers Q for the actions the current policy would pick (likely out-of-distribution) and raises Q for actions actually present in the data. The result is a learned Q-function that is a provable lower bound on the true value — the over-estimates simply can't form.

Q(s,a) \leftarrow Q(s,a) + \alpha\left[r + \gamma \max_{a'} Q(s',a') - Q(s,a)\right]

The Bellman target CQL builds on; the max over next actions is where unseen-action overestimation creeps in — so CQL adds a term that pushes those values down.

# Conservative Q-Learning (CQL): Bellman loss + a pessimism penalty
bellman_loss = (Q(s, a) - (r + gamma * V(s_next))) ** 2

# Push DOWN Q on actions the policy samples (likely OOD);
# pull UP Q on the action actually logged in the data.
pessimism = logsumexp_over_a(Q(s, a)) - Q(s, a_in_data)

loss = bellman_loss + alpha * pessimism
CQL = ordinary value learning plus a term that makes unseen actions look worse, not better.

IQL and AWR — never query an unseen action at all

Implicit Q-Learning (IQL) takes the cleanest stance: never evaluate `Q` on any action outside the dataset. It approximates the value of the best in-data action using an expectile regression of the state value, so the Bellman target only ever uses actions that really occurred. It then extracts a policy with advantage-weighted regression (AWR) — weighted behavior cloning that copies data actions more strongly when their estimated advantage is high. Simple, stable, and a frequent winner.

IQL still learns gridworld-style Q-values but only ever evaluates actions the data contains — it never takes a max over unseen actions.

Interactive gridworld where Q-values are learned cell by cell; only in-data actions are evaluated.

The common thread

Read those four methods again and you will see the same instinct wearing different clothes: pessimism about the unknown. BCQ restricts the action set, CQL lower-bounds the values, IQL refuses to look off-data, AWR leans on imitation. Whichever you choose, you are paying for safety with a little conservatism — and that trade is exactly what makes offline RL trustworthy.