JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Learning Without an Environment

What offline RL is, why a fixed dataset changes everything, and the vocabulary you need before anything else makes sense.

The one-sentence idea

In ordinary reinforcement learning, an agent learns by acting: it tries something, sees what happens, and adjusts. Offline RL removes the trying. You are handed a fixed log of past experience — states, actions, rewards — and you must squeeze the best possible policy out of it without ever touching the environment again. The synonyms you will meet are offline RL and the older batch RL; they mean the same setting.

Ordinary RL is this loop — the agent acts and the environment responds; offline RL cuts the action arrow and learns from a frozen log instead.

Diagram of the agent–environment reinforcement-learning loop: the agent sends an action, the environment returns a state and a reward.

Why anyone would want this

Online interaction is often the expensive part. A robot can break; a recommender can annoy real users; a treatment policy in healthcare cannot ethically experiment on patients. But data is frequently cheap and already lying around: logs from a deployed system, human teleoperation, an old controller. Offline RL is the promise that you can turn that historical pile into a better policy than the one that produced it.

  1. Cheap or free data: you reuse logs you already collected for other reasons.
  2. Safety: no risky exploration in the real world — the dangerous trials, if any, already happened.
  3. Scale: you can pool data from many sources, sensors, or operators at once.

The words: dataset, behavior policy, off-policy

The dataset is a collection of transitions, usually written as tuples of *(state, action, reward, next state)*, grouped into trajectories. Whatever rule actually chose those actions is the behavior policy — it might be a human, a heuristic, an old neural net, or a mix of all three. Your job is to learn a new target policy that does better. Because the policy you are improving is not the one that gathered the data, offline RL is inherently an off-policy problem — and that mismatch is the whole story.

\mathcal{D}=\{(s_i,\,a_i,\,r_i,\,s_i')\}_{i=1}^{N}\sim\pi_\beta

The offline dataset itself: a fixed set of (state, action, reward, next-state) transitions collected by some behavior policy πβ — and nothing more ever arrives.

Isn't this just supervised learning?

Tempting, but no. The simplest thing you can do is copy the data: predict the action a human took given the state. That is behavior cloning, a genuinely useful baseline. The problem is it can only ever be as good as the data — it imitates mistakes and cannot stitch together good fragments from different trajectories. True offline RL aims to exceed the behavior policy by reasoning about long-term value, not just mimicking actions.

Why offline RL isn't just supervised learning: the three paradigms differ in what signal they learn from.

Diagram contrasting the supervised, unsupervised, and reinforcement-learning paradigms.

And here is the catch that the rest of this track is about: the moment your learned policy wants to do something the data never showed, you have no way to check whether it is brilliant or catastrophic. That single fact — called distributional shift — is why offline RL is hard and why it needs special algorithms.