offline reinforcement learning

Imagine learning to manage a hospital ward from years of recorded patient charts, never once treating a live patient until you graduate. Offline reinforcement learning is exactly that: you are handed a fixed dataset of past experience — states, actions, rewards, next states — and must squeeze the best possible policy out of it, with zero new interaction. No exploring, no trying an idea to see what happens. The log is all you will ever get.

Formally you are given a dataset of transitions collected by some behavior policy, and you optimize a policy to maximize expected return using only that dataset. This is attractive wherever live experimentation is costly, slow, or dangerous — healthcare, robotics, recommendation, autonomous driving — because it turns RL into something much closer to supervised learning on logs.

The catch defines the whole field: a good policy will want to take actions the dataset barely covers, and there the value estimates are guesses with no way to check them. Everything in offline RL is, at heart, a strategy for being confident where you have data and humble where you do not.