offline reinforcement learning

Offline reinforcement learning learns a policy purely from a fixed, previously collected dataset, with no opportunity to try anything in the real environment. It is like learning to drive only from dashcam footage of other drivers: you can study what happened, but you can never test a maneuver you never saw.

The central obstacle is distributional shift. A learned policy inevitably asks about actions the dataset barely covered, and a bootstrapped Q-function extrapolates to those unseen actions with wild, usually optimistic errors that the agent then chases. Remedies fall into policy constraints that keep the learner near the data-generating behavior, value pessimism that distrusts unsupported actions, and importance weighting. Off-policy evaluation, judging a policy without running it, is itself a hard sub-problem.

More data of the same narrow behavior does not cure the problem; what matters is coverage, having support over the state-action regions the new policy will actually visit.