distributional shift (in offline RL)

Your dataset shows what happened under the old behavior. But the moment your new, improved policy starts choosing differently, it steers the system into situations the old policy rarely or never visited. That gap — between the distribution of states and actions in the data and the distribution the learned policy actually induces — is distributional shift, and it is the core disease of offline RL.

It bites twice. During training, the Bellman backup queries the value at next-state actions chosen by the learned policy, which may be out-of-distribution. At deployment, the policy walks into states the data never covered, where it has effectively never been taught what to do.

Because there is no online interaction to correct these mistakes, the errors compound silently. Almost every offline algorithm is, at heart, a different answer to one question: how do we stop the policy from relying on parts of the world we know nothing about?