batch-constrained Q-learning (BCQ)

Batch-Constrained Q-learning was the method that gave extrapolation error its name. Its idea: only ever consider actions that plausibly could have come from the data. If the dataset never shows an action near this one, do not even let the agent contemplate it.

BCQ trains a generative model of the behavior policy, samples a handful of candidate actions from it, perturbs them slightly, and then takes the max only over that restricted, in-support set. The Bellman backup therefore never queries a truly unseen action.

It was a landmark demonstration that the offline failure was about action support, not algorithm choice. Later methods like CQL and IQL achieve similar protection more simply, but BCQ's support-restriction framing remains the conceptual backbone of the field.