implicit Q-learning (IQL)

Implicit Q-Learning sidesteps the whole problem of unseen actions by never asking about them. The Bellman target's max over actions is exactly where out-of-distribution actions sneak in; IQL refuses to take that max, and instead estimates the value of the best actions already present in the data using expectile regression.

It learns a state value that is an upper expectile of the action-values seen at that state, approximating the in-sample maximum without ever evaluating an action outside the dataset. A separate policy is then extracted by advantage-weighted behavior cloning. The action-value is only ever queried at dataset actions.

Because it touches no out-of-distribution action, IQL is remarkably stable and needs no explicit constraint or conservatism penalty. The price is expressiveness: its ceiling is whatever the data contains, since it can only stitch together actions it has actually seen.