conservative Q-learning (CQL)

Conservative Q-Learning fights overestimation head-on: it deliberately pushes down the values of actions the policy wants but the data does not support, while holding up the values of actions that actually appear in the dataset. The result is a value function that is pessimistic about the unknown.

On top of the usual Bellman error, CQL adds a regularizer that minimizes the value under the current policy's action distribution and maximizes it under the data's actions. This provably yields a lower bound on the true value, so the policy can no longer be fooled into chasing inflated out-of-distribution estimates.

CQL is a workhorse — simple to bolt onto existing Q-learning code and strong across benchmarks — but its conservatism weight is sensitive: too large and it becomes overly timid, too small and overestimation leaks back in.