conservative Q-learning (CQL)

Conservative Q-learning is a remedy for the central failure of offline RL: a Q-function that is over-optimistic about actions it has never truly seen. CQL deliberately pushes Q-values down on out-of-distribution actions, so the policy cannot be lured into chasing a value that is merely a hallucinated jackpot from extrapolation.

On top of the usual Bellman error, it adds a regularizer that minimizes the expected Q under a broad or learned action distribution while maximizing Q on the actions actually present in the dataset. The net effect is that values for unsupported actions are suppressed relative to in-distribution ones, and the learned Q provably lower-bounds the true value, giving a conservative estimate. A trade-off weight sets how aggressive the pessimism is.

Because it lower-bounds policy value, CQL is a workhorse for offline control. The price is that too large a weight yields an overly timid policy that ignores genuinely good but rarely observed actions.