pessimism in offline RL

Online RL lives by optimism: when unsure, try it and find out. Offline RL must live by the opposite. With no way to test a hunch, you should assume the worst about anything the data cannot vouch for — be pessimistic exactly where the evidence is thin.

Formally, you act to maximize a lower confidence bound on value rather than the value estimate itself, subtracting a penalty that grows where data is sparse. This is the unifying principle behind conservative value penalties in CQL, support constraints in BCQ, and in-sample learning in IQL: all are ways of being pessimistic about under-covered actions.

Theory shows pessimism is more than a heuristic — it yields guarantees that scale with how well the dataset covers the comparator policy, not the whole state-action space. The art is calibrating it: enough caution to stay safe, not so much that you never improve.