JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Cause, Effect, and the Road Ahead

Agents that confuse correlation with control fail in the open world. We close with causal RL and an honest map of what would finally make RL general.

When correlation breaks an agent

Most RL learns from correlations — which states tend to precede high return. That is fragile, because a correlation can come from a confounder: a hidden cause that moves both the action and the outcome. Train a driving agent on human logs where the wipers are usually on when it rains, and it may learn 'wipers on → safe' — copying a marker of the cause rather than the cause itself. The instant the world changes, the spurious link snaps. This failure is most acute precisely in the offline and logged-data settings the frontier most wants to use.

Because the agent acts on the environment, every step is an intervention, not a passive observation — the crux of why correlation can mislead it.

Diagram of the agent–environment loop: the agent takes an action and receives a next state and reward.

Causal RL — interventions, not just observations

Causal RL equips agents with a model of cause and effect, not just statistics. Using structural causal models and the language of do-calculus, it distinguishes 'what tends to co-occur' from 'what happens if I intervene and do this.' A policy is itself an intervention, so this is the natural language for RL. The payoff lands on the very gaps from Guide 1: causal structure is the invariant that survives distribution shift, so it promises sturdier generalization, and reasoning about mechanisms — not just rewards — can sharply improve sample efficiency.

P\big(y \mid \mathrm{do}(x)\big) \neq P(y \mid x)

The heart of do-calculus: the effect of intervening to set x can differ from merely observing x — exactly the distinction a causal agent must respect.

Counterfactuals and credit

Causality also sharpens RL's oldest headache, credit assignment. A counterfactual question — 'would the outcome have been better had I acted differently, holding everything else fixed?' — is exactly the comparison an advantage tries to estimate, now on a principled footing. Counterfactual reasoning underlies cleaner off-policy evaluation and, in multi-agent settings, lets you isolate one agent's true contribution by imagining the world without its action.

A^{\pi}(s,a) = Q^{\pi}(s,a) - V^{\pi}(s)

The advantage function captures the counterfactual at the heart of credit assignment: how much better is action a than the state's baseline value?

The frontier, assembled

Step back and the pieces of this track form one picture of what a general agent would need. From Guide 1: cross the generalization and sample-efficiency gaps. From Guide 2: reuse knowledge — adapt fast, transfer, never forget, learn without rewards. From Guide 3: scale via the foundation-model recipe, with scaling laws and open-ended environments feeding endless diverse experience. From Guide 4: specify the right objective and stay safe. From this guide: ground it all in causal understanding. No single idea is sufficient; the frontier is their intersection.

An honest closing

These are open problems, not solved ones — which is the good news for you. RL today is roughly where supervised learning sat before its scaling era: powerful in narrow wins, searching for the recipe that makes it broadly reliable. If you want to contribute, pick one gap and make it measurable: build a benchmark that forces generalization, an environment generator that never repeats, a reward that resists hacking, or a causal method that survives a shift no correlational baseline can. The agents that finally learn the way we hoped will be built on exactly these questions.