JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The sim-to-real problem: crossing the reality gap

Training in simulation is cheap and safe; deploying in reality is where policies break. Meet the reality gap and the toolkit — domain randomization, system identification, domain adaptation — for closing it.

Why we train in simulation at all

RL is hungry: a policy may need millions of trials before it is any good. On a real robot, each trial costs wall-clock time, wears out hardware, and can break things — or people. So we train in a simulator, where trials are fast, parallel, free, and safe, then move the learned policy to the real device. This is simulation-to-reality transfer, or sim-to-real.

Simulation also lets us do things that are impossible in the real world: reset to any state instantly, run a thousand robots in parallel, and read perfect ground-truth state for shaping rewards. The catch is that the simulator is never the world.

The reality gap

Every simulator is wrong in small ways: friction is slightly off, motors lag a few milliseconds, the camera sees subtly different colors, contact physics is approximate. A policy that has learned to exploit those exact quirks can be brilliant in sim and useless on hardware. That mismatch is the reality gap, and it is the central villain of applied robotics RL.

Because RL optimizes, it actively hunts for the highest-reward behavior — here a Q-learning agent searches a gridworld for the best policy, the same drive that makes it exploit a simulator's quirks.

An interactive Q-learning gridworld where an agent learns action values to reach a goal.

Domain randomization: make reality just one more sample

The most influential fix flips the problem around. Instead of building one perfect simulator, build a family of imperfect ones and randomize them every episode — gravity, friction, mass, sensor noise, lighting, latency. This is domain randomization. If the policy must succeed across a wide distribution of physics, the real world looks like just another draw from that distribution, and the policy transfers.

\max_{\theta}\; \mathbb{E}_{\xi \sim p(\xi)}\!\left[\, J(\pi_\theta;\, \xi) \,\right]

Domain randomization as an objective: maximize expected return over a whole distribution of randomized physics ξ, so the real device becomes just one more sample.

for episode in training:
    # randomise the simulator before each rollout
    sim.gravity  = uniform(9.4, 10.2)
    sim.friction = uniform(0.7, 1.3)
    sim.mass     = base_mass * uniform(0.8, 1.2)
    sim.latency  = uniform(0, 40)   # milliseconds
    rollout(policy, sim)            # learn to be robust to ALL of these
Domain randomization: train across a distribution of physics so the real device is just one more sample the policy already handles.

Randomize too little and you overfit to the simulator; too much and the task becomes so hard the policy learns nothing. Practitioners often grow the randomization range over training — an automatic curriculum that starts easy and widens as the agent copes.

The randomization range is a fit dial: too little overfits to the simulator, too much makes the task so hard the policy learns nothing — the goal is the good fit in the middle.

Three curves showing underfit, good fit, and overfit.

Closing the gap from the other side

Domain randomization makes the policy robust; two complementary tools make the simulator more honest. System identification measures the real device — its true masses, friction, delays — and tunes the simulator to match, shrinking the gap before training. Domain adaptation instead aligns what the agent perceives: it maps real sensor readings (or real camera images) into the distribution the policy saw in sim, so the same policy feels at home.

  1. Identify: measure the real system and calibrate the simulator so its baseline is close.
  2. Randomize: scatter the remaining unknowns across a distribution so the policy is robust to what you could not measure.
  3. Adapt: bridge any residual perception mismatch by mapping real observations into the simulated distribution.
  4. Test on hardware early and often: the real device is the only benchmark that ultimately counts.

Together these techniques turned sim-to-real from a hope into a method — enabling, for example, dexterous robotic manipulation policies trained entirely in simulation that work on a physical hand the first time it is switched on.