JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

From Benchmarks to Real Robots

Where do these algorithms actually run? Walk the benchmark ladder, learn straight from camera pixels, and cross the gap from simulator to a robot you can touch.

The benchmark ladder

Methods are compared on a shared ladder of difficulty. The bottom rungs are the MuJoCo benchmarks of locomotion — HalfCheetah, Hopper, Walker2d, Ant, Humanoid — where the body is given and the agent must learn a gait. These are standardized, fast, and reproducible, which is exactly why papers report them.

Every benchmark on the ladder is an environment for this loop: the agent acts, the environment returns a new state and a reward.

Diagram of the reinforcement-learning loop: an agent sends an action to the environment, which returns the next state and a reward.

Higher up sits dexterous manipulation: controlling a multi-fingered hand to reorient a cube, open a door, or assemble parts. The action space is large, contact physics is stiff and chaotic, and reward is easy to get wrong — manipulation stresses everything continuous control is bad at.

Learning straight from pixels

Benchmarks usually hand the agent clean state — joint angles and velocities. A real robot mostly sees a camera. Continuous control from pixels asks the agent to learn both what the world looks like and how to act, from raw images, end to end. That is far harder: the useful signal is buried in thousands of pixels and the reward is delayed.

The practical recipe is to put a convolutional encoder in front of a SAC-style agent and help it along with image augmentation (random crops and shifts) plus an auxiliary objective that reconstructs or predicts frames. With those, pixel-based agents now rival state-based ones on many tasks — a key step toward control on real hardware.

Learning from camera pixels: a convolutional encoder turns each image into features before the SAC-style agent acts.

A convolutional pipeline: an image passes through convolution and pooling layers into a feature vector.

Crossing the reality gap

Training on a real robot is slow, costly, and risky, so we train in simulation and transfer. The catch is the reality gap: a simulator never matches reality perfectly — friction, mass, latency, and sensor noise all differ — so a policy that aces the sim can flop on the robot. Getting from sim to hardware is sim-to-real transfer, its own research problem.

The most reliable cure is domain randomization: during training, randomize the simulator's physics and appearance — masses, friction, lighting, textures, control delay. The policy is forced to work across a whole family of worlds, so the real one looks like just another sample it has already handled. It is a deceptively simple idea that underpins much of modern robotic manipulation with RL.

Randomize too little and the policy overfits the simulator; the right range generalizes across worlds.

Three fits side by side: underfit, good fit, and overfit to the training data.

A practical checklist

Bringing the whole track together, here is a sane order of operations for a new continuous-control problem.

  1. Define the action space and its bounds, and normalize observations and actions to comparable scales — unscaled inputs are a top cause of silent failure.
  2. Start from a strong baseline: SAC for sample efficiency and minimal tuning, or TD3 if you want a simpler deterministic method.
  3. Shape the reward carefully and watch for reward hacking — a continuous agent will exploit any loophole in your reward to the decimal place.
  4. If you only have a camera, switch to a pixel encoder with image augmentation; expect to need more data and patience.
  5. Before touching hardware, train in sim with domain randomization, then evaluate on held-out randomized worlds as a proxy for the reality gap.
  6. Deploy with safety limits on torque and velocity, and keep a human or hardware kill-switch in the loop.

Where the field is going

The frontier is moving past one-task-at-a-time training: learning policies from offline robot datasets, sharing skills across many tasks and embodiments, and folding control into large multimodal models that perceive and act. But the core ideas from this track — an actor that emits real-valued commands, a pessimistic critic that resists its own optimism, and exploration that is principled rather than bolted on — remain the foundation every new method is built on.