continuous control from pixels

Most continuous-control results assume the agent is handed clean state — exact joint angles and velocities. Continuous control from pixels removes that crutch: the agent sees only raw camera images, the way a real robot does, and must learn both to perceive what state it is in and to act, all from reward alone. The action stays continuous (joint torques), but now a convolutional network has to extract the relevant geometry from pixels before any controller can use it.

This is markedly harder than learning from state. Images are high-dimensional and most of each frame is irrelevant; the reward signal is a thin teacher for the heavy job of learning a good visual representation; and off-policy methods like SAC become unstable when the encoder is learned jointly with the value function. The breakthrough recipe was to lean hard on data augmentation — randomly shifting or cropping the input images — and self-supervised auxiliary losses, which give the encoder far more to learn from than reward alone.

Methods such as image-based SAC with augmentation, and contrastive or reconstruction-based representation learning, eventually matched state-based performance straight from pixels on standard benchmarks — a milestone, because real robots rarely get handed clean state. It connects continuous control to representation learning and is a stepping stone toward agents that perceive and act in the messy real world rather than a tidy simulator.