continuous action spaces

Most introductory RL pictures a handful of buttons: move left, move right, jump. But ask an agent to steer a car, set the torque at a robot's elbow, or pick a dosage, and there are no buttons — the action is a real number, or a vector of them, that can take any value in a range. That is a continuous action space: instead of choosing from a finite menu, the agent outputs a point in ℝ^n, usually inside box limits like a steering angle between −1 and 1.

The jump from discrete to continuous breaks the workhorse trick of value-based RL. Q-learning picks the best action by scanning every option and taking the max, but you cannot enumerate infinitely many real-valued actions, and the maximisation max_a Q(s,a) becomes its own hard optimisation problem at every step. So continuous control leans on policies that directly output an action — a deterministic map, or the parameters of a distribution you sample from — sidestepping the impossible max.

This is why a whole family of algorithms exists just for this setting: DDPG, TD3, and SAC all answer the question of how to learn and improve a policy when the action is a knob, not a switch. The pay-off is that the most physically real problems — locomotion, flight, manipulation, process control — are natively continuous, so mastering this space is mastering control of actual machines.