When Actions Are Continuous

A dial, not a button

A video-game agent presses one of a few buttons. A robot arm, a car, or a walking robot does something different: it sets a real number on each motor — a torque of 2.7 newton-metres, a steering angle of -0.14 radians. The set of all such choices is a continuous action space, and it is the natural language of physical control.

Contrast this with a discrete action space of, say, four moves. In the discrete case you can literally list every action and ask 'which is best?'. In a continuous action space there are infinitely many actions, often several dials at once (a robot hand may have twenty joints), so listing is hopeless.

Why DQN's argmax breaks

The workhorse of discrete deep RL, the Deep Q-Network (DQN), learns a value Q(s,a) for every action and then acts by taking the argmax — the action with the highest value. That argmax is a quick scan over the menu.

Now make the menu infinite. To take an argmax over a continuous action you would have to solve an optimization problem inside every single step of acting and learning. That inner optimization is itself hard and slow, and it has to be redone for every state. The clean DQN recipe simply does not transfer.

The fix: output the action directly

Instead of scoring every action and picking the best, we train a second network — the actor — to output a good action straight away: action = mu(state). The argmax is replaced by one forward pass. A critic still learns Q(s,a) to tell the actor how good its choices are. This actor-critic split is the backbone of every method in this track.

Bounding is handled at the actor's last layer. A common trick is to pass the raw output through a tanh, which squashes any real number into the range [-1, 1], then rescale to the motor's limits. This guarantees legal commands but introduces action saturation: push tanh hard and it flattens, so its gradient shrinks toward zero and learning at the extremes slows to a crawl.

a = a_{\text{low}} + (a_{\text{high}} - a_{\text{low}})\cdot\frac{\tanh\!\left(\mu_\theta(s)\right) + 1}{2}

The actor outputs an action directly, squashing through tanh and rescaling so it always lands inside the motor's limits.

The playground

Continuous control has a shared set of test beds so methods can be compared fairly. The classic suite is the MuJoCo benchmarks: simulated physics tasks like HalfCheetah, Hopper, Walker2d, Ant, and Humanoid. Most of these are locomotion tasks — the agent learns a gait that moves a body forward as fast and stably as it can.