squashed Gaussian policy

Real actuators have limits — a motor saturates, a steering angle bottoms out — so a continuous policy must output actions inside a bounded box, say between −1 and 1. A plain Gaussian, the natural choice for a stochastic continuous policy, has infinite tails and happily proposes actions far outside any limit. The squashed Gaussian fixes this by sampling from a Gaussian and then passing the sample through a tanh function, which gently bends the whole real line into the open interval (−1, 1).

So the policy first produces an unbounded sample u = μ_θ(s) + σ_θ(s)·ε, then the action is a = tanh(u). This keeps every action legal by construction while staying differentiable, so it pairs perfectly with the reparameterisation gradient. There is one piece of bookkeeping: squashing changes the probability density, so the log-probability used in the entropy term needs a correction term, the log-determinant of the tanh's derivative — easy to forget and a classic source of subtle bugs.

This is the default policy class inside soft actor-critic and a common choice across continuous control. It marries three needs at once: a stochastic policy for exploration and entropy, respect for action bounds, and a smooth differentiable path for low-variance learning. The tanh does have a downside — saturation near the edges flattens the gradient — which links it to the action-saturation pitfall.