normalized advantage functions (NAF)

NAF is a clever way to do plain Q-learning in continuous action spaces without a separate actor. Recall the obstacle: Q-learning needs max_a Q(s,a) and argmax_a Q(s,a), but you cannot search over infinitely many real-valued actions. NAF sidesteps this by restricting the shape of Q in the action variable to a quadratic — a downward parabola, or bowl, in action space — for which the maximum and the maximiser have neat closed-form answers.

The network outputs three things for each state: the state value V(s), the location of the bowl's peak μ(s), and a matrix defining the bowl's curvature P(s). The advantage is then a quadratic penalty for departing from μ(s), so Q(s,a) = V(s) − ½(a − μ(s))ᵀ P(s) (a − μ(s)). Because this is concave in a by construction, the best action is simply μ(s) and the max value is just V(s) — both read off instantly, with no optimisation and no actor network at all.

The elegance comes at a price: forcing Q to be quadratic in the action is a strong assumption, and when the true value landscape is multimodal or oddly shaped, NAF cannot represent it and underperforms the more flexible actor-critics. So although it predates DDPG-style methods as a continuous Q-learning idea, the field largely moved to DDPG, TD3, and SAC, which place no such restriction on the value function's shape.