universal value function approximators

A normal value function answers one question: how good is this state for my one task? A universal value function answers a whole grid of questions at once: how good is this state if my goal were g, for any goal g you care to name? It is a single function V(s, g) — or Q(s, a, g) — that takes both where you are and what you are trying to achieve, and returns the expected return for that pairing.

The point of folding the goal into the function is generalisation across goals. A plain table would have to learn each goal's values independently, but a neural approximator over (state, goal) pairs can share structure: having learned to value states for one goal, it can interpolate sensibly to goals it has never been trained on, because nearby goals usually have similar value landscapes. This is the representation that makes goal-conditioned and hierarchical agents practical at scale.

UVFAs pair naturally with hindsight relabeling, which manufactures the abundance of (state, goal) data they need to learn from, and they give a manager a clean way to evaluate which goals are worth handing to a worker. The cost is that learning a function over the joint space is harder than over states alone, and poor goal representations can make generalisation collapse.