Q-function
/ kyoo FUNK-shun /
The Q-function is a slightly richer cousin of the value function. Instead of asking 'how good is this situation?', it asks 'how good is this situation if I take that specific action next, and play well afterward?' Where the plain value function scores a state, the Q-function scores a state-and-action pair together — one number for each thing you could do from here.
That extra detail is enormously useful, because it tells you directly which action to pick: just choose the action with the highest Q-value in your current state. No need to imagine the future yourself — the Q-function has already folded the whole future into a single comparison. The 'Q' stands for 'quality': the quality of taking a given action from a given state.
This is what makes Q so practical and so famous (it powers Q-learning and the deep Q-network). But it has a built-in limit: comparing Q-values across actions only works cleanly when the actions are a tidy, finite list — turn left, turn right, jump. When the action is a continuous quantity, like 'rotate the joint by 14.3 degrees,' there are infinitely many options to compare, and pure Q methods break down. That is why robotics and other smooth-control problems often reach for policy-gradient methods instead.
Standing on a grid square, the agent has four Q-values: Q(here, up)=8, Q(here, down)=3, Q(here, left)=5, Q(here, right)=9. It picks 'right' because 9 is highest. Each number already accounts for everything that happens after that move.
One Q-value per available action; act greedily by taking the highest. 'Q' = quality of (state, action).
V(s) and Q(s,a) are tightly linked: the value of a state equals the Q-value of the best action you can take there. The Q-function carries one extra slot — the action — and that single addition is what lets an agent decide what to do without simulating the future itself.