action-value function (Q-function)

The state-value function rates situations; the action-value function rates choices. It answers a more useful question for decision-making: if I am in state s and I take action a right now, then follow my policy afterward, what return can I expect? By scoring each available action, it tells you directly which move looks best, without your having to imagine the next state yourself.

It is written Q-pi of s and a, and often just called the Q-function. The link to the state-value function is simple: the value of a state is the average of its action values over the actions your policy would pick. Q is the workhorse of many RL methods, because once you have accurate action values you can act well by a trivial rule — just take the action with the largest Q in each state.