least-squares temporal-difference (LSTD)

For linear value functions, you do not have to learn by slow gradient steps — you can solve for the best weights directly. LSTD gathers statistics from the data and computes, in closed form, the weight vector that satisfies the linear TD fixed-point equations. It is the batch, no-step-size answer to TD prediction.

With features x(s), TD's fixed point is the w solving A w = b, where A = E[x(s)(x(s) - gamma x(s'))^T] and b = E[r x(s)]. LSTD estimates A and b from samples and solves w = inverse(A) b. It uses every sample maximally (very data-efficient) and has no learning rate to tune; LSTDQ extends it to action values for use inside policy iteration (LSPI).

The cost is O(d^2) memory and up to O(d^3) to invert, so it suits moderate feature counts, not million-parameter nets. It is data-efficient but compute-heavy, and like all TD-fixed-point methods it finds the best linear fit in the projected sense, not the true value unless the true value is already linear in the features.