linear function approximation

The simplest useful approximator. You first turn each state into a feature vector x(s) — a list of numbers describing it — then estimate its value as a weighted sum of those features. If a feature measures distance to goal, a single weight decides how much that matters. The estimate is a flat plane (a hyperplane) over feature space.

The form is v(s; w) = w^T x(s). The gradient with respect to w is simply x(s), so the update is clean: w <- w + alpha[target - w^T x(s)] x(s). For prediction (policy evaluation), on-policy linear TD is one of the few approximate settings with solid convergence guarantees — it converges to a well-characterized fixed point (the TD fixed point) near the best linear fit.

Linear methods are fast, stable, and analyzable, but only as good as your features — they can represent only what is linear in x(s). Much of classic RL craft is engineering features (tile coding, RBFs) so that a linear layer on top suffices. Deep RL automates that feature-building.