function approximation

A real environment can have astronomically many states — every board position, every camera frame. You cannot store a separate number for each. Function approximation replaces that giant lookup table with a compact parameterized function: feed in a state (or state-action pair) and it computes the value or action probability from a handful of weights. Learning means adjusting those weights, not filling in cells.

Formally we pick a parameterized family, say an estimate v(s; w) close to the true value, and tune the weight vector w by gradient steps that push the prediction toward a target (a return, or a bootstrapped estimate). The same idea covers policies via a parameterized pi(a|s). Because nearby states share weights, an update at one state generalizes to similar states — that is both the power (learn from few samples) and the danger (interference, instability).

This is what lets RL scale to Go, robotics, and raw pixels; deep RL is just function approximation with a neural network. But almost every hard convergence problem in RL — divergence, the deadly triad, overestimation — is born here, because the clean guarantees of tabular methods no longer hold.