feature construction

Before a linear model can learn, someone must decide how to describe a state as numbers — that is feature construction. Raw coordinates are rarely enough; you craft features that expose the structure the value function needs, like distances, angles, products of variables, or indicators for regions. Good features make the value nearly linear; bad features make learning impossible no matter the algorithm.

A feature map x(s) sends a state into R^d, and choosing a basis is really choosing what kinds of value functions you can even express. Tricks include polynomial bases, Fourier bases, tile coding, coarse coding, and radial basis functions; each trades off resolution, generalization, and the number of weights. The art is encoding generalization: features that two states share cause an update at one to move the other. You want states with similar values to share features, and states with different values to differ.

Hand-built features were the bottleneck of classical RL — domain-specific, brittle, and labor-intensive, and a method that worked beautifully on one task often fell apart on the next. Deep networks learn features end to end from raw inputs, which is why deep RL replaced most manual feature engineering. But the principle survives unchanged: representation quality, whether learned or designed by hand, quietly decides almost everything about how fast and how well an agent learns.