vector norm

A vector norm boils a whole list of numbers down to a single number that measures its size — how 'big' or 'long' the vector is. The most familiar one is the L2 norm, plain straight-line length: square each entry, add them up, take the square root. It is just the Pythagorean theorem, scaled to as many dimensions as you like, and it answers 'as the crow flies, how far is this from zero?'

But straight-line distance is not the only sensible notion of size. The L1 norm instead adds up the absolute values of the entries — the 'taxicab' or 'city-block' distance, the route you would actually walk on a grid of streets, where you cannot cut diagonally through buildings. The two give different answers and, crucially, push models toward different behaviors when used to keep things small.

Norms are everywhere in machine learning because so much of it is about controlling size. To curb overfitting, regularization adds a penalty on the norm of a model's weights, nudging it to stay simple. The choice of norm matters in a striking way: an L1 penalty tends to drive many weights all the way to exactly zero, automatically discarding useless features (that is what makes Lasso regression do feature selection), while an L2 penalty merely shrinks weights gently toward zero without zeroing them out (Ridge regression). Norms also define the distances that nearest-neighbor methods and clustering rely on, and they measure the magnitude of a gradient — used, for instance, to clip it when it grows dangerously large.