curse of dimensionality

The curse of dimensionality is the strange way that adding more features (more dimensions) to your data makes learning harder, not easier — the opposite of what intuition suggests. The slogan: in high dimensions, everything is far away from everything else, and space becomes mostly empty.

Here is the intuition. Imagine scattering 100 dots along a one-meter line — they're cozy, about a centimeter apart. Spread the same 100 across a one-meter square and they're sparse; across a one-meter cube, sparser still. Each new dimension multiplies the volume to fill, so the amount of data you'd need to keep the space well-populated grows explosively — roughly exponentially with the number of features. With many features and limited data, points become so spread out that notions like "nearest neighbor" lose meaning, because the nearest and farthest points are almost equally far. Models then struggle to find real patterns amid the emptiness and overfit easily.

Why it matters: it explains why more features is not automatically better, why distance-based methods degrade in high dimensions, and why so much effort goes into dimensionality reduction (techniques like principal component analysis that squeeze the data into fewer, more meaningful dimensions) and feature selection (keeping only the columns that carry real signal). The honest takeaway: data is a finite resource spread across a space that grows brutally fast, so adding dimensions you can't densely fill often hurts.