Bias-Variance Tradeoff

The bias-variance tradeoff explains why a model's prediction errors come from two competing sources. Bias is error from a model too simple to capture the real pattern (it is systematically off); variance is error from a model so flexible that it chases the random noise in this particular sample and changes wildly if you collect new data.

The tension is that turning down one usually turns up the other: a very simple model has high bias but low variance, a very complex model the reverse, and the sweet spot is the middle complexity that minimizes total error on unseen data. This single idea is the lens behind underfitting, overfitting, and the reason we use regularization and cross-validation.