bias-variance tradeoff

The bias-variance tradeoff explains why a model can fail in two opposite ways, and why fixing one tends to worsen the other. Bias is error from being too simple — the model is too rigid to capture the real pattern, like drawing a straight line through data that clearly curves. Variance is error from being too sensitive — the model swings wildly depending on which examples it happened to train on, chasing every bump and wiggle. The art of machine learning is finding the sweet spot between these two.

An everyday picture: imagine archers shooting at a target. A high-bias archer's arrows land tightly clustered, but consistently off to one side — steady and steadily wrong. A high-variance archer's arrows scatter all around the bullseye — sometimes dead-on, sometimes way off, with no reliability. You want neither. You want arrows that are both close to the center and tightly grouped, and usually you cannot have perfect amounts of both at once.

Here is the tradeoff itself. Make a model more complex — more parameters, more flexibility — and you lower bias but raise variance: it can now fit the true pattern, but it also starts fitting noise (overfitting). Make it simpler and you lower variance but raise bias: it stops chasing noise, but may miss the real pattern (underfitting). The classic goal is to balance the two to minimize total error on unseen data. The honest modern caveat: very large models trained on huge data sometimes defy the simple version of this curve, so treat it as a sturdy intuition, not an iron law.