overfitting and model selection (AIC/BIC)

Imagine a student who memorizes the exact answers to last year's exam instead of learning the subject. They ace the practice paper and then fail the real test, because they fitted the quirks of one specific paper rather than the underlying material. A statistical model can make the same mistake. Overfitting is building a model so elaborate that it chases the random noise in the training data, fitting it beautifully but predicting new data badly. Model selection is the disciplined art of avoiding this — choosing a model rich enough to capture the real signal but no richer.

Adding more variables or more flexibility always improves the fit to the data you trained on; the catch is that beyond a point you are fitting noise, not signal, and out-of-sample performance gets worse. The tools to balance fit against complexity are information criteria. AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) both take the model's log-likelihood and subtract a penalty for each extra parameter; you prefer the model with the lower score. BIC's penalty grows with the sample size, so it punishes complexity harder and tends to pick simpler models than AIC. The honest gold standard, though, is to test on data the model has never seen — a hold-out set or cross-validation, where you fit on part of the data and measure error on the rest.

For actuaries the stakes are real money. A pricing GLM with too many fine territory or vehicle categories will fit last year's losses superbly and then misprice next year's, charging some customers wrongly and losing the profitable ones. The whole point of AIC, BIC, and cross-validation is to find the model that will generalize, not the one that flatters the historical data. A blunt warning that experienced modelers repeat: the model that fits your data best is almost never the model that predicts the future best, and a parsimonious, interpretable model usually beats a sprawling one that no one can sanity-check.