The whole point was never the training set
In the earlier guides of this rung you built the learning engine: a loss function that measures error, and gradient descent that tweaks the knobs to drive that error down. So here is an uncomfortable question. If learning just means making training loss as small as possible, why not make it *zero*? Memorize every example perfectly and you have a model that never misses. Why is that not the goal?
Because we will never see the training set again. We collected it, but the model's real job is on data it has *never seen* — tomorrow's emails, next month's patients, the next user's photo. The thing we actually care about has a name: [[generalization|generalization]], performance on fresh data drawn from the same world. A model that memorizes its training examples but flops on new ones has [[overfitting|overfit]]. The opposite failure, a model too crude to capture even the training pattern, is [[underfitting|underfitting]].
A picture makes it vivid. Imagine scattered dots that roughly follow a gentle curve, with a little random noise. An underfit model draws a straight line through them — too stiff, missing the bend. A good model traces the gentle curve. An overfit model snakes through *every single dot*, wiggling wildly to hit each one — including the noise. On new dots, that wild snake is hopeless; the calm curve wins. Memorizing the noise is exactly the trap.
Bias and variance: two ways to be wrong
Underfitting and overfitting are the two ends of a single dial, and the classic way to name them is the [[bias-variance-tradeoff|bias-variance tradeoff]]. *Bias* is error from wrong assumptions — the model is too simple to bend the way the truth bends, so it is consistently off no matter how much data you give it. That straight line through a curve has high bias. *Variance* is error from sensitivity — the model contorts itself to whatever data it happened to see, so a different training sample would have produced a wildly different model. The snake through every dot has high variance.
The word *tradeoff* is the heart of it. Make a model more flexible — more layers, more parameters, a higher capacity — and you lower bias but raise variance. Make it simpler and you do the reverse. For decades the picture was a U-shaped curve: total error falls as you add capacity, bottoms out at a sweet spot, then climbs again as variance takes over. Your job was to find the bottom of the U.
Reading the learning curve
You cannot fix what you cannot see, so before any cure you need the diagnostic: the [[learning-curve|learning curve]]. The setup comes from an earlier rung — you split your data into a training and a validation set, train on the first, and watch the loss on *both* as training proceeds epoch by epoch. Two lines on one chart tell you almost everything about which failure mode you are in.
If both lines are high and flat, you are *underfitting* — the model lacks the capacity or training to capture the pattern; feed it more flexibility or train longer. If the training line keeps dropping while the validation line bottoms out and then turns *upward*, the gap between them is the tell-tale sign of *overfitting*: the model is now learning quirks of the training data that hurt it elsewhere. That growing gap is sometimes called the generalization gap, and shrinking it is the whole game of this guide.
for epoch in range(max_epochs):
train_one_epoch(model, train_data) # weights move
train_loss = evaluate(model, train_data)
val_loss = evaluate(model, val_data)
log(epoch, train_loss, val_loss)
# diagnosis, read off the two curves:
# both high -> underfitting
# gap widening -> overfitting
# val at minimum -> best point to stopRegularization: gently penalizing complexity
The deepest cure for overfitting is regularization: instead of asking the optimizer to *only* fit the data, you add a second term that prefers simpler models. The new objective is `loss = data_error + lambda * complexity`. The optimizer now balances two pulls — fit the examples, but stay simple — and `lambda` is the hyperparameter dial that sets how hard you push toward simplicity. Turn `lambda` to zero and regularization vanishes; turn it too high and you force underfitting.
How do we measure 'complexity'? The most common answer is the size of the weights. Big weights let a model react sharply to tiny input changes — exactly the wiggly snake behavior — so we penalize them. L2 regularization adds the sum of squared weights; it shrinks every weight smoothly toward zero without quite reaching it, spreading influence across many small weights. L1 regularization adds the sum of absolute values; it pushes many weights *exactly* to zero, effectively deleting features and giving a sparse, more interpretable model. The classic pairing of the two — ridge for L2 and lasso for L1 — comes straight from linear regression.
In deep learning you will hear L2 regularization called [[weight-decay|weight decay]], and the two are nearly the same idea seen from two angles. Penalizing squared weights in the loss is mathematically equivalent to multiplying every weight by a number slightly less than one on each gradient step — the weights gently 'decay' toward zero unless the data keeps pushing them back up. (With adaptive optimizers like the Adam optimizer the two forms diverge slightly, which is why 'decoupled' weight decay was invented — a nuance worth knowing exists.)
Early stopping and the broader toolkit
The learning curve hands you the simplest, cheapest regularizer of all: [[early-stopping|early stopping]]. Since validation loss bottoms out and then climbs, just *stop training at the bottom*. In practice you watch the validation loss, remember the best model seen so far, and quit when it has failed to improve for some 'patience' number of epochs — then roll back to that best checkpoint. You get the well-fit model from the middle of training, before the overfitting set in, essentially for free.
For neural networks specifically, the most powerful trick of all is [[dropout|dropout]]: during each training step you randomly switch off a fraction of neurons, forcing the network to spread its bets instead of relying on any one path. It is like a team that rehearses with random members missing — no single person can become a single point of failure. Closely related is gathering *more or more varied data*: data augmentation manufactures new training examples (flip the image, add noise, reword the sentence), which is often the highest-leverage anti-overfitting move because the cure attacks the root cause — too few examples relative to model capacity.
Putting it together — and a word of honesty
- Train and plot the learning curve. Both lines high and flat? You are underfitting — add capacity or train longer before worrying about regularization at all.
- See the validation line turn upward while training keeps falling? That gap is overfitting. Now reach for the toolkit.
- Get more or more varied data first if you can (augmentation included) — it attacks the root cause.
- Add weight decay (L2) and, for neural nets, dropout; tune their strength on the validation set, not the test set.
- Turn on early stopping so you always keep the best checkpoint. Open the test set once, at the very end, to report an honest number.
Step back and the unity is striking: regularization, early stopping, dropout, more data — every one of them is a way of telling the model *'do not trust your training data so completely.'* That humility is the whole secret. A learner that fits its examples perfectly has learned the past; a learner that resists fitting them perfectly has a shot at the future.
One honest caveat to carry forward. There is no single regularizer that is best for every problem — the no free lunch theorem guarantees it, and every choice you make bakes in an inductive bias, an assumption about what 'simple' should mean for your data. Regularization does not give you generalization for free; it trades a bias you can reason about for a reduction in variance you can measure. Used with that clear-eyed view, it is the most reliable lever in all of machine learning.