How a Vision Model Learns: The Training Loop

What it really means to 'train' a vision model

Strip away the mystique and a vision model is just a function: you feed in an image, numbers flow through it, and out the other end comes a prediction — say, the probability that the picture is a cat versus a dog. What makes it special is that this function has an enormous number of tunable numbers inside it, called weights or parameters — often millions, sometimes billions. The exact values of those numbers are what make one model recognize cats brilliantly and another model output nonsense.

Here is an analogy to hold onto for the whole track. Imagine a giant mixing board with millions of knobs. Each knob is one weight. Turn the knobs one way and the model 'sounds' like cat-vs-dog chaos; turn them another way and clean, correct predictions come out. Training is the process of turning those knobs until the output matches what we want. Nobody turns them by hand — there are far too many — so we need an automatic, principled way to decide which knob to turn, in which direction, and by how much.

To turn the knobs sensibly we first need a way to measure how wrong the model currently is, as a single number. That number is the loss. A high loss means the model is making bad predictions; a low loss means it is mostly right. If we show the model a cat photo and it confidently says 'dog', the loss is large; if it says 'cat' with high confidence, the loss is small. The whole of training boils down to one mission: make the loss small.

One honest caveat before we start the hunt: a tiny loss on the photos we trained on is not the real goal. We want the model to recognize cats and dogs it has never seen. A model that memorizes the training photos but flunks new ones is suffering from overfitting — we will name it again at the end of this guide and devote a whole later guide to defeating it. For now, just keep in the back of your mind that 'low training loss' and 'genuinely good model' are related but not identical.

The loss landscape and the idea of gradient descent

Here is the key shift in perspective. The loss is not a fixed number — it depends on the weights. Change the weights and the loss changes. So we can think of the loss as a function of the weights: feed in a setting of all the knobs, get back a single loss value. If you could plot that function, every choice of weights would be a location, and the loss would be the height at that location. The result is a landscape — hills, ridges, and valleys — and we want to reach a deep valley, the lowest place we can find.

How do we head downhill when the landscape has millions of dimensions and we can't see it? Picture a hiker caught on a mountain in thick fog. They can't see the valley, but they can feel the slope under their feet — which way is steepest up. To descend, they simply step the opposite way, then stop, feel the slope again, and step again. That 'direction of steepest uphill' has a precise mathematical name: the gradient, written ∇L. Crucially, the gradient points uphill, toward larger loss — so to make the loss smaller, we move in the negative gradient direction.

The gradient-descent update rule — the heartbeat of training.

Let's read this, your first equation, very slowly — every symbol earns its place. θ (theta) is the vector holding all the weights at once; think of it as a single 'position' in weight-space, the full setting of every knob. The subscript t is the step number: θ_t is where we stand now, θ_{t+1} is where we stand after one step. ∇L(θ_t) is the gradient of the loss evaluated at our current position — a vector with one entry per weight, each saying 'increasing this weight by a hair changes the loss this much, in this direction'; together they point straight uphill. The minus sign flips uphill into downhill — that is what makes us descend. And η (eta) is the learning rate: a small positive number that scales how big a step we take. Put plainly, the rule says: *look which way is steepest uphill, then take a step of size η in the opposite direction.* Concretely, suppose one weight currently sits at 2.0, its gradient there is +4 (loss rises as that weight grows), and η = 0.1. The update gives 2.0 − 0.1 × 4 = 1.6 — the weight moved down, nudging the loss down with it. Repeat the rule thousands of times and θ drifts, step by downhill step, toward a valley where the loss is low.

Why 'stochastic'? Mini-batches and SGD

There is a hidden cost in that tidy update rule. The 'true' gradient ∇L is defined over the entire training set — to compute it exactly, you would have to run every single training image through the model, measure the loss on all of them, and average. With 50,000 images that is 50,000 forward passes just to take one step downhill. Modern datasets have millions of images; doing this for every step would make training agonizingly slow. We need a shortcut.

The shortcut is delightfully simple: instead of the whole dataset, grab a small random handful of images — a mini-batch — compute the gradient on just those, and step. The mini-batch gradient is a noisy but cheap approximation of the true one. Because each step uses a random sample, the method is called stochastic ('stochastic' just means 'random') gradient descent, or stochastic gradient descent (SGD). It is the workhorse that makes training deep networks feasible at all.

The mini-batch estimate of the full-dataset gradient.

Symbol by symbol: B is the current mini-batch — the small random set of examples we drew this step. |B| is the batch size, simply how many examples are in it (a typical value is 128). For one example i, the pair (x_i, y_i) is its image and its true label, and ℓ(x_i, y_i; θ) is the loss on that single example given the current weights θ (lowercase ℓ for one example, capital L for the whole set). The Σ sums the per-example gradients over the batch, and the 1/|B| turns that sum into an average. Why does averaging a random handful approximate the true gradient over millions of images? The same reason a political poll of 1,000 people estimates how a whole country will vote: a random sample, on average, looks like the population it was drawn from. Each example's gradient is a 'vote' for which way is uphill; any single vote is noisy, but the average of a fair random sample is an unbiased estimate of the true direction — right on target on average, just jittery from one batch to the next. Two more words you'll use constantly: one iteration (or step) is a single mini-batch update; one epoch is one full pass through the whole training set. The link is pure arithmetic — with 50,000 images and a batch of 128, one epoch is 50000 ÷ 128 = 390.6, which we round up to 391 steps per epoch (the last batch just holds the leftover 50000 − 390×128 = 80 images).

# One epoch of stochastic gradient descent (pseudocode)
eta = 0.1               # learning rate (the step size)
for x_batch, y_batch in loader:        # loader yields ~391 mini-batches
    preds = model(x_batch)             # forward pass: predictions for this batch
    loss  = loss_fn(preds, y_batch)    # a single number: how wrong we are now
    grads = gradient_of(loss, model.weights)  # noisy mini-batch gradient
    # The update rule: theta <- theta - eta * grad L
    for w, g in zip(model.weights, grads):
        w -= eta * g                   # step downhill
# Run this whole loop many times (many epochs) until the loss stops falling.

Choosing the learning rate: the single most important dial

If you tune only one thing, tune the learning rate η. It is the dial beginners get wrong most often, and a bad value can sink an otherwise perfect model. Go back to the foggy hiker: the learning rate is literally their step size. The gradient tells them which way is downhill; η decides how far they stride each time before stopping to feel the slope again.

Both extremes hurt, in opposite ways. If η is too large, each step overshoots the bottom of the valley and lands part-way up the far slope; the next step overshoots back; the loss bounces around without settling, or — worse — the steps grow until the loss blows up to infinity and your training prints NaN ('not a number', the sign of numerical overflow). If η is too small, every step is a timid shuffle: the loss does fall, but so slowly you'd wait days, and the search can stall in the first shallow dip it meets. The art is finding the largest step that still reliably goes down.

1. Loss shoots up, oscillates wildly, or hits NaN within a few steps → learning rate is TOO HIGH. Cut it by 3–10× and try again.
2. Loss drifts down almost imperceptibly, nearly a flat line → learning rate is TOO LOW (or the model isn't learning for another reason). Raise it by 3–10×.
3. Loss falls steadily and fairly smoothly, then levels off → learning rate is in a healthy range. This is what you want to see.

What counts as a sane starting value? It depends on the optimizer (the next guide introduces fancier ones than plain SGD), but as a rough map: plain SGD on vision models often likes something around 1e-1 to 1e-2, while the Adam optimizer you'll meet next usually wants something smaller, around 1e-3 to 1e-4. Try a value, watch the first dozen steps of the loss, and adjust by the checklist above. And one promise for the next guide: we rarely keep η fixed for the whole run. We typically start a bit larger to make fast progress, then shrink it over time so the late steps can settle gently into the valley — that's a learning-rate schedule, the star of the very next guide.

Batch size: speed, memory, and noise

The batch size |B| is a quieter dial than the learning rate, but it pulls three strings at once. Noise: a larger batch averages over more examples, so its gradient estimate is less noisy and points more reliably downhill — a smaller batch is jumpier. Hardware efficiency: GPUs love doing many images in parallel, so a bigger batch usually means more images processed per second, up to the point your memory runs out. Generalization: intriguingly, very large batches sometimes generalize slightly worse (they can settle into sharp, brittle valleys), while the extra noise of small batches can actually help the model cope with unseen images. There is no single 'best' batch size — it's a trade-off you tune.

Memory is usually the hard wall you hit first, so make it concrete. GPU memory during training holds two very different things. The weights (and the optimizer's bookkeeping for them) take a fixed amount no matter the batch size — a 25-million-parameter model needs the same space for its weights whether the batch is 1 or 1,000. The activations — every intermediate feature map the network computes on its way to a prediction, which must be stored for the gradient calculation — scale linearly with the batch size. So if a batch of 32 images fills, say, 6 GB of activations, a batch of 64 needs about 12 GB, and 128 about 24 GB, while the weight portion never budges. That is exactly why doubling the batch can suddenly tip you into an out-of-memory crash.

The linear scaling rule: when you change the batch size, scale the learning rate to match.

Batch size and learning rate are secretly coupled, and this rule of thumb captures it. Symbol by symbol: |B|_base is some reference batch size at which you already found a learning rate that works, η_base is that working learning rate, |B| is your new chosen batch size, and η_new is the learning rate to start with for it. The ratio |B|/|B|_base just asks 'how many times bigger is my new batch?' and scales η by the same factor. The intuition: a bigger batch gives a less noisy gradient, so you can trust it enough to take a proportionally bigger step. Worked example: if η_base = 0.1 worked at |B|_base = 256, then for |B| = 512 you'd start at η_new = 0.1 × (512/256) = 0.2. Two crucial caveats: this is a heuristic, a starting point — not a law of nature; it breaks down for very large batches. And after a big jump you almost always need warmup — beginning with a tiny learning rate for the first few hundred steps and ramping up to η_new — to stop those early, freshly-enlarged steps from blowing up. Both warmup and schedules are exactly what the next guide builds out.

Reading the training curve: signs of learning

The single most useful skill in this whole track is reading a training curve — a plot of the loss (or accuracy) against steps or epochs. It is the heartbeat monitor of your model. The healthiest possible shape is simple: the training loss falls steadily and fairly smoothly, steeply at first as the model learns the easy patterns, then more gently as it polishes the hard cases, finally flattening out near the bottom of its valley. If you see that, the machinery from the last four sections is working.

But a falling training loss alone can fool you. To see whether the model is truly learning rather than just memorizing, we split our data: most of it is the training set the model learns from, and a held-out slice it never trains on is the validation set, used only to check progress on fresh examples. Now watch both losses together. The first and most important warning sign in all of deep learning: the training loss keeps falling while the validation loss bottoms out and starts climbing. That growing gap means the model is memorizing training photos instead of learning what cats and dogs actually look like — it is overfitting. Spotting it is today's lesson; the full toolkit to cure it (more data, augmentation, regularization, early stopping) gets a dedicated guide later in this track.

1. Loss explodes upward or hits NaN → diverging. Cause: learning rate too high (or no warmup after a big batch). Lower η.
2. Loss is nearly flat from the start → not learning. Cause: learning rate too low, or a bug in data/model wiring. Raise η, then check the pipeline.
3. Loss trends down but is very jagged/spiky → too noisy. Cause: batch too small or η a touch high. Increase batch size or lower η slightly.
4. Training loss falls but validation loss rises (a widening gap) → overfitting. Cause: model memorizing. The cure is guide 4 of this track.
5. Both losses fall together and flatten near a low value → healthy training. You're done (or ready to push for more).

Step back and see what you now hold. A vision model is a function with millions of knobs; the loss scores how wrong it is; the gradient points uphill so we step the opposite way; we estimate that gradient cheaply from random mini-batches (SGD); the learning rate sets our step size and the batch size trades off noise, speed, and memory; and the training curve tells us, at a glance, whether it's all working. That loop — predict, measure, step, repeat — is the engine under every model in this track. The next four guides each upgrade one part of it: smarter steps (momentum & Adam), healthier gradients (normalization), models that generalize (beating overfitting), and a giant head start (transfer learning). You now have the map; let's make the engine run better.