Standing on Giants: Transfer Learning & Fine-Tuning

Why train from scratch? The case for transfer learning

Across this track you learned how a model learns: the training loop, optimizers like Adam, normalization to keep gradients healthy, and the regularizers that fight overfitting. With all of that, you could take a fresh network of random weights and train it on your own images. But there is almost always a smarter move. Someone has very likely already trained a giant model on millions of images, and that model has learned to see in a general way. We can borrow its eyes.

Here is the analogy. Training from random weights is like raising a newborn from birth and hoping it becomes a painter: it must first learn what edges and colours even are, then textures, then shapes, then objects — years of work. Reusing a pretrained model is like hiring an experienced artist. They already know edges, shading, perspective, and what a face or a wheel looks like; you just teach them your particular subject. Models trained on huge datasets such as ImageNet (1.2 million labelled photos), or trained self-supervised on web-scale images, have learned exactly these general visual features in their early and middle layers — and those features transfer far beyond the original task.

Reusing those learned features is what we call transfer learning, and it pays off three ways: it is dramatically cheaper (hours on one GPU instead of weeks on hundreds), it needs far less of your own labelled data, and it usually reaches higher accuracy than training from scratch — because the pretrained features are better than anything your small dataset could teach from zero. Two concrete strategies follow, and the rest of this guide is built around them: feature extraction (freeze the borrowed model, train only a new head) and fine-tuning (gently keep training the borrowed weights too).

Feature extraction: freeze the backbone, train a new head

The simplest strategy treats the pretrained network as a fixed feature machine. We keep its body — the backbone that turns an image into a rich feature vector — and we freeze it. Freezing has a precise mechanical meaning: for every weight in the backbone we set `requires_grad = False`, so during backpropagation the optimizer computes no gradient for it and never updates it. The backbone becomes a constant function: feed an image in, get the same high-quality features out, every epoch. Only a new piece on the end actually learns.

That new piece is a fresh, small head: we delete the original classifier (the part hard-wired to the source task's classes) and attach our own. Because only the head trains, this approach has very few trainable parameters, so it is fast, cheap, and — crucially — hard to overfit. Recall from guide 4 that overfitting gets worse as a model has more free parameters relative to data; freezing the backbone slashes that count to almost nothing, which is exactly why feature extraction shines when your dataset is small and similar to the source domain. The generic features still fit, and there is little capacity to memorise noise.

Concrete example: you have 5 classes (say, five species of bird) and only a few hundred photos each — far too little to train a deep network from scratch. Take a ResNet-50 pretrained on ImageNet's 1000 classes, freeze all of it, throw away its 1000-way classifier, and bolt on a single new linear layer that maps the 2048-dimensional feature vector to your 5 classes. You now train only that one layer. The code below shows every step.

import torch, torch.nn as nn
from torchvision import models

# 1. Load a backbone pretrained on ImageNet (1000 classes)
model = models.resnet50(weights="IMAGENET1K_V2")

# 2. FREEZE every backbone parameter: no gradient, no update
for param in model.parameters():
    param.requires_grad = False          # frozen -> autograd skips it

# 3. Replace the 1000-class head with a fresh 5-class head.
#    A newly created layer has requires_grad=True by default.
num_features = model.fc.in_features       # 2048 for ResNet-50
model.fc = nn.Linear(num_features, 5)     # our 5 classes

# 4. Only the new head's parameters are trainable
trainable = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.Adam(trainable, lr=1e-3)
# ...a standard training loop now updates ONLY model.fc

Fine-tuning: gently re-shaping pretrained weights

Feature extraction borrows the pretrained features exactly as they are. Fine-tuning goes further: we unfreeze some or all of the backbone and keep training it on our data, so the borrowed features can re-shape themselves to fit our task. The catch is that those weights are already very good. If we update them with the same aggressive step size we would use for random weights, the first few noisy gradients from our small dataset would shove them far away from their carefully learned values — we would overwrite the very knowledge we came to borrow. That failure has a name: catastrophic forgetting.

The fix is a much smaller learning rate. Whereas from-scratch training might use a base LR around 1e-1 (SGD) or 1e-3 (Adam), fine-tuning typically uses something 10 to 100 times smaller — just enough to nudge the weights, not to rewrite them. Think of it as tuning an already well-built instrument: tiny turns of the peg, never a full unwind. A small LR keeps each step inside the neighbourhood of the good pretrained solution, so the model adapts to your task while preserving what it already knows.

Practitioners pull several levers together. (1) A smaller base LR, as above. (2) Layer-wise (discriminative) learning rates: early layers detect universal edges and textures that almost never need to change, so they get a tiny LR; later, task-specific layers get a larger one. (3) Gradual unfreezing: train the head first, then unfreeze the top block, then the next, working downward — so the network stabilises before its delicate early layers ever move. (4) Short schedules: fine-tuning needs only a few epochs, usually with a warmup then a decay (see your guide on the learning rate schedule). The most precise of these levers is the layer-wise rate, which we can write as a formula.

Layer-wise (discriminative) learning rate: each layer's LR decays as you move toward the input.

Let's read it symbol by symbol. l is the layer index, numbered so that l=1 is the earliest layer (closest to the input pixels) and l=L is the last layer (closest to the output). L is the total number of layers. \eta_{\text{base}} (the Greek letter eta) is the learning rate we want for the final layer — our largest, fastest-adapting rate. \xi (xi) is a decay factor below 1, say 0.8. The exponent is L-l: for the last layer l=L, the exponent is 0, so \xi^0=1 and that layer gets the full \eta_{\text{base}}; for earlier layers the exponent grows, so the factor is raised to a higher power and the rate shrinks. Concretely with \eta_{\text{base}}=10^{-3}, \xi=0.8, L=5: the last layer gets 10^{-3}, then 8\times10^{-4}, 6.4\times10^{-4}, 5.1\times10^{-4}, and the earliest layer gets 10^{-3}\cdot0.8^{4}=4.1\times10^{-4} — under half the head's rate. The intuition: protect the generic low-level features learned during pretraining (barely move them), while letting the high-level, task-specific layers adapt freely.

# Discriminative / layer-wise learning rates.
# Order groups from early (generic) to late (task-specific),
# then give group l the rate eta_base * xi^(L - l).
eta_base, xi = 1e-3, 0.8
layer_groups = [model.layer1, model.layer2,
                model.layer3, model.layer4, model.fc]
L = len(layer_groups)

param_groups = []
for l, group in enumerate(layer_groups, start=1):   # l = 1..L
    lr_l = eta_base * (xi ** (L - l))               # earlier -> smaller
    param_groups.append({"params": group.parameters(), "lr": lr_l})

optimizer = torch.optim.AdamW(param_groups, weight_decay=1e-4)

How much to fine-tune? Matching strategy to your data

How aggressively you should fine-tune is not a matter of taste — it follows from your data. There are two axes to weigh. The first is dataset size: with few examples you can safely train only a few parameters; with many you can afford to move the whole network. The second is similarity to the source domain: if your images look like the pretrained model's images (everyday photos, like ImageNet), its features fit almost out of the box; if they look nothing like them (X-rays, satellite tiles, microscope slides), the deeper features may need real reshaping. Crossing these two axes gives four quadrants, and transfer learning has a standard recommendation for each.

1. Small + similar: freeze the backbone, train only a new head (pure feature extraction). You have too little data to safely move millions of weights, but the features already fit — so don't risk them.
2. Large + similar: fine-tune more layers, or all of them, with a small learning rate. You have enough data to adapt the whole network safely, and the small remaining domain gap is best closed by gentle, full fine-tuning.
3. Small + different: freeze the early generic layers, fine-tune only the later layers carefully, and augment heavily. The low-level edges/textures still transfer, but the high-level features must change — yet little data means you must move them cautiously and lean on augmentation.
4. Large + different: fine-tune the whole network (you might even consider from-scratch here). This is the highest-risk, highest-reward case: plenty of data lets you reshape everything, and the big domain gap means there is a lot to gain — but also the most that can go wrong.

Notice the through-line: the less data and the more different your domain, the more you should freeze and the harder you should regularize. Domain shift is real and easy to underestimate — medical scans and satellite imagery genuinely look nothing like the cat-and-dog photos of ImageNet, so do not assume an ImageNet backbone's deep features are right for them; the early edge/texture detectors usually still help, but the later 'this-is-a-golden-retriever' features often do not. When in doubt, freeze more, unfreeze gradually, and watch your validation curve.

Avoiding the traps: catastrophic forgetting, LR, and BatchNorm stats

Fine-tuning looks deceptively easy — change one flag, lower the LR, press go — but a few specific traps catch almost everyone. Each ties straight back to earlier guides in this track, and each is easiest to remember as a chain: symptom you observe → cause underneath → fix that works.

(a) Catastrophic forgetting. Symptom: in the first epoch of fine-tuning, training and validation accuracy collapse far below where the frozen-head model already was. Cause: the learning rate is too high, so early noisy gradients blow the good pretrained weights apart before they can settle — exactly the overwriting problem from section 3. Fix: use a low base LR (10–100x smaller than from-scratch) and add a short warmup, where the LR starts near zero and ramps up over the first few hundred steps (guide 2). Warmup means the very first, least-reliable updates are tiny, so the model never lurches away from its pretrained starting point.

(b) Stale BatchNorm statistics. Recall from guide 3 that batch normalization behaves differently in train vs inference mode: in training it normalises each batch by that batch's own mean and variance while updating a running mean/variance; in inference it freezes and uses those stored running statistics. The trap: a pretrained model's running stats were collected on the source domain (ImageNet's brightness, colour, contrast). Symptom: oddly poor accuracy on your new domain even though loss looks fine in training mode. Cause: those frozen source-domain statistics don't match your data's distribution. Fix: keep BatchNorm in training mode during fine-tuning so it recomputes statistics on your target data, or freeze the BN layers entirely (often best when your batches are tiny and noisy).

(c) Regularization doesn't go away. Symptom: validation loss starts rising while training loss keeps falling — the classic overfitting fork from guide 4 — and it appears faster during fine-tuning because the model is already powerful and your target set is small. Cause: a high-capacity pretrained network can memorise a small target set quickly. Fix: keep weight decay on to pull weights toward small, simple values, and use early stopping to halt at the validation minimum and restore the best checkpoint. On small target sets, regularize harder than you would from scratch — the model's strength is exactly what makes it prone to overfit here.

The full recipe: a master-level training playbook

You now hold every ingredient from this track. Let's assemble them into one coherent, modern recipe — the sequence a strong practitioner actually follows to train a vision model end to end. Read it as an ordered checklist, and notice how each step is something you already understand from an earlier guide; the playbook is just the order in which they click together.

1. Pick a pretrained backbone (guide 5). Don't start from random weights — borrow a model that already knows how to see. This is your single biggest head start.
2. Build a data pipeline with strong data augmentation (guide 4). Random crops, flips, colour jitter, and mixing manufacture variety so the model generalises instead of memorising — vital because your target set is usually small.
3. Choose an optimizer: AdamW for fast, robust convergence, or SGD+momentum when you want the last drop of accuracy (guide 2). AdamW is the safe default for fine-tuning.
4. Set a warmup + cosine learning rate schedule (guide 2). Warmup protects the pretrained weights through the noisy first steps; cosine decay anneals the LR smoothly to a careful finish.
5. Lean on the architecture's normalization and residual connections (guide 3) to keep gradients healthy — but mind the BatchNorm-statistics trap from section 5 when you cross domains.
6. Add weight decay, and dropout if needed (guide 4). Regularize harder on small target sets, lighter on large ones.
7. Monitor the validation curve and use early stopping (guides 1 & 4). The validation loss, not the training loss, tells you when to stop and which checkpoint to keep.
8. Fine-tune with a low or layer-wise LR (guide 5). Optionally start with a frozen-head warmup, then gradually unfreeze, giving early layers a smaller rate than late ones.

# End-to-end transfer-learning recipe (sketch)
model = load_pretrained_backbone()                  # guide 5
model.head = new_head(num_classes)                  # guide 5

train_loader = DataLoader(dataset, transforms=[     # guide 4: augmentation
    random_crop, horizontal_flip, color_jitter, mixup])

optimizer = AdamW(model.parameters(),               # guides 2 & 4
                  lr=3e-4, weight_decay=1e-4)
scheduler = warmup_then_cosine(optimizer,           # guide 2
                  warmup_epochs=2, total_epochs=30)

best, patience, bad = float("inf"), 5, 0
for epoch in range(30):
    train_one_epoch(model, train_loader,            # guides 1-3
                    optimizer, scheduler)
    val = evaluate(model, val_loader)               # guide 1
    if val < best:                                  # checkpoint the best
        best, bad = val, 0
        save(model)
    else:
        bad += 1
        if bad >= patience:                         # guide 4: early stopping
            break

That is the whole journey. You can read a learning curve and reason about the training loop; you can pick and tune optimizers and schedules; you can keep gradients stable with normalization; you can make a model generalize; and now you can stand on the shoulders of giant pretrained models and shape them into something of your own. Put together, these five guides are exactly what it takes to train a strong vision model from start to finish — go build one.