transfer learning

Training a strong vision model from scratch needs a vast labeled dataset and a lot of compute, which most projects do not have. Transfer learning sidesteps this by reusing the knowledge a model already gained from learning a large, general task, and applying it to your smaller, specific one. The intuition is that you do not re-learn how to see from zero for every new problem: a model that learned to recognize a thousand object categories has already discovered how to detect edges, textures, shapes, and parts, and those visual building blocks are useful for almost any image task — recognizing your factory's defects, classifying medical scans, or detecting a new kind of object.

The mechanism is feature reuse. You take a model pretrained on a large source dataset (classically ImageNet, today often web-scale data via models like CLIP or self-supervised methods like DINO and MAE), keep its learned weights, and adapt it to the target task instead of starting from random initialization. The reason this works is that the features a deep network learns are hierarchical and increasingly task-specific with depth: early layers learn generic, transferable primitives (edge and color detectors that look nearly identical across all vision models), middle layers learn textures and motifs, and only the last layers learn the source task's specific categories. Transfer keeps the generic lower layers and replaces or adjusts the specific upper ones.

The simplest form is to treat the pretrained network as a fixed feature extractor: freeze all its weights, run your images through it to get feature vectors, and train only a new lightweight classifier (often a single linear layer — 'linear probing') on top. This is fast, needs little data, and is hard to overfit. The richer form is fine-tuning, where you also continue training some or all of the pretrained weights at a small learning rate so the features adapt to your domain. Which to choose depends mainly on how much target data you have and how far your domain is from the source.

Transfer learning helps most when your target dataset is small and reasonably related to the source domain — exactly the common case — where it can turn an impossible task into a routine one and dramatically cut training time. Its limits appear when the domains diverge sharply (natural-image features transfer poorly to, say, raw radar or some medical modalities), where the generic early-layer features still help but the later ones may need substantial retraining or even hurt (negative transfer). Foundation models pretrained on enormous, diverse data have pushed those limits outward and made transfer the default starting point for nearly all of modern computer vision.