inductive bias

Inductive bias is the set of built-in assumptions a model makes before it sees any data — the prior beliefs baked into its architecture about what kinds of solutions are plausible. Every learner needs some: data alone never uniquely determines a function, so the architecture must lean toward some answers over others. The right bias makes a model learn fast from few examples; the wrong bias caps what it can ever represent. For vision, the central question is how much spatial prior to build in, and convolutional networks and transformers sit at opposite ends.

Convolutional networks carry strong, vision-specific biases. Locality assumes nearby pixels matter most, so a CNN looks through small windows. Translation equivariance assumes a feature is the same wherever it appears, enforced by sliding the same shared filter across the whole image (weight sharing). A coarse-to-fine hierarchy assumes objects are composed of parts of parts, enforced by stacking and pooling. These priors are usually true of natural images, so CNNs learn efficiently even from modest datasets — the architecture has pre-solved much of the problem.

A pure vision transformer deliberately discards almost all of this. After patch embedding it sees an unordered set of tokens; self-attention is global and permutation-equivariant, treating a far patch and a near patch identically, and the only weak spatial prior is that pixels within a single patch are processed together, plus whatever the position embeddings learn. With so little built in, the ViT must learn locality, translation behaviour and hierarchy from the data itself — which is exactly why it needs large-scale pretraining to match CNNs, and why on small data it overfits or underperforms.

This frames the whole field's design space as a bias-versus-data trade-off (and connects to the bias-variance idea). You can supply the missing structure in several ways: more data and augmentation (DeiT), a convolutional stem or interleaved convolutions (hybrids), relative or local windowed attention and a token hierarchy (Swin), or distillation from a CNN teacher. The art of modern vision-transformer design is choosing how much inductive bias to reintroduce so that capacity and data budget are balanced — too much bias wastes the transformer's flexibility, too little wastes data.