hybrid cnn-transformer

A hybrid CNN-transformer deliberately marries the two architectures so each covers the other's weakness. Convolutions are superb at cheap, local, translation-equivariant feature extraction but have a limited, fixed receptive field; self-attention models long-range, content-dependent relationships but is expensive and short on built-in spatial priors. The hybrid idea is simple: let convolutions do the early, local, high-resolution work and let attention do the global reasoning, getting locality and efficiency together with global context.

The most common and well-tested form is a convolutional stem: replace ViT's single linear patch-embedding with a few small convolution layers that downsample the image into the token grid. This injects locality and translation equivariance at the input and, as Xiao et al. (2021) showed in 'Early Convolutions Help Transformers See Better', dramatically stabilizes optimization — hybrid stems tolerate larger learning rates, are far less sensitive to hyperparameters, and converge faster than the raw linear patchifier. It is one of the cheapest reliable upgrades to a plain ViT.

Beyond the stem, hybrids interleave the two operators more deeply. Networks such as CvT add convolutions inside the token and the query/key/value projections; LeViT uses a convolutional pyramid for fast inference; MobileViT and EdgeNeXt blend mobile convolution blocks with transformer blocks for on-device efficiency; and CoAtNet stacks convolution stages followed by attention stages, arguing that convolution generalizes better with little data while attention has higher capacity, so ordering them captures the best of both and scales gracefully.

The underlying principle is a bias-versus-capacity trade-off. Pure ViTs have minimal inductive bias and need scale; pure CNNs have strong bias and a ceiling on global modelling. Hybrids let a designer dial in exactly as much convolutional prior as the data budget and task demand — which is why, in the small-to-medium data regime and in efficiency-constrained deployment, hybrid CNN-transformers are frequently the strongest practical choice.