convnext

ConvNeXt (Liu et al., Facebook AI Research, 2022) is a pure convolutional network designed to answer a pointed question raised after Vision Transformers (ViT) appeared to surpass CNNs: was the transformer's self-attention mechanism actually responsible for its superior accuracy, or was it the many other modern training and design choices that came bundled with transformers? The authors started from a standard ResNet-50 and incrementally modernised it, step by step adopting the design and training recipes popularised by ViT and the Swin Transformer, while keeping the network entirely convolutional and measuring the accuracy gain at each step. The result, ConvNeXt, matches or exceeds comparable Vision Transformers on ImageNet and downstream tasks.

The modernisation steps are instructive precisely because none of them is exotic. The training recipe is upgraded (longer schedules, AdamW optimiser, strong augmentations like Mixup and RandAugment, stochastic depth, label smoothing). The macro design borrows from transformers: a 'patchify' stem that splits the image into non-overlapping patches with a 4x4 stride-4 convolution, and a stage compute ratio adjusted to match Swin. The micro design adopts large 7x7 depthwise convolutions (echoing the wide receptive field of self-attention), an inverted bottleneck like MobileNetV2 and transformer feed-forward blocks, far fewer activation and normalization layers, GELU instead of ReLU, and LayerNorm instead of BatchNorm. Each change adds a fraction of a percent, and together they close the gap with transformers.

ConvNeXt's importance is as much rhetorical as architectural. It provided strong evidence that much of the Vision Transformer's advantage came from training methodology and macro-architectural choices rather than from self-attention being fundamentally necessary, demonstrating that a well-modernised pure CNN remains highly competitive. This reframed the CNN-versus-transformer debate as less of a binary and more of a convergence: the two families have borrowed heavily from each other. A follow-up, ConvNeXt V2 (2023), added a fully convolutional masked-autoencoder self-supervised pretraining scheme and a global response normalization layer, further improving results. ConvNeXt is now a standard strong baseline backbone for modern vision pipelines.