Designing the Network Itself: NAS, RegNet, and ConvNeXt

From hand-design to search

Take a moment to look back at the road we have travelled. LeNet taught a network to read digits; AlexNet scaled that idea up and won ImageNet; VGG and Inception made networks deeper and cheaper; ResNet's skip connections let us train a hundred layers; and EfficientNet found a principled way to balance depth, width, and resolution. Every single one of those architectures came from the same source: a human being, sitting down, drawing boxes, running experiments, and trusting their intuition. The progress was real, but it was slow, and it leaned on rare expertise.

So here is the natural next question, and it is the one this final guide is about: instead of asking 'which network should I design?', what if we asked 'can a machine design the network for us?' This is the meta-level of everything you have learned. We are no longer choosing between architectures — we are trying to discover the process that produces good architectures in the first place. The headline idea is called neural architecture search, and the field has since matured into something even more ambitious: searching not for one network, but for the design rules that describe whole families of good networks.

Neural Architecture Search: machines design networks

Any neural architecture search system, no matter how fancy, is really just three moving parts working together. First, a search space: the set of all networks we are allowed to build. Second, a search strategy: how we pick which candidates to try. Third, a performance estimation strategy: how we score a candidate once we have one. Get a feel for these three and you understand NAS, because everything else is engineering detail stacked on top.

The search space is the menu. It lists the operations a block may use — a 3×3 convolution, a 5×5 depthwise convolution, a pooling layer, a skip connection, and so on — plus the rules for how blocks may connect to one another. A small, well-chosen menu makes the search fast but might exclude the best dish; a huge menu can contain brilliant networks but is astronomically large. A classic trick is to search for one good cell (a small reusable sub-graph) and then stack copies of it, exactly the way VGG and ResNet repeat a single block — so the thing being searched is the cell, not the whole net.

The search strategy is how we explore that menu, and here a cooking analogy helps. Imagine a chef trying to invent the best recipe. A reinforcement-learning chef treats cooking as a game: a controller proposes a recipe, tastes the result, and is rewarded when it tastes good, gradually learning which choices earn praise. An evolutionary chef keeps a population of recipes, throws out the worst, and breeds and mutates the best into the next generation. A differentiable chef — the method called DARTS — is cleverer still: instead of picking one ingredient at a time, it blends all ingredients together with adjustable weights, then uses ordinary gradient descent to dial up the good ones and dial down the bad ones, turning a discrete search into a smooth optimization we already know how to solve.

The performance estimation strategy answers a brutal practical problem: to know if a candidate is good, you would normally have to train it fully, which can take days on many GPUs — and a search may want to try thousands of candidates. Nobody can afford that. So we estimate instead: train for only a few epochs and extrapolate, train on a smaller dataset or a shrunken network, or — the big idea — share weights across candidates so that a single 'supernetwork' is trained once and every sub-architecture inherits its weights for free. These proxies are imperfect, but they turn an impossible bill into a payable one. The very first NAS results famously burned thousands of GPU-days; these cheaper proxies are what dragged the cost down to something a normal lab can run.

Why is NAS so expensive in principle? Because it is really two optimization problems nested inside each other — a bilevel problem. We are searching over both what the network is and how it is trained, at the same time. We can write the goal like this:

NAS as a bilevel objective: an outer search over architectures wraps an inner training of weights.

Read it from the inside out. The symbol α (alpha) is the architecture choice — the menu selections that say 'use a 5×5 depthwise conv here, a skip connection there.' The symbol w is the ordinary network weights, the numbers gradient descent learns. The inner problem, w*(α) = argmin_w L_train(w, α), says: once you have fixed an architecture α, find the weights w that minimise the training loss — that is just training a network the normal way, and the winning weights are written w*(α), 'the best weights for that architecture.' The outer problem, min_α L_val(w*(α), α), then says: among all architectures, pick the α whose trained model gets the lowest validation loss. The sting is in the dependency — to evaluate even one α in the outer loop, you must first solve the entire inner loop, i.e. train a whole network. An optimization inside an optimization: that nesting is exactly why naive NAS was so costly. And this is not just theory — the base network of efficientnet from guide 4, the small 'B0' that compound scaling then enlarges, was itself discovered by NAS. So when you admired EfficientNet's clean scaling rule, you were already looking at a machine-designed network.

RegNet: designing design spaces

Neural architecture search gives you a champion — one specific network, often excellent but also opaque: you get a winner without learning why it won. RegNet (from Facebook AI Research, 2020) asks a deeper question. Instead of searching for the single best network, it searches for the best design space — a simple set of rules that describes an entire population of strong networks. The slogan is: NAS finds an instance; RegNet finds the pattern.

How do you search for a design space rather than a network? You start with a very loose space (almost anything goes), sample many random networks from it, train them briefly, and look at the distribution of how well they do. Then you tighten the space — add one constraint — and check whether the whole population got better. Constraint by constraint, you sculpt a broad, messy space down to a narrow one where almost every network you can draw is good. What survives is not a network but a recipe — and, crucially, a recipe humans can read.

The headline finding is beautifully simple. When the researchers looked at the good networks that survived, the number of channels (the width) in successive blocks was not random, and it was not hand-tuned — it grew along a straight line. Concretely, the ideal per-block width follows a quantized linear rule:

Block width grows as a straight line in the block index j, then each value is rounded to a tidy quantized width.

Every symbol here is concrete. j is the block index — 0 for the first block, 1 for the second, and so on as we go deeper. w₀ (read 'w-naught') is the initial width: how many channels the very first block has. wₐ is the slope of the line — how many extra channels we add per block as the network deepens. And u_j is the resulting continuous, not-yet-rounded width for block j. For a quick number: if w₀ = 24 and wₐ = 16, then block 0 wants 24 channels, block 1 wants 24 + 16 = 40, block 2 wants 56, and so on — a clean staircase. Because real layers need whole, hardware-friendly channel counts, each u_j is then quantized: rounded and grouped so that nearby blocks share one width, carving the line into a few neat stages. The profound part is the contrast: where NAS hands you a chaotic graph you cannot read, RegNet says the entire depth-versus-width story of a great network is captured by just two numbers, w₀ and wₐ — something a human can read, remember, and reason about.

The Transformer challenge

To set up the finale, we have to step briefly outside the world of pure convolutions. Around 2020, a rival architecture arrived from the language world: the Vision Transformer, or ViT. On large enough datasets it began to beat convolutional networks at image classification, and that lit a fire under the whole field. If a model with no convolutions at all could win, people asked, were convolutions — the very idea this entire track is built on — about to become obsolete?

Here is the gist, kept deliberately light since a later track will teach attention properly. A ViT chops the image into a grid of small square patches — say 16×16 pixels each — and treats each patch like a 'word.' It then uses a mechanism called attention to let every patch look at, and exchange information with, every other patch in the image, no matter how far apart they sit. Compare that to a convolution, whose little kernel only ever sees a small neighbourhood at a time. The Transformer's appeal is exactly this global reach from the very first layer.

But here is the subtle question the ConvNeXt authors zeroed in on. Vision Transformers did not arrive alone — they came bundled with a whole new bag of training and design habits: patch-based stems, far fewer normalization layers, a different activation function, heavy data augmentation, longer training schedules. So when a ViT beat a ResNet, what actually won? Was it the attention mechanism replacing convolution? Or was the convolution fine all along, and the ResNet was simply being judged with an outdated design recipe? The next section answers exactly that.

ConvNeXt: modernizing the ConvNet

ConvNeXt (2022) is the satisfying answer, and it is the perfect note to end this track on because it is pure synthesis — every idea you need, you already have. The authors ran a disciplined experiment. They took a plain, ordinary ResNet-50 — the exact network from guide 3, residual connections and all — and then, one change at a time, applied the design choices that came with the Transformer era, measuring the accuracy after each tweak. No new magic operation; just modernizing an old design, step by step.

1. Patchify the stem. Replace ResNet's gentle 7×7 stride-2 entry with a single 4×4 stride-4 convolution that slices the image into non-overlapping patches — copying the ViT's patch idea exactly.
2. Use depthwise convolutions with LARGE kernels. Swap the dense 3×3 convs for depthwise convs (one filter per channel, from guide 4's MobileNet) and grow the kernel to 7×7, so each layer sees a wide neighbourhood, the way attention does.
3. Adopt an inverted bottleneck. Instead of squeeze-then-expand, expand the channels in the middle of the block and project back down — the very shape MobileNetV2 introduced in guide 4.
4. Use fewer normalization and activation layers. A ResNet block sprinkles a normalization and a ReLU after every conv; ConvNeXt keeps just one normalization and one activation per block, matching the spare style of a Transformer block.
5. Switch to LayerNorm and GELU. Replace BatchNorm with LayerNorm and ReLU with the smoother GELU activation — again, the exact choices the Transformer brought along.

# A ConvNeXt block (PyTorch-style pseudocode).
# Input and output both have shape (N, C, H, W): N images, C channels.
def convnext_block(x):
    shortcut = x
    # 1) depthwise conv with a LARGE 7x7 kernel: cheap spatial mixing, stays at C channels
    x = depthwise_conv(x, kernel=7, padding=3)
    x = layer_norm(x)                 # ONE normalization, LayerNorm (not BatchNorm)
    # 2) inverted bottleneck: widen 4x with a 1x1 conv, then narrow back
    x = pointwise_conv(x, out=4 * C)  # 1x1 conv expands the channels
    x = gelu(x)                       # ONE activation, GELU (not ReLU)
    x = pointwise_conv(x, out=C)      # 1x1 conv projects back down
    # 3) residual add: the idea straight from ResNet, untouched
    return shortcut + x

When the dust settled, this purely convolutional network matched — and on some benchmarks beat — the Vision Transformers it was racing, at the same compute budget. That yields the profound takeaway of the whole guide: convolution was never the bottleneck. The operation that powered everything from LeNet to ResNet was fine all along. What had aged was the recipe around it — the stem, the kernel sizes, the bottleneck shape, the number and kind of normalization and activation layers. Update the recipe, keep the convolution, and the gap to Transformers closes. ConvNeXt is essentially a love letter that folds the residual lessons of guide 3 and the efficiency lessons of guide 4 into one modern, fast, simple design.

The big picture: principles of architecture design

Step all the way back and the entire track collapses into a short list of recurring design levers — the handful of knobs that, in different combinations, produced every architecture we met. Depth: stack more layers for richer features (VGG, ResNet). Width: more channels per layer for more capacity (Wide ResNets, RegNet). Resolution: feed larger images for finer detail (EfficientNet's third dimension). Skip / residual connections: let gradients and signals flow past layers so very deep nets can train (ResNet). Cardinality and grouping: split work into parallel groups, including depthwise convs (ResNeXt, MobileNet). Channel attention: let the network reweight its own channels by importance (Squeeze-and-Excitation). Efficiency factorizations: decompose expensive operations into cheap ones (1×1 convs, depthwise-separable convs). And now, two meta-levers: automated search and principled design spaces.

So how do you actually choose one for a real task? The honest answer is that it is governed less by chasing the top of a leaderboard and more by your compute and latency budget. A sensible default workflow:

1. Running on a phone, browser, or embedded device? Reach for MobileNet or a small EfficientNet — they are built to hit a latency budget with depthwise-separable convolutions and compound scaling.
2. Running on a server or GPU and want robust, well-understood accuracy? A ResNet is still a superb, hard-to-beat baseline, and a modern ConvNeXt pushes the same convolutional family to the accuracy frontier.
3. Need the best accuracy-per-FLOP across a range of sizes? Use a searched or design-space family (EfficientNet, RegNet) and pick the member that fits your budget by turning its scaling dials.
4. Always measure on your hardware and your data. Theoretical FLOPs are not wall-clock latency, and a smaller, well-suited model often beats a bigger, more famous one.

That lens is the real prize of this whole track. You began by watching a tiny network learn to read digits; you end able to reason about machine-designed networks and modernized ConvNets that trade blows with Transformers. The specific architectures will keep changing — but the way of seeing them, the habit of asking which principle is doing the work, is yours to keep. Go read the frontier.