From LeNet to AlexNet: How ConvNets Learned to See

Same bricks, different buildings

Before we meet our first famous network, take a breath: you already own every brick we are about to use. In earlier tracks you learned the convolution (a small filter that slides over an image looking for a local pattern), pooling (a downsampler that shrinks a feature map while keeping the strongest signals), the ReLU nonlinearity, and the idea of a feature map (the grid of responses a filter produces). You also saw the basic assembly line: convolution to detect, pooling to summarise, and finally one or two fully-connected layers that turn all those features into a class decision. Nothing in this track replaces those bricks. We are simply going to stack them in smarter and smarter ways.

That is the single most important idea of this whole track, so let us say it plainly: an architecture is not a new operation, it is a choice of how to stack the bricks you already have. Think of baking. Flour, eggs, sugar and butter are fixed ingredients—just like convolution, pooling and ReLU. But the recipe decides everything: the same four ingredients become a thin pancake, a chewy cookie, or a tall layer cake depending only on how much of each, in what order, and how long you cook it. A CNN architecture is exactly that recipe: how many conv layers, how wide, in what order, with which tricks. Change the recipe and the same bricks learn to read a postage-stamp digit one day and recognise a thousand kinds of animals the next.

To keep you oriented, here is the map of the five guides in this track, laid out as a timeline of big ideas. (1) 1998 — LeNet, the tiny digit reader writes down the template. (2) 2012 — AlexNet scales that template up and wins ImageNet, lighting the deep-learning fuse. (3) Then VGG and Inception go deeper and cheaper. (4) ResNet smashes the depth ceiling and trains networks 100+ layers deep. (5) Finally we chase efficiency (pocket-sized models) and even let machines design the networks themselves. Every step on this road is a new recipe for the same bricks—so once you truly understand the first two blueprints, the rest will feel like variations on a theme you already know.

LeNet-5: the template that started it all

Our story starts in the late 1990s with a very practical problem: banks needed machines to read the handwritten digits on millions of checks. Yann LeCun and colleagues built LeNet-5, a small convolutional network that read those digits reliably enough to be deployed for real. It is tiny by today's standards—a handful of layers and about sixty thousand parameters—but its layout is the ancestor of every network in this track. Learn LeNet well and you have read the blueprint that AlexNet, VGG and ResNet all inherit.

1. Convolution: a layer of small 5×5 filters slides over the 32×32 input and lights up wherever it finds simple local patterns—strokes, edges, corners.
2. Pooling (subsampling): a 2×2 average-pool halves the spatial size, keeping the gist of each neighbourhood while throwing away the exact pixel position.
3. Convolution again: a second set of filters now sees the pooled features and learns to combine edges into bigger motifs—loops, junctions, parts of a digit.
4. Pooling again: another 2×2 pool shrinks the map once more, so later layers see a compact, high-level summary.
5. Fully-connected → fully-connected: every remaining feature is wired to a dense layer, then another, and the final layer outputs ten scores—one per digit, 0 through 9.

Why does this particular order make sense? Because vision is naturally hierarchical. The earliest layers can only see a tiny patch at a time, so all they can possibly detect are small local patterns like edges and strokes. Pooling then blurs away the exact position ('there is an edge roughly here' is enough), which both shrinks the data and makes the network tolerant to small shifts. Stacking a second conv on top lets it combine nearby edges into larger parts, and a third stage would combine parts into whole shapes. Only at the very end, once the network has built up rich, position-tolerant features, do the fully-connected layers look at everything at once and decide: 'all things considered, this is a 7.' Detect locally, summarise, combine, then decide globally—that is the logic baked into the ordering.

Now let us make the shrinking precise, because watching the numbers is the best way to feel how a CNN works. When a convolution slides a square filter over a map, the output is smaller than the input (the filter cannot hang off the edge), and exactly how much smaller is given by one tidy formula:

The convolution output-size formula (applied along one dimension; height works the same way).

Let us name every symbol in plain words. W is the input width (how many pixels across the incoming map is). K is the kernel size (the filter is K×K; here K = 5). P is the padding (how many extra zero-pixels you glue around the border; LeNet uses P = 0). S is the stride (how many pixels the filter jumps each step; LeNet uses S = 1). The little ⌊ ⌋ means 'round down to a whole number,' because you can only have a whole number of output pixels. The one-line intuition: the numerator W − K + 2P is simply 'how far the kernel's top-left corner can travel,' dividing by S counts the steps it takes, and the +1 counts the starting position itself. Plug in LeNet's first conv layer: out = ⌊(32 − 5 + 0)/1⌋ + 1 = ⌊27⌋ + 1 = 28. So a 32×32 input becomes a 28×28 feature map—exactly the shrink LeNet uses—and you just watched the number happen.

# Track how a LeNet input changes shape, layer by layer.
# Format: (channels, height, width)

x = (1, 32, 32)     # one grayscale digit, 32x32 pixels

# Conv1: 6 filters, 5x5 kernel, stride 1, no padding
# out = floor((32 - 5 + 0)/1) + 1 = 28
x = (6, 28, 28)     # spatial 32 -> 28, depth 1 -> 6

# Pool1: 2x2 average pool, stride 2  (halves H and W)
x = (6, 14, 14)

# Conv2: 16 filters, 5x5  ->  floor((14 - 5)/1) + 1 = 10
x = (16, 10, 10)    # depth grows 6 -> 16

# Pool2: 2x2 average pool, stride 2
x = (16, 5, 5)

# Flatten: 16 * 5 * 5 = 400 numbers
# FC: 400 -> 120 -> 84 -> 10 class scores (digits 0-9)

Notice the two trends in that trace, because they are the heartbeat of almost every CNN: spatial size shrinks (32 → 28 → 14 → 10 → 5) while channel depth grows (1 → 6 → 16). The network trades 'where exactly' for 'what, and how richly.' One honest period detail: LeNet predates several modern habits. It used tanh/sigmoid activations (smooth S-shaped squashing functions) rather than ReLU, and average pooling rather than today's more common max pooling. Those choices were reasonable in 1998 and the network worked beautifully on digits—but as we are about to see, they become a bottleneck the moment you try to scale up to real photographs.

The ImageNet moment: AlexNet

Fast-forward fourteen years to 2012, the moment the modern deep-learning era is usually said to begin. The benchmark everyone cared about was ImageNet, an annual competition to classify natural photographs into a thousand categories. Year after year, error rates had inched down slowly with hand-engineered features. Then a CNN called AlexNet entered—and roughly halved the previous best error overnight, dropping the top-5 error from about 26% to about 16%. That is not a polite improvement; it is the kind of jump that makes an entire field stop and stare. Almost immediately, computer vision pivoted from hand-designed features to learned deep networks.

What makes the achievement vivid is the sheer scale. AlexNet was trained on about 1.2 million labelled training images spanning 1000 classes—not clean 32×32 digits but messy, full-colour photographs of dogs, mushrooms, ships and keyboards, each downsized to 224×224 pixels. Training a network that big was only feasible because the authors ran it on GPUs (graphics processors built for massively parallel arithmetic), splitting the model across two of them for several days. Scale of data, scale of model, and scale of compute all arrived at once—and that combination, not any single trick, is what the moment really proved.

Here is the punchline that ties this guide together: at a high level, AlexNet is the same template as LeNet, just much bigger. It stacks five convolutional layers (some followed by pooling) to extract features, then three fully-connected layers to classify—conv→pool feature extractor, then a dense classifier head, exactly as before. The differences are matters of degree and ingredients: far more filters and channels, real 224×224 colour input instead of tiny digits, and a set of training tricks that LeNet lacked. Same recipe, scaled up and modernised. That is why understanding LeNet first pays off so fast—AlexNet is its grown-up descendant, not a different species.

What actually made AlexNet work

Bigger alone is not enough—a huge network can be impossible to train and quick to memorise its training set. AlexNet worked because it paired its scale with three training innovations. Let us take them one at a time, building the intuition for each, because every one of them is still standard practice today.

Innovation 1: ReLU instead of sigmoid/tanh. LeNet's smooth S-shaped activations have a hidden flaw: for large positive or large negative inputs they saturate—their output flattens out, so their slope (gradient) becomes almost zero. During training, learning signals flow backward by multiplying gradients layer by layer, so a near-zero slope chokes that signal and learning crawls; in a deep net it can stop entirely (the 'vanishing gradient' problem). ReLU sidesteps this with almost embarrassing simplicity:

The Rectified Linear Unit (ReLU): keep positive inputs, zero out negative ones.

Symbol by symbol: x is the pre-activation value—the raw number coming into the neuron before any nonlinearity. max(0, x) simply returns whichever is larger, 0 or x. So if x is positive, the function passes it through unchanged; if x is negative, it clamps the output to exactly 0. Picture a one-way valve or a gate that only opens for positive flow. Now the crucial part for training: for any x > 0 the function is just the line y = x, whose slope is a constant 1—a full-strength, undiminished gradient that the learning signal can ride straight through. Compare that to a saturated sigmoid, whose slope out in its flat region is close to 0, throttling the signal to a trickle. A constant gradient of 1 versus near-0 is exactly why ReLU networks trained several times faster, and it is a big part of why AlexNet was trainable at all.

Innovation 2: dropout in the fully-connected layers. A network with tens of millions of parameters can simply memorise its training data—doing great on examples it has seen and failing on new ones, the failure we call overfitting. Dropout fights this by, during each training step, randomly switching off a fraction of the neurons (AlexNet used 50% in its dense layers). The analogy: imagine a sports team where you randomly bench half the players every practice. No single star can become indispensable, so the team learns redundant, robust skills—every player must be able to step up. Likewise, no neuron can rely on one specific partner always being present, so the network learns more general, resilient features instead of brittle memorised ones.

Innovation 3: data augmentation. With 1.2 million images the model was still hungry for more variety, so the authors multiplied their data for free by transforming each image in label-preserving ways: taking random crops, horizontal flips (a mirrored cat is still a cat), and small colour jitter (nudging brightness and hue). Each epoch the network sees slightly different versions of every photo, which teaches it that a cat is a cat regardless of position, mirroring or lighting—again fighting overfitting, this time by enriching the inputs rather than thinning the network. One last historical footnote: AlexNet also used local response normalization (LRN), a scheme that made strongly-firing neurons suppress their neighbours. It gave a small boost at the time but was later found to help little, so modern networks dropped it—don't be confused when you never see it again.

Counting the cost: where the parameters hide

We keep saying these networks are 'big,' so let us learn to measure that precisely by counting parameters—the learnable weights and biases the network adjusts during training. Counting them is a wonderfully concrete skill, and it sets up the efficiency theme that drives the rest of this track. There are just two formulas to know, one for a convolutional layer and one for a fully-connected layer:

Parameter counts: convolutional layer (left) and fully-connected layer (right).

Let us name every symbol. For the conv layer: K is the kernel size (a filter is K×K pixels), C_in is the number of input channels (the depth of the incoming map), and C_out is the number of filters, which equals the number of output channels. The product K·K·C_in is the number of weights inside one single filter (it spans the full input depth); the +1 is the one bias each filter gets; and we multiply by C_out because we have that many filters. For the fully-connected layer: N_in is the number of inputs (neurons feeding in), N_out is the number of outputs (neurons in this layer); every input connects to every output, giving N_in·N_out weights, plus N_out biases—which is exactly (N_in + 1)·N_out. The deep reason FC layers are parameter-hungry: a conv filter reuses its small set of weights at every position across the image, whereas an FC layer gives every input–output pair its own private weight, with no sharing at all.

def conv_params(k, c_in, c_out):
    return (k * k * c_in + 1) * c_out   # +1 is one bias per filter

def fc_params(n_in, n_out):
    return (n_in + 1) * n_out           # +1 is one bias per output neuron

# AlexNet's FIRST conv layer: 11x11 kernel, 3 input channels (RGB), 96 filters
print(conv_params(11, 3, 96))      # 34,944        (~35 thousand)

# AlexNet's FIRST fully-connected layer (FC6):
# input  = 256 channels x 6 x 6 = 9216 numbers
# output = 4096 neurons
print(fc_params(9216, 4096))       # 37,752,832    (~37.7 million)

Sit with those two numbers, because they are the surprise of this guide. The first convolutional layer—the one doing the heavy visual lifting on raw pixels—holds only about 35 thousand parameters. The first fully-connected layer holds about 37.7 million, roughly a thousand times more, from a single layer. AlexNet has about 60 million parameters in total, and its three dense layers account for nearly all of them (~95%+); the five conv layers together are a small minority. The same lopsided pattern holds for VGG, as we will see next guide. So the takeaway is a memorable slogan: the depth is where the visual work happens, but the parameters are where the fully-connected layers are.

Takeaways and the road ahead

Let us lock in the mental model. Every CNN we have met—and every one we will meet—is the same template: a feature extractor built from stacked convolution and pooling layers, followed by a classifier head that turns features into class scores. LeNet wrote that template down for digits; AlexNet scaled it up and proved it works on a million real photographs. Nothing was reinvented; the bricks stayed the same. What changed was the recipe.

And progress in this field, you now know, comes from pulling just three levers: depth (how many layers stack up, letting the network compose simple patterns into complex ones), width (how many filters/channels per layer, i.e. how much it can see at once), and training tricks (ReLU, dropout, augmentation, and more, which make a big network actually trainable and stop it from merely memorising). Hold those three levers in mind and the rest of the track becomes a story of who pulled which lever, how far, and what new idea they needed to pull it safely.

Here is what is coming, concretely. Guide 2 takes the depth lever further while attacking that parameter bloat: VGG shows that stacks of small 3×3 convolutions beat big ones, and Inception (GoogLeNet) introduces the 1×1 convolution and parallel multi-scale branches to go deeper for less cost. Guide 3 is the turning point: ResNet adds 'skip connections' that finally let us train networks 100+ layers deep without the signal dying. Guide 4 chases efficiency—models slim enough to run on your phone. And Guide 5 closes the loop by letting algorithms design the networks themselves (NAS, RegNet, ConvNeXt). Same bricks the whole way; you are now ready to watch the buildings get taller, leaner, and smarter.