Efficient ConvNets: Vision Models That Fit in Your Pocket

The efficiency problem: FLOPs, latency, and phones

Until now in this track, accuracy has been king. From LeNet to AlexNet, then VGG and Inception, then 100-layer ResNets, every chapter asked the same question: how do we make the network recognise images better? But there is a second question that the real world cares about just as much: can you actually run the thing? A network that needs a rack of GPUs is useless for unlocking your phone with your face, or for the live camera filter that has to redraw 30 times a second. In this guide the goal changes from 'most accurate' to 'most accuracy you can afford on a tiny chip.'

To talk about 'affordable,' we need three plain metrics. Parameters are the number of learned weights — they decide how much memory the model takes up (a million parameters at 4 bytes each is about 4 MB). FLOPs (floating-point operations), usually counted as multiply-adds, measure how much raw arithmetic one forward pass costs — this is the compute bill. Latency is the one users feel: the actual wall-clock time, in milliseconds, from feeding in an image to getting an answer on a specific device.

With that framing, the rest of this guide is a tour of the tricks that let modern networks live in your pocket. We will start with the single most important one — the depthwise separable convolution, the engine inside MobileNet — and then add channel shuffles, channel attention, and a clever way to scale a whole network up or down. None of these throw away the convolutional ideas you already know; they re-arrange them to be cheap.

Depthwise separable convolutions: split the work

Think about what an ordinary convolution layer actually does. Each output channel is produced by a small filter that slides across the image and, at every position, looks at all the input channels at once and across a little spatial window. So a single standard convolution is doing two jobs glued together: mixing information across space (the neighbouring pixels) and mixing information across channels (the different feature maps). Doing both at once, for every output channel, is where the cost piles up.

The depthwise separable convolution simply unglues those two jobs into two cheap steps done in sequence. Step one, the depthwise convolution: give each input channel its own little spatial filter and let it filter only itself — no channel mixing at all. Step two, the pointwise convolution: a 1×1 convolution (exactly the trick from the VGG/Inception guide) that looks at a single pixel position across all channels and mixes them into the new output channels — no spatial mixing at all. The analogy: instead of one worker doing a giant combined job, you hire one specialist to handle 'space' and another to handle 'channels.' Two simple jobs in a row turn out to be far cheaper than one fused job. This factorisation is the engine inside MobileNet and Xception, and later inside EfficientNet.

Cost of a standard conv vs. its depthwise-separable replacement, and the resulting ratio.

Let us read every symbol. D_K is the kernel size (e.g. 3 for a 3×3 filter), M is the number of input channels, N is the number of output channels, and D_F is the width/height of the feature map (so D_F^2 counts the spatial positions the filter must visit). The standard cost D_K^2\cdot M\cdot N\cdot D_F^2 says: for each of D_F^2 positions, and each of N outputs, you do a D_K^2\times M multiply-add — everything multiplies together. The separable cost splits into the depthwise term D_K^2\cdot M\cdot D_F^2 (each of M channels filtered by its own D_K^2 kernel, no N factor!) plus the pointwise term M\cdot N\cdot D_F^2 (a 1×1 mix, so no D_K^2 factor). Dividing the separable cost by the standard cost, the D_F^2 cancels and the algebra collapses to \tfrac{1}{N}+\tfrac{1}{D_K^2}. Plug in a typical layer with D_K=3 and many output channels (say N=256): the ratio is \tfrac{1}{256}+\tfrac{1}{9}\approx 0.004+0.111\approx 0.115 — roughly 1/9th the compute, about an 8–9× saving. The intuition for why splitting wins: the expensive standard conv pays the D_K^2 spatial factor and the N channel factor multiplied together; by separating space from channels you pay them added together instead, and a sum of two small numbers is far less than their product.

import torch.nn as nn

# Standard conv: one big op that mixes space AND channels at once
std = nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1)

# Depthwise separable = two cheap ops done in sequence:
# 1) depthwise: each channel filtered on its own (groups == in_ch)
depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=3, padding=1, groups=in_ch)
# 2) pointwise: a 1x1 conv that mixes channels (no spatial window)
pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1)

separable = nn.Sequential(depthwise, pointwise)  # same in/out shape, ~9x cheaper

MobileNet and Xception: two takes on the same idea

The beautiful thing about the depthwise separable convolution is that the same building block serves two completely different ambitions, depending on who picks it up. MobileNet grabs it to go small and fast: it stacks depthwise-separable blocks to build a network that is a fraction of the size of a comparable standard CNN, light enough to run on a phone's CPU in real time. Same accuracy ballpark, a fraction of the cost.

On top of that base, MobileNet adds two honest 'dials' you can turn to trade accuracy for speed without redesigning anything. The width multiplier thins every layer by a factor (say 0.5×), so every layer has half as many channels — fewer features, less compute, slightly less accuracy. The resolution multiplier feeds the network smaller input images (say 160×160 instead of 224×224), which shrinks every feature map and therefore the work at every layer. Neither dial is magic; they are deliberate, predictable knobs that let one architecture span phones from flagship to budget.

Xception arrives at the very same operation from the opposite direction, and chasing the opposite goal: maximum accuracy, not minimum size. Recall the Inception module from guide 2, which split a layer's channels into a few parallel groups and processed each group with its own filters. Xception asks: what if we push that splitting to its extreme limit — one group per channel? A separate spatial filter for every single channel, followed by a 1×1 to recombine, is exactly a depthwise separable convolution. So Xception ('Extreme Inception') is the limit case of Inception, and used as a high-accuracy backbone it actually beats the Inception it grew out of.

ShuffleNet: grouped convs plus a channel shuffle

For the truly tiny budget — think a cheap camera chip or a smartwatch — even MobileNet's 1×1 pointwise convolutions become the dominant cost (remember, after we make the spatial part cheap, the channel-mixing 1×1 is what is left). ShuffleNet attacks that last cost with grouped 1×1 convolutions, borrowing the 'cardinality' idea from ResNeXt in guide 3: instead of one 1×1 conv that mixes all channels, split the channels into, say, 4 groups and let each group mix only within itself. That immediately cuts the 1×1 cost by roughly the number of groups.

But grouping creates a real problem. If every layer keeps its channels in fixed groups and groups never talk to each other, information gets siloed: the features computed in group 1 only ever feed group 1 in the next layer, group 2 only feeds group 2, and so on. The network effectively splits into parallel narrow networks that never share what they learn — and that hurts accuracy badly.

The fix is the wonderfully simple channel shuffle. After a grouped convolution, you rearrange the channels so that the next layer's groups each receive a mixture drawn from all of the previous groups. Picture a party where everyone first chats only at their own table (the groups), then between rounds everyone reshuffles seats so each new table holds one person from every old table — now ideas spread across the whole room. Or picture a deck of cards dealt into four piles, then re-dealt so each new pile contains cards from all four originals. After a shuffle, group boundaries no longer trap information, yet you still pay only the cheap grouped-conv cost.

Squeeze-and-Excitation: cheap channel attention

So far every channel a convolution produces is treated as equally important. But for a given image, some feature channels are far more useful than others — a 'fur texture' channel matters for a cat, a 'wheel' channel does not. The Squeeze-and-Excitation block (SE) is a tiny, almost-free add-on that lets the network learn to reweight its channels by importance, per input. It is an early, lightweight form of attention. The analogy: a sound mixer with one volume slider per channel — SE turns up the channels carrying useful signal and turns down the ones carrying noise, and it learns the slider settings from data.

1. SQUEEZE: collapse each channel's whole H×W feature map into a single number with global average pooling (the same pooling idea from guide 2). This gives one compact descriptor value per channel — a summary of 'how much is this channel firing overall.'
2. EXCITE: feed those per-channel numbers through a tiny two-layer bottleneck network (a small fully-connected layer that reduces dimension, a ReLU, then another that restores it) ending in a sigmoid, producing one weight between 0 and 1 for every channel.
3. SCALE: multiply each original channel's feature map by its learned 0–1 weight. Important channels keep most of their strength; unimportant ones get dialled toward zero.

Excite then scale: a two-layer gate turns the squeezed descriptor into per-channel weights.

Symbol by symbol: \mathbf{z} is the squeezed descriptor vector — one number per channel produced by the global average pooling in step 1, so its length equals the channel count C. W_1 is the first small fully-connected layer; it reduces the dimension (typically by a factor like 16, so C numbers become C/16) to keep the block cheap. \delta is the ReLU non-linearity applied in between. W_2 is the second fully-connected layer; it restores the dimension back up to C. \sigma is the sigmoid, which squashes each output into the range 0 to 1, giving the importance weights \mathbf{s}. Finally s_c is channel c's learned weight and x_c is channel c's original feature map, so \tilde{x}_c = s_c\cdot x_c is that channel scaled by its weight. Tiny worked example: suppose after the gate channel 1 gets s_1=0.95 (kept almost fully) and channel 2 gets s_2=0.05 (almost silenced); the layer has effectively decided, for this particular image, that channel 1 is informative and channel 2 is not. The whole gate costs only \sim C^2/16 extra parameters — a rounding error next to the convolutions — which is why SE can be dropped into ResNet, MobileNet, and almost any backbone for a free accuracy bump.

import torch
import torch.nn as nn

class SEBlock(nn.Module):
    def __init__(self, channels, r=16):
        super().__init__()
        self.pool = nn.AdaptiveAvgPool2d(1)              # SQUEEZE: H,W -> 1,1
        self.fc1  = nn.Linear(channels, channels // r)    # W1: reduce dim
        self.fc2  = nn.Linear(channels // r, channels)    # W2: restore dim

    def forward(self, x):                  # x: (N, C, H, W)
        n, c, _, _ = x.shape
        z = self.pool(x).view(n, c)        # z: per-channel descriptor (N, C)
        s = torch.relu(self.fc1(z))        # delta(W1 z)
        s = torch.sigmoid(self.fc2(s))     # sigma(W2 ...) -> weights in [0,1]
        return x * s.view(n, c, 1, 1)      # SCALE: reweight each channel

EfficientNet: scale depth, width, and resolution together

We now have cheap building blocks. The last question is: once you have a good small network and a bit more compute budget, how should you grow it? There are exactly three axes. Depth = more layers (stack more blocks). Width = more channels per layer (the same axis MobileNet's width multiplier thinned). Resolution = bigger input images (more pixels, larger feature maps). Earlier networks usually cranked just one — VGG went deep, others went wide — and each axis on its own hits diminishing returns surprisingly fast.

EfficientNet's insight is to scale all three together, in a fixed, balanced ratio, controlled by a single user knob. The analogy: if you want a bigger, stronger body you don't just grow longer legs — you grow legs, torso, and arms in proportion, or you end up unbalanced and weak. EfficientNet found, by a small search, the right proportions in which to grow depth, width, and resolution, then exposes one number that grows all three in lock-step.

Compound scaling: one knob φ grows depth, width, and resolution at fixed rates α, β, γ.

Here is the rule unpacked. \phi (phi) is the single 'compute budget' knob the user turns: \phi=0 is the small baseline network, and each step up spends more compute. \alpha, \beta, \gamma are the fixed per-axis growth rates found once by a small grid search — \alpha for depth, \beta for width, \gamma for resolution, each at least 1 so the network only ever grows. Raising each to the power \phi means turning the knob multiplies all three axes simultaneously: depth by \alpha^\phi, width by \beta^\phi, input resolution by \gamma^\phi. The constraint \alpha\cdot\beta^2\cdot\gamma^2\approx 2 is the clever bit: it pins things so that each unit of \phi roughly doubles the total FLOPs. Why are \beta and \gamma squared but \alpha is not? Because adding channels (width) costs compute in two dimensions — more input channels times more output channels — and enlarging the image (resolution) costs in two spatial dimensions, height and width; depth adds layers in just one dimension, so it appears to the first power. Concrete example: the published EfficientNet found roughly \alpha\approx1.2,\ \beta\approx1.1,\ \gamma\approx1.15. Check the budget: 1.2\times1.1^2\times1.15^2 \approx 1.2\times1.21\times1.32 \approx 1.92\approx 2. So at \phi=1 depth grows about 1.2×, width about 1.1×, resolution about 1.15×, and compute roughly doubles — a balanced step, not a lopsided one. The intuition in one line: when you can afford more compute, spend it evenly across all three axes instead of pouring it all into one.

The satisfying part is what the network is made of. EfficientNet's repeated building block, called MBConv, is itself assembled from the very pieces in this guide: an inverted-bottleneck depthwise separable convolution (Section 2) — the same engine as MobileNet — with a Squeeze-and-Excitation block (Section 5) bolted inside to reweight its channels. So compound scaling grows a network whose every block is already an efficiency machine. Every trick we met assembles into one family.