Smarter, Lighter Convolutions

The cost problem of plain convolutions

In guide 2 we counted the parameters in a convolutional layer: a stack of filters needs K × K × C_in × C_out learnable numbers, where K is the side of the convolution kernel, C_in is how many input feature channels there are, and C_out is how many output channels you want. Parameters tell you how much memory the weights take. But they do not tell you how much work it takes to run the layer on an image. For that we need to count the actual arithmetic — and that count is what decides whether your model is fast enough to run on a phone.

The key insight is that those K × K × C_in × C_out multiply-adds are not done once — they are repeated at every output position. A filter slides across the whole feature map, and at each landing spot it redoes the full dot product. So the true compute cost multiplies the parameter count by the number of output positions H_out × W_out.

The compute cost of one standard convolutional layer.

Read this factor by factor. H_out · W_out is the number of pixels in the output feature map — every one of them is a separate place where the filter does its work. K · K is the multiply-adds inside one kernel window for a single channel. C_in appears because each output value sums contributions from all input channels, so you repeat the K · K window once per input channel. And C_out appears because you are not learning one filter but C_out of them, each producing its own output channel. Multiply all four groups and you get the total number of multiply-add operations the layer performs on one image. Every trick in this guide is an attack on one or more of these factors.

Let's put numbers on it. Suppose a mid-network layer keeps a 56 × 56 feature map, uses 3 × 3 kernels, and maps 256 input channels to 256 output channels. That is 56 · 56 · 3 · 3 · 256 · 256 ≈ 1.85 billion multiply-adds — for a single layer, on a single image. A real network stacks dozens of such layers, and a video app must do this 30 times a second. On a phone with a tiny power budget, plain convolutions become the bottleneck fast.

So here is the goal of this guide, stated plainly: keep most of the modelling power while slashing the number of multiply-adds. We do it with three tools, each introduced next — the 1 × 1 convolution (mix channels cheaply), the dilated convolution (see wider for free), and the depthwise-separable convolution (factor an expensive conv into two cheap ones). None of them is a hack; each is a clean idea about which factor above you actually need.

1×1 convolution: mixing channels

The first tool sounds almost too small to matter: a 1x1 convolution, also called a pointwise convolution. Its kernel is a single pixel wide and tall — K = 1. At first that seems pointless, because there is no neighbourhood to slide a window over. But look at what is left once you remove the spatial part: the depth.

With a 1 × 1 kernel there is no spatial mixing at all — each output pixel depends only on the stack of channel values sitting at that one input pixel, never its neighbours. But it fully mixes channels: each output channel is a weighted sum of all C_in input channels at that location. In other words a 1 × 1 conv is a small linear layer that runs down the depth axis, applied independently at every pixel. It reshapes the channel dimension while leaving the spatial grid untouched.

Why is this so useful? The headline use is cheaply changing the channel count. Imagine a feature map with 256 channels that you would like to filter spatially with an expensive 3 × 3 conv. Running the 3 × 3 conv directly on 256 channels is costly (recall C_in and C_out both sit in the cost formula). Instead, first squeeze 256 → 64 channels with a 1 × 1 conv, do the expensive spatial work on the slimmer 64-channel tensor, then expand 64 → 256 again with another 1 × 1 conv. This squeeze-and-expand pattern is called a bottleneck, and it is everywhere in modern networks.

Work the squeeze concretely. A 56 × 56 × 256 feature map goes through a 1 × 1 conv producing 64 output channels: 56 × 56 × 64. The cost is H · W · K · K · C_in · C_out = 56 · 56 · 1 · 1 · 256 · 64 ≈ 51 million multiply-adds. Compare that to the ~1.85 billion of the 3 × 3, 256→256 layer from section 1: the 1 × 1 conv is over 30× cheaper, and it has handed the next layer a tensor with 4× fewer channels to chew through. The 1 × 1 also adds genuine modelling power — followed by a nonlinearity, it lets the network learn useful cross-channel interactions (which channel should reinforce or cancel which).

Here is the deeper way to see it: a 1 × 1 conv is a tiny multilayer-perceptron-style linear map across depth, and the same weight matrix is reused at every single pixel. That is weight sharing again — the very same principle that made the ordinary convolutional layer efficient in guide 2, now applied purely along the channel axis. One small set of weights, replayed across the whole spatial grid.

# A 1x1 conv is literally a per-pixel matrix multiply across channels.
# x: input feature map, shape (H, W, C_in)
# W: weight matrix, shape (C_in, C_out)   <- ONE matrix, shared over all pixels
# y: output feature map, shape (H, W, C_out)

for i in range(H):
    for j in range(W):
        # take the channel vector at pixel (i, j): length C_in
        # multiply by the shared weight matrix -> length C_out
        y[i, j] = x[i, j] @ W   # no neighbours touched: pure channel mixing

Dilated convolution: wider view, same cost

In guide 3 we met the receptive field: how big a patch of the original image a single later-layer neuron can actually see. We grew it by stacking layers and by pooling/striding. But pooling and stride pay for reach by throwing away resolution — the feature map shrinks. For tasks that need a label at every pixel, like segmentation, that loss of detail hurts. The dilated convolution (also called atrous, French for 'with holes') gives you reach without that price.

The idea is delightfully simple: keep the same K × K convolution kernel, but spread its taps apart, inserting gaps of size controlled by the dilation rate d. At d = 1 it is an ordinary conv with the taps touching. At d = 2 you skip one input pixel between each tap, so the kernel reaches across a wider area while still using only its K × K weights. Nothing is added to the parameter count, and the feature map does not shrink — the stride is still 1.

The effective footprint of a dilated kernel.

Unpack each symbol. K is the real number of taps along one side — the genuine weights you learn and pay for. d is the dilation rate, the gap inserted between neighbouring taps. K_eff is the side length of the region the kernel now spans. Why this formula? A K-tap row has (K − 1) gaps between its taps. Dilation widens each of those gaps by (d − 1) extra empty slots. So the footprint grows from K to K + (K − 1)(d − 1), even though only K weights are active — the rest of the span is holes that contribute nothing.

Check it on the example. Take K = 3 and d = 2: K_eff = 3 + (3 − 1)(2 − 1) = 3 + 2 = 5. So a 3 × 3 kernel at dilation 2 spans a 5 × 5 region of the input — its receptive field has jumped from 3×3 to 5×5 — yet it still holds exactly 3 × 3 = 9 weights and costs the same 9 multiply-adds per channel per position as before. You bought a 5 × 5 view at a 3 × 3 price.

The intuitive picture: spreading your fingers lets one hand feel a wider area with the same number of fingertips. You are not adding fingers (weights); you are reaching them further apart. Stack a few dilated layers with growing d (say 1, 2, 4) and the receptive field expands exponentially while the feature map stays full resolution — which is exactly why dilation became the workhorse of dense-prediction networks for segmentation.

Depthwise separable convolution

Now the centrepiece. A standard convolution does two jobs at once: it mixes information spatially (across the K × K neighbourhood) and across channels (summing over C_in) in a single fused operation. The depthwise separable convolution asks: what if we split those two jobs into separate, much cheaper steps? It turns out you can — and the result approximates a full conv at a fraction of the cost. This factorization is the engine inside MobileNet and Xception.

1. Depthwise convolution: apply ONE K × K spatial kernel per input channel, independently. Channel 1's filter only ever touches channel 1, channel 2's only touches channel 2, and so on. This does the spatial filtering but performs NO cross-channel mixing — the channels stay separate. Output has the same C_in channels.
2. Pointwise convolution: apply a 1 × 1 conv to mix those C_in channels into C_out output channels. This is exactly the channel-mixing tool from section 2, and it does NO spatial work (K = 1).

Together, depthwise (handles space) + pointwise (handles channels) recreate what a standard conv did jointly. Now the payoff, in FLOPs per output position. A standard conv costs K · K · C_in · C_out multiply-adds per position. The depthwise step costs K · K · C_in (one K×K filter per channel, no C_out factor because it does not change the channel count). The pointwise step costs 1 · 1 · C_in · C_out = C_in · C_out. So the separable version costs K · K · C_in + C_in · C_out — a sum of two small terms instead of a product of all four.

Depthwise-separable cost divided by standard cost.

Let's walk every step of this division so the speedup is obvious. The numerator is the separable cost: depthwise K·K·C_in plus pointwise C_in·C_out. The denominator is the full conv cost K·K·C_in·C_out. Split the fraction into two pieces. The first piece, (K·K·C_in)/(K·K·C_in·C_out), cancels K·K·C_in top and bottom and leaves 1/C_out. The second piece, (C_in·C_out)/(K·K·C_in·C_out), cancels C_in·C_out and leaves 1/(K·K). Add them: ratio = 1/C_out + 1/(K·K). Two clean reciprocal terms.

Now read what the two terms mean. When C_out is large (256, 512, …), 1/C_out is tiny and basically vanishes — so the saving is dominated by the second term, 1/(K·K). Plug in the common K = 3: 1/(K·K) = 1/9. With, say, C_out = 256 you get ratio = 1/256 + 1/9 ≈ 0.0039 + 0.111 ≈ 0.115, which is about 1/8.7. So a depthwise-separable block does roughly the same job for 8–9× fewer multiply-adds at K = 3. (With K = 5 the dominant term is 1/25, so the win grows to ~20×.) That single factored idea is what lets accurate vision models run in real time on a phone.

# Standard conv vs depthwise-separable conv (PyTorch-style), same in/out shapes.
import torch.nn as nn

# (a) Standard 3x3 conv: spatial + channel mixing fused. Expensive.
std = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1)

# (b) Depthwise-separable: two cheap steps that together do the same job.
dw = nn.Conv2d(256, 256, kernel_size=3, padding=1,
               groups=256)          # groups=C_in => one K x K filter PER channel
pw = nn.Conv2d(256, 256, kernel_size=1)  # 1x1 => mix channels, no spatial work
sep = nn.Sequential(dw, pw)

# FLOPs per output position:
#   standard : 3*3*256*256          = 589,824
#   separable: 3*3*256 + 256*256    =   2,304 + 65,536 = 67,840
#   ratio    : 67,840 / 589,824     ~= 0.115  (about 8.7x cheaper)

Putting the pieces together: efficient architectures

The three tools are not rivals — they compose into a single repeatable building block. The pattern that powers efficient mobile backbones is the inverted-residual bottleneck: a 1 × 1 conv expands the channels, a depthwise conv does the spatial filtering on that wide tensor, and a final 1 × 1 conv projects back down. It is the bottleneck idea from section 2, but turned inside out — expand first, filter wide, then squeeze.

1. Expand: a 1 × 1 1x1 convolution raises the channel count (e.g. 24 → 144), giving the next step a richer space to work in. A nonlinearity follows.
2. Filter spatially: a depthwise conv (the spatial half of the depthwise separable convolution) processes each of the 144 channels independently — cheap because there is no cross-channel sum here. Optionally use a dilated convolution here to widen the receptive field for context-hungry tasks.
3. Project: a second 1 × 1 conv squeezes back down (144 → 24). When input and output shapes match, add a residual skip connection straight across the block (the reason it is called 'residual').

Designing with these blocks is really about balancing three things the deployment cares about: accuracy (how good the predictions are), latency (how fast one image runs), and memory (how much RAM the activations and weights need). Simple guidance: reach for a 1 × 1 conv whenever you want to change channel count or add cheap cross-channel reasoning; reach for dilation when you need a bigger receptive field but must keep full spatial resolution (segmentation, detection); reach for depthwise-separable whenever a plain 3 × 3 conv is the FLOP hotspot and you can trade a sliver of accuracy for a large speedup.

You now have a toolkit for building networks that are wide in reach yet light in compute. But there is a catch we have sidestepped: stacking dozens of these blocks into a very deep network does not train itself. Gradients can vanish, activations can drift, and naive deep stacks can actually get worse. Guide 5, the finale, tackles exactly that — how to actually train deep CNNs, with the normalization, residual connections, and initialization tricks that let these towering efficient stacks learn.