efficientnet
EfficientNet (Tan and Le, Google, 2019) addresses a question that earlier work treated ad hoc: when you have more compute and want a bigger, more accurate model, should you make the network deeper, wider, or feed it higher-resolution images? The EfficientNet insight is that these three dimensions are not independent and should be scaled together in a balanced way. If you increase input resolution, you also want more layers to grow the receptive field and more channels to capture the finer patterns, otherwise the extra resolution is wasted. The paper formalises this as compound scaling: pick a single coefficient phi, and scale depth, width, and resolution by fixed powers of a constant raised to phi, so all three grow in a fixed, principled ratio.
The method has two stages. First, a small baseline network called EfficientNet-B0 is found by neural architecture search optimising for both accuracy and efficiency; it is built mainly from MobileNetV2-style inverted-residual blocks (MBConv) augmented with squeeze-and-excitation attention. Second, that fixed baseline is scaled up by the compound rule to produce a family B1 through B7, each larger and more accurate than the last. The constants controlling the depth/width/resolution split are found once by a small grid search on B0 and then reused. This gives a clean, reproducible recipe for climbing the accuracy-efficiency frontier rather than hand-tuning each model size.
The result was a substantial advance on the accuracy-versus-FLOPs and accuracy-versus-parameters Pareto frontiers: EfficientNet models matched the accuracy of previous state-of-the-art networks while using several times fewer parameters and operations. A later revision, EfficientNetV2 (2021), improved training speed and introduced fused MBConv blocks and a progressive training schedule that grows image size during training. EfficientNet's lasting contribution is the principle of balanced compound scaling, which clarified how to grow models sensibly and influenced how subsequent architectures, including some vision transformers, are scaled.
Compound scaling intuition: doubling the FLOPs budget is split as depth times alpha, width times beta, resolution times gamma, with alpha times beta-squared times gamma-squared kept near 2 (width and resolution enter quadratically because they scale both spatial dimensions or both height and width of every feature map), so each new budget level inflates all three together rather than pouring everything into one.