regnet

RegNet (Radosavovic et al., Facebook AI Research, 2020) comes from a paper titled 'Designing Network Design Spaces' and represents a shift in how architectures are discovered. Rather than searching for a single best network (as neural architecture search does) or hand-crafting one model, the authors search for a good design space: a parameterised family of architectures whose every member tends to be good. The methodology is to start from a very broad, loosely constrained space (called AnyNet), sample many random models, train them, and then statistically analyse which design choices correlate with good models, progressively tightening the space by adding constraints that improve the whole population's quality.

That analysis yields a strikingly simple result. The good networks obey a near-linear rule for how layer width should grow with depth: the per-block widths follow a quantized linear function (widths increase roughly linearly across the network and are then rounded to convenient values), and a handful of parameters, a starting width, a slope, a depth, a group width for grouped convolution, and a bottleneck ratio, suffice to specify an excellent network. The resulting family is named RegNet because the networks are regular: they have no irregular, hand-tuned, per-stage idiosyncrasies, just a clean rule. RegNets come in compute-targeted sizes (RegNetX without attention, RegNetY adding squeeze-and-excitation).

RegNet's value is twofold. Practically, RegNet models match or beat EfficientNet across a range of compute budgets while being simpler and often faster to train and run on GPUs, because their regular structure parallelises well. Conceptually, the paper argues for a more scientific, population-level approach to architecture design: instead of treating networks as a search for one lucky configuration, study the distribution of good models to extract human-interpretable design principles. This 'design spaces' philosophy was influential, and RegNet backbones were adopted in large-scale vision systems, including as the vision encoders behind some self-supervised and multimodal models.