neural architecture search

Neural architecture search (NAS) is the project of automating architecture design: instead of a human deciding how many layers to stack, which filter sizes to use, and how to wire the connections, an algorithm searches over a space of possible architectures and discovers good ones on its own. Any NAS method is defined by three components. The search space specifies which architectures are even allowed (for example, what operations may appear in a repeatable cell and how cells connect). The search strategy is how the space is explored (reinforcement learning, evolutionary algorithms, or gradient-based optimisation). The performance estimation strategy is how the quality of a candidate is judged without fully training every one, since that would be ruinously expensive.

The early landmark works used reinforcement learning: a controller network proposes an architecture, that architecture is trained and evaluated, and its validation accuracy is used as a reward to update the controller (Zoph and Le, 2017). Evolutionary approaches instead maintain a population of architectures and mutate and select the fittest (for example AmoebaNet). These methods produced genuinely strong networks such as NASNet but at staggering cost, often thousands of GPU-days for a single search, because each candidate had to be trained nearly from scratch. This cost was the central obstacle and the focus of most subsequent research.

Two ideas made NAS practical. Weight sharing, or the one-shot approach, trains a single large supernet that contains every candidate architecture as a sub-path, so individual candidates can be evaluated by inheriting the supernet's weights instead of training from scratch (ENAS, then DARTS). DARTS in particular made the search differentiable by relaxing the discrete choice of which operation to use into a continuous, learnable weighting, so the architecture can be optimised by ordinary gradient descent, cutting search cost to a few GPU-days. NAS has produced influential, deployed models, the EfficientNet baseline B0, MobileNetV3, and many specialised on-device networks were found or tuned with NAS, and hardware-aware NAS now optimises directly for measured latency on a target chip. Caveats remain: results can be sensitive to the search space (a well-designed space does much of the work), reproducibility and fair comparison have been contentious, and a strong manual baseline like RegNet or ConvNeXt is often competitive with searched models, so NAS is a powerful tool rather than an automatic replacement for design insight.