Neural Networks Taxonomy - 13. Neural Architecture Search

13. Neural Architecture Search (NAS)

Neural architecture search automates the design of the network itself. Rather than a human choosing how many layers of what kind to stack, a search process proposes architectures from a defined space, trains and evaluates them, and uses the result to propose better ones, as the loop in Fig 15 makes clear. Several of the strongest convolutional backbones were discovered this way.

Origins and rise

Zoph and Le framed the search as a reinforcement-learning problem in 2017, with a controller rewarded for proposing accurate networks.^[166] That was powerful but hugely expensive; DARTS made the search differentiable, relaxing the discrete choice of operations into a continuous one solvable by gradient descent.^[167] Discovered models such as MnasNet and EfficientNet followed.

Cell-based and weight-sharing search

The reinforcement-learning controller was made practical by two ideas. NASNet searched not for a whole network but for a small cell — a reusable block — on a small proxy dataset, then stacked copies of it to any depth, so a search run on CIFAR-10 transferred to ImageNet.^[168] Even so, every candidate still had to be trained from scratch. ENAS removed that cost by forcing all candidates to share one set of weights, treating each as a subgraph of a single over-parameterised network, and collapsed the search from thousands of GPU-days to less than one.^[169]

Differentiable search

DARTS recast the discrete choice of operation on each edge as a weighted mixture, making the whole architecture differentiable and solvable by gradient descent^[167] — but the relaxation is fragile. PC-DARTS sampled only a fraction of the channels on each edge, cutting the memory that had confined searches to tiny proxies,^[170] while P-DARTS grew the network's depth progressively so the searched and evaluated architectures finally matched.^[171] Two later fixes tackled a notorious failure mode in which the search collapses onto parameter-free skip connections: Fair DARTS removed the unfair advantage those connections enjoy,^[172] and DARTS‑ stepped out of the collapse without hand-tuned indicators.^[173]

Hardware-aware search

Efficiency on a real device depends on more than parameter count, so the reward itself was made hardware-aware. MnasNet folded measured on-phone latency directly into a multi-objective reward, yielding mobile models on the accuracy-latency frontier,^[174] and MixNet then let a single searched layer mix several depthwise kernel sizes at once.^[175] On the differentiable side, ProxylessNAS searched directly on the target task and hardware rather than a proxy, modelling latency as a differentiable loss,^[176] and FBNet steered a DARTS-style search toward a specific chip with a latency lookup table.^[177] The same principle carried to language, where HAT searched hardware-aware transformer configurations tuned to the latency of the deployment device.^[178]

Train-once supernets

Searching afresh for every deployment target is wasteful when many targets share one problem. Once-for-All trained a single elastic supernet from which specialised sub-networks of differing depth, width, and resolution can be extracted for any latency budget without retraining,^[179] and BigNAS pushed the idea further, training one big single-stage model whose sliced sub-networks are deployment-ready as sampled.^[180] Making weight sharing fair is delicate, however: SCARLET-NAS stabilised supernet training so that a candidate's shared-weight accuracy actually predicts its stand-alone accuracy.^[181]

Searching beyond convolutions

Search need not stop at convolutional cells. AutoFormer applied weight-sharing search to vision transformers, entangling the weights of candidate blocks so that thousands of sub-transformers could be ranked from one supernet.^[182] The field's impact is clearest, though, in the backbones now in everyday use: EfficientNet paired a searched mobile block with compound scaling,^[18] while RegNet searched over design spaces rather than individual networks, distilling the outcome into simple, quantised rules for width and depth.^[183]

Applications

• Designing efficient image backbones under size or latency budgets.
• Hardware-aware models tailored to a specific chip.
• Automating model design where expert tuning is scarce.

Strengths and limitations