Designing for the device: NAS and the deployment pipeline

From compressing to searching

Every earlier guide took an architecture as given and made it cheaper. But the most efficient model for a phone may have a fundamentally different shape than the one you would design for a GPU — different depth, width, kernel sizes, and operator choices. Neural architecture search (NAS) automates the discovery of that shape: instead of hand-tuning a network, you define a search space of possible architectures and let an algorithm find the best point in it under your constraints.

Early NAS was infamous for spending thousands of GPU-days training each candidate from scratch. Modern NAS is far cheaper, largely thanks to weight sharing: train one big 'supernet' once so that every candidate architecture is a sub-network of it, then evaluate candidates by inheriting the supernet's weights instead of retraining. The search becomes a search over sub-networks, not a thousand training runs.

Hardware-aware NAS: put latency in the loss

Searching for accuracy alone reinvents the cloud model. The key move in efficient AI is hardware-aware NAS: fold the real cost on the real target device directly into the objective, so the search optimizes accuracy and on-device latency or energy together. Crucially, the cost term is measured (or predicted by a latency model) on the actual chip — because FLOPs are a poor proxy. Two layers with equal FLOPs can differ 5× in latency depending on memory access patterns and which operators the accelerator has hardware for.

\mathcal{L}(\alpha, w) = \mathrm{CE}(w, \alpha) + \lambda\,\mathbb{E}_{\alpha}\!\left[\mathrm{LAT}(\alpha)\right]

Hardware-aware NAS folds measured on-device latency straight into the training objective, with λ trading accuracy against real speed.

# hardware-aware search objective
reward = accuracy(arch) - lambda * latency_on_device(arch)
#                                    ^ measured on the target chip,
#                                      not estimated from FLOPs

A hardware-aware objective trades accuracy against measured on-device latency.

Putting it together: a real edge pipeline

On-device inference is where all these tools meet. A realistic path from a trained cloud model to a shipping edge deployment usually stacks several of them in sequence, re-measuring on the target after each stage:

Pick or search for a device-friendly architecture (hardware-aware NAS, or a known-good mobile backbone) so the shape fits the silicon from the start.
Shrink it structurally — structured prune redundant channels and heads, optionally distill from the larger teacher to recover accuracy.
Quantize last — post-training quantization to INT8 or INT4, since it is the cheapest, most forgiving step and composes on top of everything else.
Export to a portable runtime — convert to a format like ONNX (or a vendor mobile runtime) so the same graph runs across the device's CPU, GPU, or NPU.
Benchmark on the real device for latency, peak memory, and energy; iterate until the budget and the accuracy floor are both met.

Where the frontier is going

Two ideas pull this field forward. First, the boundary between cloud and edge is blurring: serving systems increasingly split a request, running a cheap draft on-device and escalating only hard parts to a server — efficiency techniques becoming a routing decision across a fleet. Second, the lowest-precision frontier (ternary weights, native low-bit training) suggests that the most efficient models will not be compressed after training but architected, from the first step, to be cheap.

p_{\text{accept}}(x) = \min\!\left(1, \frac{p(x)}{q(x)}\right)

Speculative decoding accepts a cheap on-device draft token with probability min(1, p/q), escalating to the larger model only when the target disagrees.