mobile inference

Mobile inference is the special, very common case of edge deployment: running vision models on smartphones and similar embedded systems-on-chip. Phones are surprisingly capable — they ship dedicated neural accelerators such as Apple's Neural Engine, Qualcomm's Hexagon NPU, and Google's Tensor — but to hit interactive speeds within a battery and thermal budget you must pair lightweight models with the right runtime and let the accelerator do the work.

A mobile SoC is heterogeneous: it has a CPU, a GPU, a DSP, and an NPU, and the runtime decides where each operator runs — a process called delegation. The common stacks are TFLite (with NNAPI, GPU, Core ML, or Hexagon delegates), Core ML on iOS, PyTorch's ExecuTorch, and MNN / NCNN / TNN. Models are usually int8-quantized because NPUs are integer-first, and they use mobile-friendly architectures — MobileNet's depthwise-separable convolutions, ShuffleNet, EfficientNet-Lite, MobileViT — and distilled, quantized detectors and segmenters.

The pitfalls are specific. NPU operator support is incomplete, so an unsupported op falls back to the CPU and can tank end-to-end latency; quantization is often mandatory because some NPUs cannot run float at all; and thermal throttling means a short benchmark burst is far faster than sustained video processing. Always profile on real devices across multiple vendors and SoC generations, not just on a single flagship phone.