on-device inference

On-device inference runs a model directly on the phone, watch, car, or microcontroller in front of the user, instead of sending the data to a server. The payoff is privacy, since the data never leaves the device, plus offline availability and low, predictable latency. The price is a hard ceiling on memory, compute, energy, and heat.

Fitting inside that budget combines the techniques in this field — low-bit quantization, pruning, distillation, efficient architectures — with hardware-specific runtimes such as Core ML, LiteRT, ONNX Runtime, llama.cpp, and ExecuTorch that target mobile NPUs, DSPs, and GPUs. For transformer decoding the binding constraint is usually memory bandwidth rather than raw FLOPs, so weight quantization and KV-cache management dominate real performance, and thermal throttling caps sustained throughput.

The field is steadily moving billion-parameter language models onto consumer hardware, but the engineering is unforgiving: a model that looks fine in a one-shot benchmark can still stutter once the device warms up, the battery saver kicks in, or the operating system reclaims its memory.