on-device LLM

Most LLM features call a server in a data centre, but a model small enough can run right on the phone, laptop, or car in your hand. On-device inference means the model's weights live on the gadget and generate text locally, with no round trip to the cloud. The data never leaves the device, the feature works offline, and there is no per-call bill.

Fitting a capable model into a few gigabytes of phone memory relies on small architectures plus aggressive quantization — storing weights in four bits or fewer instead of sixteen — and runtimes tuned to the device's CPU, GPU, or dedicated neural accelerator. Frameworks load the compressed model, manage the limited memory, and stream tokens within tight power and thermal budgets. Hybrid designs are common: the device handles routine requests and quietly hands the hardest ones to a cloud model.

Running locally buys privacy, offline use, low latency, and zero marginal cost — compelling for personal assistants, keyboards, and anything handling sensitive data. The price is capability: a model that fits on a phone is far smaller than a frontier model, so it reasons less well and knows less, and it competes with everything else for the battery and memory. The sweet spot is well-scoped, latency- or privacy-sensitive tasks rather than open-ended general intelligence.