Shipping to the Edge and Mobile: The Master's Playbook

The Edge Constraint: No Cloud, Tight Budgets

Welcome to the capstone. Across this track you learned each lever on its own — what "fast enough" means, how to shrink a model, how to quantize it, and how runtimes map it onto silicon. Now we pull every lever at once to run a real vision model in real time on a phone. The first question is why bother: why not just send each frame to a powerful cloud GPU? Four reasons push us toward running the model on the device itself. Privacy: the camera frames never leave the user's phone, which matters enormously for faces, documents, and medical images. Latency: there is no network round-trip, so you avoid the 50–300 ms (and wildly variable) delay of a cloud call. Offline operation: the app works in a tunnel, on a plane, or in a rural clinic with no signal. Cost: there is no per-call server bill — the user's own hardware does the work.

But moving on-device is a Faustian bargain. The moment you commit to inference on a phone, you inherit a brutal trio of constraints. Memory is tiny: a phone may give your app a few hundred megabytes, not the tens of gigabytes a server GPU has, so a 400 MB model simply will not load. There is no datacenter cooling: a phone is a sealed slab of glass, so after a few seconds of heavy compute it heats up and the OS thermal-throttles — it deliberately slows the chip to protect it, and your beautiful 30 FPS quietly collapses to 12 FPS. Energy is rationed: every inference drains a battery the user expects to last all day, so we care not just about milliseconds but about millijoules per frame. On the edge, you are an engineer working inside a shoebox, in the heat, on a budget.

This is the master's playbook, so we set an ambitious but realistic bar. We will take a model that today runs at ~180 ms per frame on a phone CPU (about 5 FPS — a stuttery slideshow) and, by stacking every technique from this track, land it comfortably under 33 ms on the phone's neural accelerator while losing barely any accuracy. That is roughly a 6–10× speedup. By the end you should be able to look at any vision model and a target device and answer, with numbers: will it fit, will it be fast enough, and how do I get it there?

Mobile Inference Stacks

To run on-device you do not write raw GPU code; you hand your model to a mobile runtime that knows how to drive that phone's chips. The landscape, conceptually, is small. On Android the common path is TFLite (a lightweight interpreter) talking to NNAPI, the OS layer that finds whatever accelerator the phone has. On Apple devices the equivalent is Core ML, which automatically spreads work across the CPU, the GPU, and the Apple Neural Engine. Both can also use a GPU delegate to run work on the mobile GPU via shaders. And underneath the best phones sits an NPU — a neural processing unit, the mobile cousin of the dedicated accelerators you met in guide 4, a block of silicon built to do nothing but matrix-multiply, fast and cheaply. Forget the exact API names; hold one mental model — which engine runs on which silicon.

Here is the single most important fact about mobile, and it ties straight back to guide 3: INT8 quantization is effectively mandatory. Mobile NPUs are integer-first machines — they are physically built to multiply 8-bit integers, and many of them either run FP32 very slowly or cannot touch it at all, kicking those layers back to the CPU. So 8-bit integer inference is not an optional polish step on a phone the way it can feel on a server; it is the price of admission to the NPU. A model you have not quantized is a model the fast silicon will largely refuse to run. This is why, on the edge, quantization graduates from "nice optimization" to "non-negotiable design decision made early."

So how does the runtime decide what goes where? Through delegates. A delegate is a plug-in that says "I can run these operations on this accelerator." When the runtime loads your model, it walks the graph op by op: every op the delegate supports gets routed to the NPU (or GPU); every op it does not support stays on the CPU. This is exactly the fallback mechanism we warned about in guide 4 — and it is double-edged. It is wonderful that an unsupported op (say, an exotic activation) does not crash your app; it just runs slower on the CPU. But if that one CPU op sits in the middle of your network, every tensor must be copied NPU→CPU→NPU around it, and those transfers can cost more than the op you were trying to accelerate.

# Pseudocode: load an INT8 model and attach an accelerator delegate
interpreter = Interpreter(model="detector_int8.tflite")

# Prefer the integer NPU; degrade gracefully if it's missing
try:
    interpreter.add_delegate(NpuDelegate())   # routes INT8 ops to the NPU
except DelegateUnavailable:
    interpreter.add_delegate(GpuDelegate())   # fall back to fp16 on the GPU

interpreter.allocate_tensors()

# Ops the delegate can't run silently stay on the CPU.
# ALWAYS profile to confirm the heavy conv layers actually landed on the NPU,
# and that the graph isn't fragmented into costly NPU<->CPU hops.
report = interpreter.profile_ops()
assert report.fraction_on_npu > 0.90, "graph fragmented: transfer overhead may dominate"

Designing for the Budget: Latency and Energy

Masters work backward from a budget. We have 33 ms per frame; what fills it? A rookie mistake is to think "the model" is the whole cost. In reality the wall-clock latency of one frame has three parts: preprocessing (decode the camera buffer, resize to the network's input size, normalize the pixels), inference (the actual forward pass), and postprocessing (for a detector, decoding raw outputs into boxes and running non-maximum suppression to remove duplicates). Decode/resize and NMS are pure CPU work and can easily eat 30–40% of your budget. If you only ever optimize the model, you can make inference twice as fast and still miss 30 FPS because the other two stages never moved.

End-to-end latency is a sum of stages; frame rate is its reciprocal.

Read the first equation as "the time for one frame is preprocessing time plus inference time plus postprocessing time." Here L_{\text{total}} is the total seconds per frame, L_{\text{pre}} the preprocessing time, L_{\text{infer}} the forward pass, and L_{\text{post}} the postprocessing. The second equation says frames-per-second is simply one divided by the per-frame time — if each frame takes a quarter-second, you get 4 FPS. Let's put numbers on our detector: suppose L_{\text{pre}}=5 ms, L_{\text{infer}}=20 ms, L_{\text{post}}=6 ms. Then L_{\text{total}}=31 ms and \text{FPS}=1/0.031\approx 32 — we just clear 30 FPS. Now notice: if we halved inference to 10 ms but ignored the rest, L_{\text{total}}=21 ms → 47 FPS, but the 11 ms of CPU pre/post is now more than half the budget and becomes the next thing to attack. The equation tells you exactly where to spend your effort.

Now the analytical peak of the whole track. Why is one model fast and another, with the same number of operations, slow? The answer is whether it is compute-bound (limited by how fast the chip can multiply) or memory-bound (limited by how fast it can fetch numbers from memory). The lens for this is arithmetic intensity: how many arithmetic operations you do per byte you move. A model with low intensity moves a lot of data for little math, so the multipliers sit idle waiting for memory — that is the silent killer on the edge, because moving bytes also burns the most energy. Counting FLOPs alone is not enough; you must weigh them against bytes moved on this device's accelerator.

Arithmetic intensity and the roofline: you are capped by raw compute OR by intensity times bandwidth, whichever is smaller.

Unpack it. I is arithmetic intensity, in FLOPs per byte — total floating-point operations divided by total bytes shuttled between memory and the compute units. P_{\text{peak}} is the chip's maximum compute rate (FLOPs/s), \text{BW} is its memory bandwidth (bytes/s), and P_{\text{attainable}} is the best speed you can actually hit. The roofline says your achievable performance is the smaller of two ceilings: the flat compute roof P_{\text{peak}}, and the slanted memory roof I\cdot\text{BW}. The crossover (the "ridge") is at I = P_{\text{peak}}/\text{BW}. Below it you are memory-bound (the slanted roof bites); above it you are compute-bound (the flat roof bites). Worked example: a layer does 2\times10^{9} FLOPs and, in FP32, moves 200 MB, so I = 2\times10^{9}/2\times10^{8} = 10 FLOP/byte. On a device with P_{\text{peak}}=1\text{ TFLOP/s} and \text{BW}=50\text{ GB/s}, the ridge sits at 10^{12}/(5\times10^{10}) = 20 FLOP/byte. Since 10 < 20, we are memory-bound, and P_{\text{attainable}} = 10\times 5\times10^{10} = 0.5\text{ TFLOP/s} — only half the chip's power, the rest wasted waiting on memory.

Here is the punchline that ties quantization to the roofline. Quantizing FP32 → INT8 shrinks every weight and activation by 4×, so bytes moved drop 4× while the FLOP count stays the same. In our example, bytes fall from 200 MB to 50 MB, so I rises from 10 to 2\times10^{9}/5\times10^{7} = 40 FLOP/byte. Now 40 > 20: we have crossed the ridge and become compute-bound, free to use the chip's full integer throughput instead of starving on memory. That is the deep reason quantization is so potent on the edge — it is not just a smaller file; it literally moves your model rightward across the roofline, rescuing a memory-bound network and, because fewer bytes cross the bus, cutting energy per inference at the same time.

The Full Stack Working Together

Now we assemble everything. The thesis of this whole track is that the levers stack — and they stack best in a deliberate order, because each one prepares the model for the next. Applied to our detector, the master pipeline runs: distill → structured-prune → quantize → export → compile. Crucially, you do not do these blind and once; you fine-tune to recover a little accuracy after each lossy step, so errors do not compound. Think of it as renovating a house: change the structure first (walls), then the fittings (paint), never the reverse.

1. Distill: train a small "student" detector to mimic a large accurate "teacher" via knowledge distillation, so the small architecture starts out punching above its weight.
2. Structured-prune: remove whole channels/filters with structured pruning so the network is genuinely smaller and faster on real hardware (not just sparse-on-paper), then fine-tune to recover accuracy.
3. Quantize: convert weights and activations to INT8 with quantization — try fast post-training quantization (PTQ) first; if accuracy drops too far, use quantization-aware training (QAT) to fine-tune the model to live in 8 bits.
4. Export: serialize the finished model to a portable graph (ONNX or the runtime's native format) so it leaves the training framework cleanly.
5. Compile: build a device-specific engine (e.g. TensorRT for an NVIDIA edge board, or a TFLite/Core ML model for the phone) so kernels are fused and tuned for the exact target silicon.

*Why this order, and not any other?* Distillation and pruning change the architecture — how many layers and channels exist — so they must come first; there is no point quantizing channels you are about to delete. Quantization comes last among the lossy steps because it depends on the final weight distribution: prune first and the surviving weights settle into new values, and only then do you calibrate the 8-bit ranges around them. And you fine-tune after each step because every lossy transform nudges accuracy down a little; recovering between steps stops a small loss from snowballing into a useless model. Put crudely: shape the architecture, then compress the numbers, then lower the precision, retraining briefly at each gate. This is the difference between compression that holds accuracy and compression that quietly wrecks it.

Let's make the payoff concrete with a realistic before/after. Before — the FP32 cloud model: 25 million parameters at 4 bytes each ≈ 100 MB, mAP 37.0, running at 180 ms per frame on a phone CPU (≈5 FPS, useless for live preview). After — distilled, 40% structured-pruned (down to ~15 M params), then INT8-quantized and compiled for the NPU: the file is ~15 MB (about 6.7× smaller, easily fitting in the app's memory), latency is ~18 ms on the NPU (a 10× speedup, comfortably under the 33 ms wall), and accuracy lands at mAP ≈ 35.5 — a loss of only ~1.5 points, recovered largely by the distillation and per-step fine-tuning. Size, latency, and accuracy moved together, and that simultaneous movement is the entire thesis of this track proven on one model.

Two Worlds: Throughput at Scale vs Latency at the Edge

You now understand two deployment worlds deeply enough to see they are opposites. In guide 1 we separated latency (how long one request takes) from throughput (how many requests per second you finish). On the server, you optimize throughput: thousands of users send images, and you can wait a few milliseconds to gather many of them into one big batch, so the GPU's thousands of cores all work at once. On the edge, there is exactly one user and one camera; frames must be answered now, one at a time, at batch size 1, and latency is everything. Same models, opposite objectives.

Why does batching help the server but barely help the edge? A GPU or accelerator pays a fixed overhead to launch work and to load weights from memory; if it loads the weights once and then reuses them across 64 images, that cost is amortized 64 ways and per-image cost plummets — this is exactly arithmetic intensity from Section 3, batching raises it. But at batch 1 there is nothing to amortize over, the weights are loaded for a single image, and the accelerator runs at low utilization. On the edge you cannot batch (you have one live frame), so you instead chase utilization a different way: a tiny model, INT8, and a compiled engine that keeps even a single-stream forward pass busy. The server hides latency behind throughput; the edge has nowhere to hide.

Server economics: unit cost falls as throughput rises, which is why servers batch aggressively.

Read it as "the cost of one inference equals what the machine costs per hour, divided by how many inferences it completes per hour." C_{\text{hour}} is the hardware (rental) cost per hour in dollars, \text{throughput} is inferences finished per second, and the \times 3600 converts seconds to an hour. Worked example: a cloud GPU costs C_{\text{hour}}=\4. Batched well, it hits a throughput of 2000 inferences/s, so cost per inference = 4/(2000\times3600) \approx \5.6\times10^{-7} — about \\(0.56 per million images. Run the same GPU at batch 1 and throughput collapses to, say, 400/s, giving \)4/(400\times3600)\approx\2.8\times10^{-6} — about \$2.78 per million, five times pricier. That factor of five is exactly why a server operator batches relentlessly: throughput is money. The edge does not see this bill at all — the user's phone is the hardware — which is precisely why on the edge we optimize the other axis, single-stream latency, and treat throughput as irrelevant.

The Master's Deployment Checklist

One last graduation. Knowing how to compress and compile a model makes you a strong engineer; running vision in production over months, as the world changes around your model, makes you a master. A shipped model is not a finished artifact — it is a living service that can silently rot. The discipline below is what separates a clever benchmark from a system people can depend on. Notice that the very first item makes the lessons of this track permanent: you do not just achieve a latency target once, you defend it forever.

1. Regression-test BOTH accuracy and latency in CI: every model change runs an automated suite that fails the build if mAP drops below a floor OR if p95 latency creeps above the 33 ms budget — speed is a test, not a hope.
2. Monitor for data and concept drift after deployment: the real world stops matching your training set (new phone cameras, night scenes, fashions), so track input statistics and prediction confidence and alarm when they wander.
3. Version models and keep a safe rollback: tag every deployed model, ship via staged rollout, and be able to revert to the previous version in minutes if metrics or crash rates spike.
4. A/B compare the new compressed model against the old one on live traffic before full rollout, so you measure real-world accuracy and engagement, not just offline benchmarks.
5. Collect on-device telemetry for real-world p95 latency: your lab phone is not the user's three-year-old, thermally throttled phone — measure the tail latency users actually feel, across the real device fleet.

Every item on that list rests on one habit you have carried since the start: profiling as an always-on feedback loop, not a one-time experiment. You profiled to find the bottleneck before compressing; you profile in CI to guard latency; you profile on-device telemetry to catch thermal throttling in the wild. Profiling is the instrument panel of edge deployment — without it you are flying blind, optimizing things that don't matter and missing the regression that does. Measure first, always.