Runtimes and Accelerators: ONNX, TensorRT, and the Hardware Stack

From Framework to Runtime: The Deployment Graph

When you train a model in PyTorch or TensorFlow, you are running ordinary Python. Each line — a convolution here, a ReLU there — executes immediately, one operation at a time. This is called eager execution, and it is wonderful for research: you can print a tensor mid-network, set a breakpoint, change the architecture between two runs. But that same flexibility is exactly what makes a model slow and heavy to ship. Every forward pass pays for the Python interpreter, for autograd bookkeeping it no longer needs, and for a giant framework that was built to train, not to serve.

For deployment we do something different: we export the model once into a frozen computational graph, then hand that graph to a lean inference runtime. A computational graph is just a directed diagram of the math your model does. The nodes are operators — `conv`, `relu`, `add`, `matmul` — and the edges are the tensors that flow between them (one operator's output is the next one's input). Nothing about the numbers changes; we have simply written down the fixed sequence of steps instead of re-deciding it in Python on every single inference.

This frozen graph is the raw material every deployment tool works on. Over this guide we'll follow it through the whole stack: first ONNX, a standard file format so the graph isn't trapped in one framework (run by ONNX Runtime); then graph optimizations that rewrite the graph to be faster; then vendor engines like TensorRT that compile it for one exact chip; then the art of matching the graph to the hardware; and finally profiling to confirm the speedup is real. The single number we are trying to drive down throughout is inference latency — how long one prediction takes.

ONNX: A Common Language for Models

A model trained in PyTorch is, by default, a PyTorch thing — to run it you need PyTorch installed. ONNX (Open Neural Network Exchange) breaks that lock-in. It is a single standard file format that describes the computational graph — the same nodes-and-edges idea from Section 1 — in a way that is independent of the framework that produced it. Export your PyTorch model to a `.onnx` file and you can run it with ONNX Runtime on a Windows laptop, a Linux server, or inside a C++ app, with no PyTorch anywhere in sight. ONNX decouples where a model was trained from where it runs.

import torch

model.eval()  # turn off dropout / training-mode batchnorm behaviour
dummy = torch.randn(1, 3, 224, 224)  # one example with the input shape

torch.onnx.export(
    model, dummy, "model.onnx",
    input_names=["image"], output_names=["logits"],
    opset_version=17,                      # which operator dictionary to use
    dynamic_axes={"image": {0: "batch"}},  # let batch size vary at run time
)

Notice the `opset_version` argument. An opset (operator set) is a versioned dictionary of exactly what each operator means — what inputs `Conv` takes, how `Resize` rounds, and so on. ONNX evolves, so operators get added and refined over time, and each release is stamped with an opset number. Export and runtime must agree: if you export with opset 17 but your runtime only understands up to opset 13, or you use a brand-new operator that an old opset never defined, the export or the load simply fails. Most 'my ONNX export crashed' problems are an opset mismatch — bumping or lowering the opset version is the first thing to try.

ONNX Runtime is the engine that actually executes the file, and its key trick is execution providers (EPs) — pluggable backends that each know how to run operators on a particular piece of hardware. The CPU EP runs ops on the processor; the CUDA EP runs them on an NVIDIA GPU; there are EPs for other hardware accelerators too. You hand ONNX Runtime a priority list of EPs, and for each operator it picks the highest-priority provider that supports it, falling back down the list when one can't. The same `model.onnx` therefore runs on a CPU-only box and on a GPU server — you just change the provider list, not the model.

import onnxruntime as ort

sess = ort.InferenceSession(
    "model.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)  # try the GPU first; fall back to CPU for any op CUDA can't run
out = sess.run(None, {"image": batch})[0]

Graph Optimizations: Fusion and Folding

Here is the surprising part: a runtime can make your model meaningfully faster without changing a single number it computes. The trick is to rewrite the graph into an equivalent but cheaper-to-execute form. ONNX Runtime, TensorRT, and every serious engine run a battery of these graph optimizations when they load your model. The two most important are operator fusion and constant folding.

Operator fusion merges several adjacent operators into one. A classic pattern — visible in the residual block above — is `Conv → BatchNorm → ReLU`. Run unfused, the GPU computes the whole Conv output, writes that entire tensor out to slow global memory, reads it all back to do BatchNorm, writes it again, reads it a third time to do ReLU, and writes once more. Fused, a single kernel does conv-then-normalize-then-clamp while each little patch of data is still in fast on-chip registers, and writes the final result once. Recall the memory-bandwidth bottleneck from Guide 1: for these layers the chip spends most of its time moving data, not multiplying it, so cutting three round-trips to memory down to one slashes inference latency even though the arithmetic is identical.

Constant folding does at build time any computation whose inputs are already known, so it never runs at inference. The headline example is folding BatchNorm into the preceding convolution. At inference, BatchNorm is just a fixed per-channel scale and shift (its running mean and variance are frozen). Algebraically, scaling-and-shifting a convolution's output is the same as using slightly rescaled conv weights plus a tweaked bias — so the engine pre-computes those adjusted weights once when it builds the graph, and the BatchNorm node disappears entirely. Same outputs, one fewer operator forever.

Beginners often ask: if fusion doesn't remove any multiply-adds, why is it faster? Two reasons, both about overhead rather than FLOPs. First, memory traffic — as above, fewer intermediate tensors means far fewer trips to slow DRAM, and many vision layers are memory-bound, not compute-bound. Second, kernel-launch overhead — every separate operator is a GPU kernel launch with a fixed setup cost; firing one fused kernel instead of three removes two launches. Engines like TensorRT fuse extremely aggressively for exactly these reasons.

TensorRT and Vendor Engines

TensorRT is NVIDIA's vendor engine, and it goes a step beyond ONNX Runtime. ONNX Runtime mostly interprets your graph — it has a good kernel for each operator and dispatches to it. TensorRT instead compiles your graph into a single hardware-specific engine file, purpose-built for one exact GPU. Think of the difference between running an interpreted script versus compiling a program with every optimization flag on for your precise CPU.

1. Autotuning kernels: for each layer TensorRT has many candidate implementations (different tiling, algorithms, data layouts). It actually benchmarks them on your GPU and keeps the fastest for that exact shape and chip.
2. Per-layer precision selection: it can run some layers in FP16 or INT8 and others in FP32, choosing per layer whatever is fastest while staying accurate enough.
3. Aggressive fusion: it fuses conv+bias+activation and many other patterns even more thoroughly than a general runtime, because it only has to be correct on one hardware target.

That per-layer precision is where Guide 3 comes back. To run a layer in INT8 inference, TensorRT must know how to map each tensor's floating-point range onto the 256 integer levels — it needs a scale for every activation. It learns those scales with a calibration pass: you feed it a few hundred representative images, it records the real range of activations flowing through each layer, and picks scales that lose the least information. This is exactly the post-training quantization calibration from Guide 3, now performed automatically inside the engine builder. No calibration data, no safe INT8.

# Build an INT8 engine from an ONNX file, tuned for THIS machine's GPU.
trtexec \
  --onnx=model.onnx \
  --int8 \
  --calib=calib.cache \        # scales learned from representative images
  --saveEngine=model.plan       # the compiled, GPU-specific engine

The catch: a TensorRT engine is married to the hardware it was built on. The `.plan` file is tied to a specific GPU architecture and often a specific TensorRT/driver version — move it to a different accelerator and it may refuse to load, so you rebuild on each target. Building also takes real time (seconds to many minutes) because of all that autotuning. You pay that cost once, at deploy time, in exchange for the lowest latency available on that chip. For a service that will answer millions of requests, it is almost always worth it.

Matching the Model to the Accelerator

A trained graph and an accelerator are not a guaranteed match. The same exported model can be blisteringly fast on one chip and embarrassingly slow on another — not because the chip is weaker, but because the model uses operators or shapes the chip isn't built to accelerate. Getting good performance is partly a co-design problem: you shape the model with the target hardware in mind, the way you'd design furniture to fit through the door it has to pass.

Accelerators only go fast on the operations they have dedicated circuitry for. A GPU's tensor cores are specialized matrix-multiply units that scream on convolutions and matmuls — but only when shapes line up with their tile sizes (e.g. channel counts that are multiples of 8 or 16) and the precision is one they support (FP16/INT8). A mobile NPU is even pickier: it implements a fixed, limited menu of operators, and anything off that menu it simply cannot run on its fast path.

1. Prefer hardware-friendly ops: standard conv, ReLU, and matmul are accelerated everywhere; exotic activations and fancy custom layers often are not. Swap them for supported equivalents when you can.
2. Align shapes to the hardware: keep channel counts multiples of 8 or 16 so tensor cores stay full; pad rather than leave an awkward 13-channel layer.
3. Check the op-support matrix first: before you commit to an architecture, read the supported-operator list for your TensorRT version or NPU SDK, and confirm ONNX Runtime (or your engine) can place every op on the accelerator.

Internalize this: the very same `.onnx` file can be fast or slow purely depending on this matching. Two engineers can deploy identical weights and see a 10x latency gap because one kept every op on the accelerator and the other left a fallback in the hot path. Hardware-aware design isn't a final polish — it's a constraint you carry from the first architecture sketch.

Profiling the Deployed Engine: Closing the Loop

You've exported, fused, compiled, and matched. Now — exactly as in Guide 1 — you measure, because none of those optimizations are real until you've seen them in the numbers. The tool for this is layer-level profiling: instead of one stopwatch around the whole network, you time every operator in the engine and see where the milliseconds actually go.

A good layer-level profile answers three deployment questions at once. Did fusion happen? If the optimized graph still shows separate Conv, BatchNorm, and ReLU rows, your fusion silently didn't fire — maybe an unsupported pattern blocked it. What's the bottleneck now? The one or two layers eating most of the time are where further work pays off; everything else is noise. Did anything fall back to CPU? Ops running on the CPU provider, or sudden gaps where data is copied off the accelerator, expose the silent fallbacks from Section 5.

import onnxruntime as ort

opts = ort.SessionOptions()
opts.enable_profiling = True            # record per-op timings
sess = ort.InferenceSession(
    "model.onnx", opts,
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

for _ in range(50):                     # warm up, then run
    sess.run(None, {"image": batch})

prof = sess.end_profiling()             # JSON trace: per-op time + which EP ran it
# Open the trace, sort ops by duration, and look for CPUExecutionProvider rows.

Finally, sweep the batch size on the finished engine. Recall Guide 1's two speeds: latency is how long one request waits; throughput is how many predictions per second the engine sustains. Larger batches keep the accelerator's compute units fuller, so throughput climbs — but each individual request now waits for the whole batch, so latency rises. Run the engine at batch 1, 2, 4, 8, 16… and plot both. The right batch size is the largest one that still meets your latency budget; that's the single knob that trades the two speeds against each other on this exact engine.

1. Profile the optimized engine layer by layer.
2. Confirm the wins: fusion fired, and there are no surprise CPU fallbacks.
3. Find the new bottleneck op and address it (different op, shape, or precision), or sweep batch size for the throughput you need.
4. Re-export / re-build, re-measure, and repeat until you hit your latency and throughput targets.

This deploy-measure-tune loop is what turns a model that merely runs into one that is genuinely fast on its target. So far we've assumed a generous server with a big GPU. Guide 5 takes everything here — export, fusion, quantization, op-matching, profiling — to the harshest target of all: the edge, where memory is tiny, power is scarce, and there's no falling back to a beefy CPU. The discipline is the same; the constraints just get unforgiving.