onnx

You train a vision model in PyTorch but want to deploy it in C++ on a server, or on a phone, or feed it to TensorRT — without rewriting the model in each place. ONNX (Open Neural Network Exchange) is a standard file format that captures a model's computation graph and weights so it can move between frameworks; ONNX Runtime (ORT) is a fast engine that executes those graphs across many hardware backends. Together they are the lingua franca of model deployment.

An ONNX model is a serialized computation graph (stored as protobuf) built from standardized operators — Conv, MatMul, Relu, Resize, Attention, and so on — versioned by an "opset," together with the weights stored as initializers. You produce one with torch.onnx.export, tf2onnx, or similar. ONNX Runtime then applies graph optimizations (constant folding, operator fusion, layout transforms) and dispatches the work to pluggable "execution providers" — CPU, CUDA, TensorRT, DirectML, CoreML, NNAPI, OpenVINO — so the same file runs across very different platforms.

The friction is operator coverage. Custom ops, dynamic control flow, or exotic layers may not export cleanly, or may export but fall back to a slow generic kernel; dynamic input shapes and opset-version mismatches are common debugging headaches. ONNX also standardizes quantized operators (the QDQ — quantize/dequantize — representation), so int8 PTQ models are portable too. In most production stacks ONNX is the neutral intermediate step before TensorRT, a mobile runtime, or an edge engine.