tensorrt

A model that runs correctly on an NVIDIA GPU is usually nowhere near as fast as the GPU can go. TensorRT is NVIDIA's inference optimizer and runtime: you feed it a trained model (often via ONNX) and it compiles a highly optimized "engine" specialized to your exact GPU, chosen precision, and input shapes — frequently several times faster than naive PyTorch inference of the same network.

Its main optimizations are four. (1) Layer and tensor fusion: collapse conv+bias+activation, or the pieces of attention, into a single kernel to cut kernel-launch overhead and memory round-trips. (2) Precision lowering: run in FP16, int8 (with calibration), or FP8, choosing precision per layer to balance speed and accuracy. (3) Kernel auto-tuning, called tactic selection: benchmark several candidate kernel implementations on the actual GPU and keep the fastest. (4) Memory planning: reuse activation buffers to shrink the footprint. The output is a serialized engine you load and run.

The cost is portability. An engine is specialized to a GPU architecture and TensorRT version — one built for Ampere is not optimal or loadable on Hopper — and the build step itself can be slow; dynamic shapes require predefined optimization profiles. TensorRT is the standard for real-time vision on NVIDIA hardware: detection, segmentation, and autonomous driving (DRIVE), and it integrates via Torch-TensorRT and is commonly served through the Triton Inference Server.