computational graph

A computational graph is a map of how a result is computed, drawn as a network of operations. Each node is either an input or parameter, or an operation (add, multiply, matrix-multiply, ReLU, and so on), and arrows show which values feed into which operations. Reading the graph left to right tells you how to compute the output; the graph is the bookkeeping that lets a framework remember every step it took, which is exactly what is needed to later compute derivatives automatically.

Formally it is a directed acyclic graph (DAG): directed because data flows one way along the edges, acyclic because there are no loops (a value cannot depend on itself). Leaf nodes are inputs and learnable parameters; internal nodes are primitive operations whose local derivatives are known; the root is the output (for training, the scalar loss). Each node stores enough to compute both its forward value and, given the gradient flowing back into it, the gradients to send to its inputs (its vector-Jacobian product). A whole neural network, every layer, activation, and the loss, is one such graph.

Frameworks build the graph in two styles. A static graph (TensorFlow 1.x, and what tracing or compilation such as torch.compile, JAX's jit, or ONNX export produce) is defined once and then executed many times, which enables whole-graph optimization, fusion, and deployment. A dynamic graph (PyTorch's eager mode, define-by-run) is built afresh on every forward pass as the Python code runs, so it can change shape with the data and use ordinary control flow, easier to debug at some runtime cost. Modern systems blur the line by tracing dynamic code into a static graph for speed.

The graph is the substrate for automatic differentiation: backpropagation is just a traversal of this graph in reverse. It also enables the systems-level magic of deep learning: operator fusion, memory planning, gradient checkpointing (trading compute to store fewer intermediate nodes, vital for training large ViTs and diffusion models), and compiling the same graph to GPU, TPU, or mobile NPU. When you debug shape mismatches or out-of-memory errors in a vision model, you are really debugging this graph.