automatic differentiation

Automatic differentiation (autodiff) is how frameworks compute exact derivatives of code without you doing any calculus by hand. You write the forward computation (a sequence of additions, multiplications, matrix multiplies, activations) and autodiff mechanically applies the chain rule to that exact sequence to produce the gradient. It is neither the error-prone symbolic differentiation of a math package (which can blow up into huge expressions) nor the inaccurate finite-difference approximation; it computes the true derivative, to floating-point precision, by composing the known derivatives of each elementary operation.

Autodiff works because every program, however complex, is ultimately a composition of primitive operations whose local derivatives (Jacobians) are known. The chain rule says the derivative of a composition is the product of the local derivatives along the path. There are two modes for accumulating that product: forward mode walks the graph input-to-output and is efficient when there are few inputs and many outputs; reverse mode walks output-to-input and is efficient when there are many inputs and few outputs. Reverse mode applied to a scalar loss is exactly backpropagation.

For neural networks (millions to billions of parameters as inputs but a single scalar loss as output) reverse mode is overwhelmingly the right choice: it computes the gradient with respect to all parameters in one backward pass costing roughly the same as one forward pass (a constant-factor overhead, typically 2 to 4 times, plus the memory to store intermediate activations). Forward mode would need one pass per parameter. This favorable cost is why training enormous vision models is even possible. The price is memory: reverse mode must keep intermediate activations from the forward pass to use during the backward pass.

Autodiff is the engine inside every deep-learning framework (PyTorch autograd, TensorFlow, JAX with grad, jit, and vmap), and it is what frees researchers to invent new architectures (ViT, DETR, NeRF, Gaussian splatting, diffusion samplers) and just call backward, trusting the gradients are correct. Practical notes: non-differentiable points (ReLU's kink, max-pooling's argmax) are handled with subgradients or by routing gradient to the selected element; truly non-differentiable operations (hard sampling, argmax) need workarounds like the straight-through estimator or reparameterization; and memory pressure is managed with gradient checkpointing.