chain rule

The chain rule tells you how to find the slope of a chain of dependencies. Suppose a change in A causes a change in B, and a change in B causes a change in C. How much does C move when you nudge A? The chain rule's answer is beautifully simple: multiply the rates together. If A moving causes B to move twice as fast, and B moving causes C to move three times as fast, then A moving causes C to move six times as fast.

Think of a sequence of gears. Turn the first gear, it spins the second, which spins the third. The final gear's speed is the product of all the gear ratios along the way. The chain rule is exactly that logic written for slopes: to differentiate a function nested inside another function, you differentiate the outer layer, then multiply by the derivative of what's inside, peeling the onion one layer at a time.

Without the chain rule, deep learning would be impossible. A neural network is a deep stack of functions feeding into one another — the output of one layer is the input to the next, often dozens of layers deep. To train it, you need the derivative of the final error with respect to a weight buried near the very beginning. The chain rule lets you get there by multiplying together the local slopes along the path. Backpropagation is essentially the chain rule applied with disciplined bookkeeping. The flip side is a real danger: multiply many small slopes and the product shrinks toward zero (the vanishing gradient problem), so early layers can stop learning.