activation function
/ ak-tih-VAY-shun FUNK-shun /
An activation function is the little decision step a neuron applies to its sum before passing the result on. After a neuron adds up its weighted inputs and bias, it has a single number — but a raw number isn't very expressive. The activation function reshapes it: maybe squashing it between 0 and 1, maybe chopping off anything negative. Think of a dimmer switch that doesn't respond evenly — quiet at first, then suddenly bright. That non-even response is the point.
The reason this matters is subtle but central. Adding and scaling numbers — what weights and biases do — can only ever produce straight-line relationships. Stack a hundred such layers and you still get one straight line, no better than a single layer. The activation function bends that line. By inserting a curve between layers, it lets the network build up wiggly, complex shapes — the kind real data actually has. This bending is called nonlinearity, and it's the single ingredient that makes deep networks more powerful than plain arithmetic.
There's a whole family of these functions, each a different curve: the smooth S-shaped sigmoid and tanh, the simple bent-line ReLU and its leaky cousin, and softmax for turning scores into probabilities. Choosing among them affects how easily a network learns. For many years sigmoid and tanh dominated; today ReLU is the default for hidden layers, largely because it trains faster and sidesteps a problem where learning signals fade away in deep stacks.
Try to learn the XOR pattern (output 1 when exactly one of two inputs is on) using only weighted sums — you can't; no single straight line separates the cases. Add one hidden layer with a nonlinear activation, and the network solves it easily. That tiny example, which stumped early researchers in the 1960s, is the cleanest demonstration of why activations are not optional.
Without a nonlinear activation, stacking layers gains you nothing — they collapse into one.
If every activation were linear, a deep network would be mathematically identical to a single layer — no matter how many layers you add. The nonlinearity, not the depth alone, is what unlocks expressive power.