activation function

An activation function is the nonlinear decision step applied to a neuron's weighted sum. After a neuron computes z = w·x + b it does not output z directly; it passes z through a function phi that bends or squashes it. Intuitively this is the part that lets a neuron say something more interesting than the input scaled linearly: it can say fire only if z is positive, or saturate toward 1 for large z, or respond gently around zero. Without this bend, no matter how many layers you stack, you could still only draw straight decision boundaries.

The reason is algebraic: a composition of affine maps is itself an affine map, so a deep network of purely linear layers collapses to a single linear layer. Inserting a nonlinear phi between layers breaks that collapse and gives the network its expressive power; indeed the universal approximation theorem requires phi to be nonlinear (and non-polynomial). The activation must also be differentiable almost everywhere so gradients can flow back through it during training, because backpropagation multiplies by phi'(z) at each neuron, and the shape of phi' directly controls how well gradients survive.

Historically the sigmoid sigma(z)=1/(1+e^-z) and tanh were standard, squashing inputs into (0,1) or (-1,1); their flat tails cause vanishing gradients, which made deep nets hard to train. The rectified linear unit ReLU(z)=max(0,z) replaced them in most vision work (from AlexNet in 2012 onward) because its gradient is exactly 1 for positive inputs, so there is no saturation on that side, and it is cheap to compute. Variants address ReLU's weaknesses: leaky ReLU and PReLU keep a small slope for negatives; ELU and GELU smooth the kink (GELU is the default in Vision Transformers and most modern Transformers); and SiLU/Swish, x·sigma(x), is common in efficient CNNs.

A subtle but crucial point: the activation used inside hidden layers (chosen for trainability, usually from the ReLU family) differs in purpose from the function on the final layer (chosen to match the task: softmax for mutually exclusive multi-class probabilities, sigmoid for independent binary labels, or identity/none for regression). Choosing the wrong one, such as a ReLU on a regression output that needs negative values, or a softmax when labels are not mutually exclusive, silently breaks the model.