ReLU

ReLU is the simplest useful activation function imaginable, and it powers most of modern deep learning. The rule is one line: if the input is positive, pass it through unchanged; if it's negative, output zero. That's it. Picture a one-way valve — signal flows freely in the positive direction and is blocked entirely in the negative. Despite its plainness, this little bend is enough to give a network all the nonlinearity it needs.

Why did ReLU dethrone the smooth S-curves? Two reasons. First, it doesn't saturate on the positive side — for any positive input the slope is exactly 1, so the learning signal passes back through unweakened, dodging the vanishing-gradient problem that plagued sigmoid and tanh in deep networks. Second, it's dirt cheap to compute: just "keep it or zero it," with none of the costly curves of an exponential. Together these let researchers train networks far deeper than before, which is much of what made the deep-learning era possible.

ReLU isn't flawless. Because it outputs a flat zero for all negative inputs, a neuron can sometimes get stuck always outputting zero — a so-called "dead" neuron that never recovers, since zero output means zero learning signal. A common fix is leaky ReLU, which lets a small trickle through on the negative side (say, the input times 0.01) instead of a hard zero, keeping the neuron alive. Variants like this are everywhere, but the plain ReLU remains the sensible default.