rectified linear unit

The rectified linear unit, ReLU, is the simplest useful nonlinearity: it passes positive numbers through unchanged and clamps negative numbers to zero, ReLU(z) = max(0, z). Picture a one-way valve, or a diode in electronics: signal flows when the input is positive and nothing flows when it is negative. This trivial-looking rule turned out to be one of the most important practical discoveries in deep learning, because networks built with it train far more easily than those built from the smooth squashing functions that preceded it.

Two properties matter. First, sparsity: for a typical input roughly half the neurons output exactly zero, giving an efficient representation in which only the relevant units are active. Second, and more importantly, the gradient of ReLU is exactly 1 for every positive input. During backpropagation each neuron multiplies the incoming gradient by phi'(z); with sigmoid that factor is at most 0.25 and usually far smaller, so gradients shrink exponentially with depth (the vanishing-gradient problem). ReLU's gradient of 1 lets the error signal pass undiminished through the active path, which is what made training very deep networks practical.

ReLU's weakness is the mirror image of its strength. For negative inputs the gradient is 0, so if a neuron's weights drift to a region where it outputs negative for every training example, it receives zero gradient forever and never recovers, a dead neuron, the dying-ReLU problem, often triggered by too-large learning rates. Fixes keep a small slope on the negative side: leaky ReLU uses max(alpha·z, z) with a fixed small alpha such as 0.01; PReLU learns alpha; and ELU, GELU, and SiLU smooth the corner entirely, which can help optimization and is now standard in Transformers and many modern CNNs.

ReLU and its kin are the default hidden activation throughout classic vision architectures: AlexNet, VGG, and the ResNet family all use plain ReLU after each convolution (and after batch normalization). Its cheapness matters at scale; applied billions of times per forward pass over high-resolution feature maps, the activation itself should be negligible in cost, and max(0,z) is about as cheap as an operation gets.