sigmoid activation

The sigmoid (logistic) function squashes any real number into the open interval (0, 1), producing a smooth S-shaped curve: sigma(z) = 1/(1 + e^-z). Large positive z maps near 1, large negative z near 0, and z=0 maps exactly to 0.5. Because the output looks like a probability, sigmoid is the natural choice when you want a neuron to express how likely is this, for instance the probability that a pixel belongs to the foreground or that an image contains a cat (a single yes/no question).

Sigmoid has the elegant derivative sigma'(z) = sigma(z)·(1 - sigma(z)), which is convenient because the forward output already gives you the gradient. It is monotonic, smooth, and centered output-wise at 0.5. Its close relative tanh(z) = 2·sigma(2z) - 1 maps to (-1, 1) and is zero-centered, which historically trained slightly better as a hidden activation.

Sigmoid's defining flaw drove the field to ReLU. Look at sigma': it peaks at only 0.25 (at z=0) and decays to nearly 0 whenever the magnitude of z is large; the neuron is saturated, pinned near 0 or 1, and its gradient vanishes. In a deep network, backpropagation multiplies many such small factors together, so the gradient reaching early layers shrinks exponentially toward zero and those layers barely learn. This vanishing-gradient problem made pre-2010 deep networks notoriously hard to train and is the central reason hidden layers switched to ReLU. Sigmoid outputs are also not zero-centered, which biases gradient updates.

Sigmoid is far from obsolete; it just moved jobs. It is the correct output activation for binary classification and for multi-label problems where each class is an independent yes/no (for example an object detector asking whether each of 80 categories is present), each paired with binary cross-entropy. It also appears inside gating mechanisms (LSTM and GRU gates, squeeze-and-excitation blocks in CNNs, attention gates) precisely because its (0,1) range acts as a soft switch. The lesson: use sigmoid where you need a bounded probability-like value, not as a generic hidden nonlinearity.