sigmoid function

The sigmoid is a smooth S-shaped curve that takes any number — huge, tiny, negative, positive — and gently squashes it into the range between 0 and 1. Picture a slide that's nearly flat at the top, steep in the middle, and nearly flat again at the bottom. Feed in a big positive number and it comes out near 1; a big negative number comes out near 0; zero comes out at exactly 0.5. Because the output looks like a probability, sigmoid is a natural way to express "how confident am I that the answer is yes?"

Historically the sigmoid was the workhorse activation of neural networks, and it remains the standard final step for yes/no (binary) classification, where you want a single output you can read as a probability. It's the very same curve that powers logistic regression. The smoothness is genuinely useful: because the curve has a gentle slope everywhere, the learning algorithm can always tell which way to nudge the weights.

But that smoothness hides a trap. At the flat ends — for large positive or large negative inputs — the slope is almost zero. When the slope is near zero, the learning signal that flows backward through the network gets multiplied by almost nothing and nearly vanishes. Stack several sigmoid layers and these tiny slopes multiply together, so the earliest layers barely learn at all. This is the vanishing-gradient problem, and it's the main reason ReLU has replaced sigmoid in the hidden layers of modern deep networks. Sigmoid lives on mostly at the output.