activation functions in LLMs

An activation function is the bit of nonlinearity that lets a network do more than stack flat linear maps, which would collapse into one flat map no matter how deep. Inside an LLM it lives in the feed-forward block, bending each hidden value through a curved gate that decides how strongly that unit speaks up. Without it, the whole model would be a single matrix multiply and could not learn the rich, layered patterns of language.

Early transformers used ReLU, a hard hinge that zeroes out negatives. Modern LLMs prefer smooth curves: GELU, which softly weights an input by how likely it is to be positive, and SiLU, also called Swish, the input times its own sigmoid. Smoothness gives cleaner gradients and a small but real quality edge. The current default is to fold SiLU into a gate, the SwiGLU pattern, where one projection modulates another rather than just bending a single value.