the Transformer

A Transformer is the blueprint behind almost every modern LLM. Picture a factory line where every word in your sentence is handled at the same time rather than one after another, and at each station every word gets to glance at every other word and pull in whatever context it needs. That single trick — letting words look at each other directly — is what makes Transformers so good at language.

Concretely, a Transformer is a stack of identical blocks. Each block has two main parts: an attention layer, where tokens exchange information based on relevance, and a small feed-forward network that processes each token on its own. Information rises up the stack through a running sum called the residual stream, with normalization keeping the numbers well behaved. Repeat this block dozens or hundreds of times and the model builds up rich, layered meaning.

Introduced in the 2017 paper 'Attention Is All You Need', the Transformer replaced recurrent networks because it trains in parallel on modern hardware and handles long-range dependencies cleanly. It is not magic, though: it has a fixed context window, and its cost grows quickly with sequence length. Almost everything else in this domain is built on top of this one architecture.