JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why Attention? From Reading Left-to-Right to Looking Around

The problem the transformer was invented to solve, and the one idea — letting every word look at every other word — that solves it.

The sentence that needs the whole sentence

Read this: *"The trophy didn't fit in the suitcase because it was too big."* What does it mean? You only know because you weighed the other words — trophy makes sense, suitcase doesn't. Now flip the last word to small, and it suddenly means the suitcase. A single word's meaning depends on the company it keeps. Any model that hopes to predict the next word has to do this kind of weighing all the time. That weighing is what we call attention.

A transformer is the neural network architecture built entirely around this idea. It is the engine inside essentially every modern large language model. By the end of this track you will be able to trace a sentence as it flows through one, layer by layer, and say exactly what each part does and why.

What came before, and why it struggled

Before transformers, the dominant tool for language was the recurrent neural network (RNN). It read a sentence the way you might whisper it down a phone line: one word at a time, squeezing everything it had seen so far into a single fixed-size memory before moving on. This had two stubborn problems.

A recurrent network unrolled in time: it reads one word after another, so distant words must survive a long chain of steps.

Diagram of an RNN unrolled into a left-to-right chain of repeated cells, one per time step.

  1. Forgetting. By the time the RNN reached it at the end, the early words trophy and suitcase had been overwritten many times. Long-range links faded.
  2. No parallelism. Word 50 could not be processed until word 49 was done. Training on billions of words this way is painfully slow, so models stayed small.

Earlier sequence-to-sequence models bolted an early attention mechanism onto an RNN as a patch for forgetting. The 2017 paper Attention Is All You Need made the radical move: throw the recurrence away and keep only the attention.

Attention in one picture

Imagine every word in the sentence sitting at its own desk, holding a vector — a list of numbers that encodes what the word means so far. Self-attention lets each desk send a quick query to every other desk, ask *"how relevant are you to me?"*, and then pull in a blend of the answers. it queries the room, finds trophy highly relevant, and updates its own vector to absorb some of trophy's meaning.

Click any word to see which others it attends to — self-attention lets every position look at every other in one parallel step.

Interactive self-attention demo: selecting a word highlights its attention weights over the rest of the sentence.

This blending is what we call context mixing: information flows sideways between positions so that each word's representation becomes context-aware. Crucially, every desk does this at the same time. There is no left-to-right bottleneck, which is exactly why the architecture is so fast to train and why it can scale to enormous size.

\text{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V

The whole idea in one line: each word's new vector is a softmax-weighted blend of the others — the formula the next guide unpacks.

What you'll build up

Attention is the heart, but a transformer is more than attention. Over the next four guides we will assemble the full engine, one part at a time, until you can read any architecture diagram without flinching.

  1. Guide 2 — queries, keys, and values: the actual arithmetic of how one word decides who to listen to.
  2. Guide 3 — many heads and a one-way street: parallel attention specialists, and the mask that stops the model from cheating.
  3. Guide 4 — the rest of the block: the feedforward layer, residual connections, normalization, and how position is encoded.
  4. Guide 5 — stacking the engine: why we repeat the block dozens of times, and what depth buys us.