JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why Models See Numbers, Not Words

A neural network can only multiply and add numbers. So the very first job of any language model is to turn your sentence into a list of integers. Here is why, and what those integers are.

Computers do arithmetic, not reading

When you type *"the cat sat"* into a chatbot, it feels like the model reads English. It does not. Underneath, a large language model is a giant pile of numbers — billions of parameters — and the only thing it can do is multiply, add, and compare numbers. It has no notion of letters, spaces, or the word cat. So before any thinking can happen, the text has to be converted into numbers. This conversion step is called tokenization, and it is the unglamorous foundation everything else stands on.

A vocabulary: the model's fixed list of pieces

The trick is to agree, in advance, on a fixed list of text pieces the model is allowed to know about. That list is the vocabulary, and each piece in it is a token. A token is usually not a whole word and not a single letter — it sits in between, often a common chunk like `cat`, ` sat`, or `ing`. A modern model's vocabulary size is typically 30,000 to 200,000 tokens. Every token in the vocabulary is given a number: its position in the list. The model knows token #1842, not the word it spells.

Because the list is fixed, turning text into numbers becomes a lookup. Find each chunk in the vocabulary, write down its number, and you have a sequence of token IDs — a list of plain integers. That integer list is the real input to the model; the original string is left at the door.

Tokenization turns the sentence into a sequence of token IDs, and a lookup table maps each ID to an embedding vector — the two front doors of the model.

Diagram: a sentence splits into tokens, each token becomes an integer ID, and each ID indexes a row of an embedding matrix shown as a numeric vector.

Tokenize "the cat sat" by hand

  1. Take the raw string: "the cat sat".
  2. Split it into vocabulary pieces. A real tokenizer keeps the spaces, so it might give: ["the", " cat", " sat"] — notice the leading spaces belong to the tokens.
  3. Look up each piece's number in the vocabulary: "the" → 1820, " cat" → 5169, " sat" → 7493.
  4. Hand the model the integer list [1820, 5169, 7493]. The words are gone; only numbers remain.
text   :  "the cat sat"
tokens :  ["the", " cat", " sat"]
ids    :  [1820, 5169, 7493]      <- this is what the model receives
Three views of the same input. The model only ever sees the bottom row.

Numbers in, one number out, repeat

Once the model has the token IDs, it does its math and produces a guess for the next token — this is next-token prediction, the single task every chat model is built around. It outputs a number (a token ID), that ID is looked up back into text, appended to the sentence, and the whole thing runs again to predict the token after that. A long answer is just this loop running hundreds of times. So tokens are not only the input — they are also the unit the model thinks and bills in.

\hat{p}(x_{t+1}\mid x_1,\dots,x_t)=\operatorname{softmax}(W h_t)

Next-token prediction: from the running context of token IDs, the model outputs a probability for every token in the vocabulary.

Numbers in, one number out, repeat: each predicted token is appended to the sequence and fed back in to predict the next.

Loop diagram: token IDs enter the model, it emits the next token ID, the new ID is appended to the sequence, and the loop repeats.

Where this track is going

You now have the map. Guide 2 explains how text is split into vocabulary pieces — the clever subword schemes that let a finite list cover any sentence. Guide 3 shows how each token ID becomes a vector of numbers the model can actually compute with. Guide 4 reveals why those vectors capture meaning. Guide 5 turns it all into practical instinct: token counting, cost, and the famous tokenization gotchas. Numbers all the way down — let's go meet them.