JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

From IDs to Vectors: The Embedding Layer

A token ID like 5169 is just a name tag — the number itself means nothing. The embedding layer trades each ID for a learned list of numbers that the model can actually reason with. Meet the model's first weights.

Why the ID number is useless on its own

We left guide 1 with a list of token IDs like [1820, 5169, 7493]. It is tempting to feed those integers straight into the math. But the value of the number is meaningless: token 5169 (` cat`) is not "bigger than" or "three times" token 1820 (`the`). The IDs are just positions in a list, like seat numbers in a theatre. Doing arithmetic on seat numbers is nonsense.

The classic fix is one-hot encoding: represent token 5169 as a vector that is all zeros except a single 1 in position 5169. That removes the false ordering, but for a 100,000-word vocabulary it is a 100,000-long vector that is 99,999 zeros — wasteful, and it still says nothing about meaning: every token is equally far from every other. We need something denser and smarter.

The embedding matrix: a giant lookup table

The solution is the embedding matrix: a big table of learned numbers with one row per token in the vocabulary. To turn token 5169 into something useful, you simply pull out row 5169. That row is a list of, say, 4096 real numbers — the token's embedding, a dense vector that stands in for the word inside the model. No multiplication, no fuss; embedding is literally a table lookup by row.

embedding_matrix:  shape [vocab_size, d_model]  =  [100000, 4096]

lookup(5169) -> embedding_matrix[5169]
            ->  [0.12, -0.85, 0.33, ..., 0.07]   # 4096 numbers, the " cat" vector
Each token ID indexes one row. The whole table is learned during training.
\mathbf{e}_t = \mathrm{onehot}(t)^{\top} E = E_{t,\,:}

Multiplying a one-hot vector by the embedding matrix E just selects row t — the lookup is the same operation, done efficiently.

Embedding dimension: how wide is a word?

How many numbers are in each row? That count is the embedding dimension, often written `d_model` — values like 768, 4096, or 8192 are common. Think of it as the number of coordinates describing each token, or the width of the pipe that the rest of the model runs on. A larger dimension gives the model more room to encode subtle distinctions (tense, tone, topic, formality), but costs more memory and compute. Crucially, the individual coordinates are not human-labelled: there is no single "plural-ness" slot. Meaning is smeared across all the numbers together — a point guide 4 makes vivid.

Where do the numbers come from?

Here is the beautiful part: nobody fills in the embedding table by hand. It starts as random numbers and is learned. Every embedding value is a parameter adjusted by training. As the model is trained on next-token prediction, each time it makes a mistake the error is traced back and the embeddings are nudged so that tokens which behave alike drift toward similar vectors. Over billions of examples, ` cat` and ` dog` end up close together because they show up in similar contexts. The table is shaped entirely by use.

Where the numbers come from: each embedding value starts random and is learned by gradient descent — the same training that tunes every other weight.

Interactive gradient descent rolling down a loss surface toward the minimum.

A bonus: tying the two ends together

At the output end the model faces a mirror-image task: turn its final internal vector back into a score for every token in the vocabulary, so it can pick the next word. That needs a second big table — and it has exactly the same shape as the input embedding matrix. Many models therefore use tied input/output embeddings: reuse the same table for both reading tokens in and scoring tokens out. This saves a large chunk of parameters and often helps quality, since a token's "input meaning" and "output identity" stay consistent. We will revisit this in guide 5.