embedding matrix

Where do all those token vectors live? In one big table called the embedding matrix — a grid with one row per token in the vocabulary and one column per dimension of the vector. If the vocabulary holds 50,000 tokens and each vector is 4,096 numbers long, the matrix is 50,000 by 4,096, which is more than 200 million numbers sitting in a single table.

Looking up a token's embedding is just selecting its row by id — no multiplication, only indexing. The whole matrix is a learned parameter: every number in it is adjusted during training, so the table starts random and slowly fills with meaning. Because it carries a row for every token, it is often one of the largest single components of a model, which is part of why vocabulary size matters so much for memory. Many models reuse this very same matrix at the output, a trick called weight tying.