token embedding

A token id like 9246 is just a label — on its own it tells you nothing about meaning, and the numbers 9245 and 9246 are no more related than any two house numbers. A token embedding turns that bare id into a vector: a list of numbers, perhaps a few thousand of them, that the model can both compute with and, crucially, learn. During training these numbers are nudged so that tokens used in similar ways drift toward similar vectors.

The embedding is the model's very first layer. It looks up the id's row in a large table and hands the resulting vector to the rest of the network. Unlike older fixed word vectors, these embeddings are trained jointly with everything else, so they end up encoding whatever the task needs. This is the bridge that carries you from discrete symbols, which are just ids, into continuous geometry, the world of vectors where neural networks actually do their work.