embedding dimension

Embedding dimension is simply how long each token's vector is — how many numbers describe a single token. Common sizes run from a few hundred up to many thousands, with values like 768 or 4,096 being typical. You can picture it as how many independent dials the model has available to describe each piece of text: more dials, more ways to express subtle differences.

More dimensions give the model more room to represent fine distinctions, but they cost more memory and computation everywhere downstream, since this width usually flows through the entire network. With too few dimensions, different meanings get crammed together and confused; with too many, you waste capacity and risk fitting noise. The dimension is a fixed design choice, a hyperparameter, identical for every token and decided before training begins.