vocabulary size

A model's vocabulary is the complete list of tokens it knows — its entire alphabet of pieces. Vocabulary size is simply how long that list is, typically anywhere from about 30,000 to a few hundred thousand entries in modern LLMs. Every piece of text the model ever handles must be expressed using only tokens from this fixed list, so the list's size shapes how text gets chopped up.

The choice is a real trade-off. A larger vocabulary lets each token carry more text, so sequences are shorter and faster to process, but it makes the embedding and output layers bigger, costs more memory, and creates many rare tokens that are seen too seldom to be learned well. A smaller vocabulary keeps those layers lean but slices text into more, smaller pieces. Multilingual models tend toward larger vocabularies so that every language receives fair coverage.