tokens

An LLM does not read letters or whole sentences — it reads tokens, bite-sized chunks of text. A token can be a common whole word like "the", a word-piece like "ing", a single character, a punctuation mark, or even a space. Before the model ever sees your prompt, a tokenizer chops the text into a sequence of these chunks. Think of tokens as the model's real alphabet, except the letters are common fragments learned from huge amounts of text rather than the 26 letters you know.

Each token corresponds to an integer id, and the model only ever works with those numbers. A rough English rule of thumb is one token is about four characters, or roughly three-quarters of a word, so a thousand tokens is around 750 words. Counting tokens matters because pricing, context limits, and speed are all measured in tokens, not words. Other scripts such as Chinese and Japanese, and source code, often tokenize quite differently — sometimes far less efficiently.