subword tokenization

A dictionary can never contain every word: people invent new ones, make typos, and glue pieces together. Subword tokenization solves this by keeping common whole words as single tokens but splitting rare or unseen words into smaller, reusable pieces. "Unhappiness" might become "un" plus "happiness", and "tokenization" might become "token" plus "ization". The pieces are chosen so that the parts are themselves common, which means almost any word can be spelled out of fragments the model already knows.

It is a deliberate middle ground between two bad extremes. One token per word gives a huge vocabulary that still chokes on any new word; one token per character gives a tiny vocabulary but stretches text into very long sequences. Subword schemes — byte-pair encoding, WordPiece, and Unigram — learn a fixed set of pieces from a training corpus so frequent words stay whole while everything else degrades gracefully into parts. The practical payoff is that no input word is ever truly unknown.