byte-pair encoding
/ BITE-pair en-KOH-ding /
Byte-pair encoding, or BPE, is a clever way to break words into reusable pieces that sits between two bad extremes. Treat every whole word as a token and your vocabulary balloons and still chokes on any word it never saw. Treat every single letter as a token and sequences become absurdly long. BPE finds a middle path: it keeps common words whole but splits rare ones into familiar fragments — so "tokenization" might become "token" + "ization," pieces the model has met many times before.
The build process is delightfully simple. Start with text as individual characters. Then repeatedly find the most frequent adjacent pair and merge it into a new unit, again and again, for a fixed number of rounds. Common letter-pairs like "th" merge early; whole frequent words eventually emerge; rare words stay as a handful of subword chunks. The merge rules learned on the training text then apply to any new text — which means BPE never meets a truly unknown word, because in the worst case it can always fall back to single characters.
This matters because BPE (and close relatives like WordPiece) is the tokenizer under the hood of essentially every modern large language model. It controls vocabulary size, handles new and misspelled words gracefully, and works across many languages. One honest wrinkle worth knowing: the splits are driven by raw frequency, not by meaning, so BPE sometimes carves words at points that ignore real morphology — useful and robust, but not linguistically tidy.
The made-up word "unhappiness" has likely never been seen whole, but BPE splits it into known pieces: "un" + "happi" + "ness." The model recognizes each fragment, so the new word is not a complete mystery.
Rare or new words fall back to familiar subword pieces — so nothing is ever fully unknown.
BPE merges are driven by raw frequency, not meaning, so its splits don't always match real word roots and prefixes. Its great virtue is robustness: it can always fall back to characters, so it never hits a truly unknown word.