byte-pair encoding (BPE)

Byte-pair encoding builds a vocabulary the way you might compress text by hand: find the most common adjacent pair and give it a shortcut. You start with individual characters or bytes, then repeatedly look across the whole corpus for the most frequent neighbouring pair and merge it into a single new token. Run that merge step a few thousand times and you end up with a vocabulary of common fragments and whole words, grown bottom-up purely from how often things appear together.

When you tokenize new text, the same learned merges are applied greedily, in the order they were learned. BPE is the workhorse behind GPT-style models and many others because it is simple, fast, and exactly reversible. Modern versions operate on raw bytes, so they can never fail on an emoji or an unfamiliar script — in the worst case a character is spelled out byte by byte. The number of merges you choose directly sets the final vocabulary size.