vocabulary merges

A subword vocabulary is not designed by hand — it is grown from data using merge rules. A merge rule is a single instruction such as "whenever you see the piece 't' followed by 'h', join them into 'th'". During training the tokenizer learns an ordered list of thousands of these rules, each one combining two existing pieces into a slightly longer one, gradually assembling common fragments and whole words.

Training scans a large corpus, repeatedly finds the best pair to merge — most frequent for byte-pair encoding, most likelihood-boosting for WordPiece — records that rule, and applies it before searching for the next. The finished artefact is two linked things: the vocabulary, which is the list of tokens, and the merges, which say how to build them. At inference the merges are replayed in the same order, which is why tokenization is deterministic. Change the merges and the very same text tokenizes differently.