WordPiece

WordPiece, the scheme behind BERT and its relatives, is a close cousin of byte-pair encoding. It also grows a vocabulary by merging pieces, but it changes the rule for which pair to merge. Instead of always joining the most frequent neighbouring pair, it joins the pair that most increases the likelihood of the training text under a simple word model. Loosely, it asks "which merge best explains the data?" rather than simply "which pair is most common?".

In practice the two methods produce similar but not identical splits. Word-pieces that continue a word are usually marked, for example with a "##" prefix as in "play" plus "##ing", so the original spacing can be restored exactly. Like byte-pair encoding, WordPiece guarantees coverage by falling back to single characters for anything it cannot piece together. The likelihood-driven criterion is its defining difference, and it tends to favour merges that genuinely make the corpus more predictable.