Unigram tokenizer

The Unigram model takes the opposite path to byte-pair encoding. Rather than building a vocabulary up from characters, it starts with a large pile of candidate pieces and prunes it down. It treats tokenization as probability: each token is given a probability, and the score of any way to cut a word is the product of its tokens' probabilities. The model then keeps the smaller vocabulary that best explains the training text.

Training alternates between estimating each token's probability and dropping the candidates that contribute least, repeating until the target vocabulary size is reached. Because many segmentations of the same word are possible, Unigram can also sample alternative splits during training, a trick called subword regularization that makes the resulting model more robust to messy input. It is the default model inside SentencePiece, and its probabilistic framing is what sets it apart from the merge-based methods.