SentencePiece

Most tokenizers quietly assume you have already split text into words using spaces — an assumption that breaks down for Chinese, Japanese, Thai, and source code, where word boundaries are not marked by spaces. SentencePiece is a tokenizer toolkit that drops that assumption entirely. It trains directly on raw text, treating the space itself as just another character, often displayed as a special underscore symbol so it is never lost.

Because it is language-agnostic and fully reversible, you can always reconstruct the exact original string, spaces and all, from the tokens alone. SentencePiece is really packaging rather than a single algorithm: under the hood it can train either a byte-pair encoding or a Unigram model. Many multilingual and open models, including the LLaMA and T5 families, use it precisely because it treats every language the same way and needs no language-specific pre-splitting step.