Tokenizers
Up to 8. Colours stay with a tokenizer while it is selected.Sentence
Build a word, merge by merge
BPE tokenizers only. # is the merge's rank; the highlighted piece is the one just merged. In the Playground, click any token for its merge tree.Merge timelines across methods
Who picks the merges
Tokens each language can use
New tokens whose pair the language had, whoever picked them. Comparable across every method.
What each rule balances
How widely new tokens are shared
Share of each method's new tokens used by exactly k languages.