Pre-tokenize
split on whitespace/punctuation boundaries
Init vocab
every byte is a starting token
Count pairs
frequency of every adjacent token pair
Merge top pair
add merged token to vocab
Repeat
until target vocab size reached
Training a BPE tokenizer is entirely separate from training the language model, uses no gradients, and only needs to happen once. It runs over a large, representative text corpus and produces two artifacts: a vocabulary (a lookup table from token strings to integer IDs) and a merge list (the ordered sequence of merge rules learned, which is needed to tokenize new text the same way).