Tokenization · L3 Code

Build a working byte-level BPE trainer and encoder from scratch, then compare it to a real tokenizer.

Corpus
raw text as bytes
get_stats
count adjacent pairs
merge
replace most frequent pair
loop
until vocab_size reached

The full BPE trainer is under 40 lines of Python.

Below is a minimal but complete byte-level BPE trainer and encoder, close to what powers GPT-2's tokenizer (minus the regex pre-tokenizer, omitted for clarity). It operates directly on byte sequences, so it never needs an <UNK> token.