A neural network is a pile of matrix multiplications. It has no concept of the letter 'c' or the word 'cat' — it only knows how to consume vectors of numbers. Tokenization is the translation layer that turns raw text into a sequence of integers (and, at the other end, integers back into text). Every single thing a language model does — its context window, its cost, its ability to spell, its confusion about arithmetic — is downstream of this one preprocessing step.
Why not just use characters, or just use words?
Character-level
- Tiny vocabulary (~256 bytes)
- Never fails on unseen text
- Sequences become extremely long
- Model must relearn spelling of every word
Word-level
- Short sequences, one token per word
- Vocabulary explodes into millions
- Any typo or new word is 'unknown'
- Wastes IDs on rare inflections
Subword tokenization — the modern default — sits between these extremes. Common words ("the", "and") get their own single token. Rare or unseen words get chopped into familiar fragments: "unbelievable" might become ["un", "believ", "able"]. This gives you a fixed, manageable vocabulary (tens of thousands of entries) while still being able to represent *any* string, because in the worst case a word falls all the way back to individual bytes.
Byte Pair Encoding (BPE), in one sentence
BPE builds its vocabulary bottom-up: start with every individual byte or character as a token, then repeatedly find the *most frequent adjacent pair* of tokens in a big training corpus and merge it into one new token. Do this tens of thousands of times and you end up with a vocabulary of frequently-occurring chunks, learned automatically from data rather than designed by hand.
Start: bytes
every unique byte is a token
Count pairs
most common adjacent pair across corpus
Merge
e.g. 't'+'h' → 'th'
Repeat
tens of thousands of times
Vocab
fixed set of subword tokens
Token economics: the unit you pay for
Every API you call bills per token, and every context window is measured in tokens, not words or characters. A rough rule of thumb for English is about 4 characters per token, or roughly 0.75 words per token. That ratio is not universal — code, math, and non-English text tokenize very differently, sometimes far worse.
Where tokenizers quietly break things
- Arithmetic — "1234" might tokenize as ["123","4"] in one context and ["12","34"] in another, so the model never sees digits in a consistent positional grid, which is a real contributor to LLMs being bad at multi-digit math.
- Code — leading whitespace, indentation, and camelCase are tokenized inconsistently, which is part of why code models train custom tokenizers.
- Non-English languages — scripts underrepresented in the training corpus fragment into many more tokens per word, making those languages more expensive and effectively shrinking the usable context window for those users.
- Word games — ask a model to count the letters in a word and it may fail, because it never sees individual letters — it sees whatever subword chunk the tokenizer produced, and the letters inside are invisible to it.