"The cat sat on the"
input so far
Model predicts
probability over every token
Compare to truth
actual next word was "mat"
Adjust weights
nudge toward predicting "mat"
Repeat trillions of times
across the whole corpus
Pretraining is the process of teaching a transformer to do exactly one thing: given some text, guess the next token. That's the whole task. No labels, no human annotators reading each example — the 'label' for any position is just whatever token actually comes next in the raw text, which means you can turn essentially any text on Earth into training data with zero manual effort. This is what self-supervised learning means in practice.
Where the compute and money actually go
This single loop — predict, compare, adjust — is repeated across trillions of tokens, using thousands of GPUs running in parallel for weeks or months. Pretraining a frontier model costs tens to hundreds of millions of dollars in compute alone; it is, by a wide margin, the most expensive phase in an LLM's life cycle, dwarfing fine-tuning and typical inference costs per model instance.
Pretraining
- Objective: predict the next token, generically
- Data: enormous, mostly unlabeled web text
- Cost: months, thousands of GPUs, 100M+
- Result: a 'base model' with broad raw capability
Fine-tuning / post-training
- Objective: follow instructions, be helpful/safe
- Data: small, curated, often human-labeled
- Cost: days to weeks, a tiny fraction of pretraining
- Result: an assistant people can actually use
Scale as the headline variable
The defining discovery of the last decade of this field is that this one objective, run at increasing scale — more parameters, more data, more compute — doesn't just get incrementally better, it crosses thresholds where qualitatively new abilities appear that smaller versions of the same architecture simply cannot do at all: multi-step arithmetic, following complex multi-part instructions, translating between languages never explicitly paired in training. These are usually called emergent abilities.