"Trained on the internet" is the popular shorthand, and it's misleadingly casual — the raw internet is mostly spam, boilerplate, duplicated content, and low-quality text, and feeding it to a model unfiltered produces a measurably worse model than feeding it a smaller, carefully cleaned subset. Data curation is not a cleanup step tacked onto pretraining; it is one of the two or three highest-leverage levers a lab has, alongside model architecture and compute budget.
Where the raw text comes from
- CommonCrawl — a public, continuously updated snapshot of the web, petabytes of raw HTML. The starting point for nearly every large pretraining corpus, and also the source of nearly all its garbage.
- Code repositories — GitHub and similar sources. Including code in the mixture measurably improves a model's structured reasoning even on tasks that have nothing to do with programming.
- Books and long-form text — coherent, well-edited, and long — a different distribution from short web pages, valuable for teaching sustained argument and narrative structure.
- Reference and academic sources — Wikipedia, papers, textbooks. Dense in verified facts relative to their length, though a tiny slice of total tokens.
- Synthetic data — text generated by another model, deliberately, to fill a gap the natural web doesn't cover well (e.g. clean step-by-step math reasoning).
The core problem: quantity and quality trade off, constantly
CommonCrawl alone contains many trillions of tokens — more raw text than any model needs. The bottleneck was never *volume*; it has always been *how much of that volume is worth learning from*. A page that's mostly navigation menus, a spam farm auto-generating thousands of near-identical pages, a forum thread that's 90% quoted replies — none of it is 'wrong' exactly, but training on it in proportion to its raw frequency wastes compute on tokens that teach the model very little, or actively teach it bad habits (mimicking SEO spam patterns, for instance).
Raw crawl
petabytes of HTML, mostly noise
Extract & clean
strip boilerplate, keep readable text
Filter for quality
heuristics + trained classifiers
Deduplicate
remove exact and near-duplicate text
Decontaminate
remove benchmark test data
Mix by domain
weight web/code/books deliberately
Licensing and provenance, increasingly load-bearing
Beyond quality, labs increasingly have to answer *where did this text legally come from* — copyright litigation, licensing agreements with publishers, and opt-out mechanisms (like robots.txt-style crawler exclusions) all shape what a corpus can legally contain. This used to be an afterthought; for any lab shipping a commercial product today, it is a first-class constraint on the data pipeline, not a footnote.