Large Language Models
Data Sourcing & Curation
Where the tokens come from, and why filtering is the real work.
You can jump straight in, but this star assumes Pretraining. Not recommended as a first stop.
Four layers of depth
Each layer ends with a quiz. Finish layer 4 and you own this concept.
- L1IntuitionEvery capability an LLM has traces back to some document it read. Where those documents come from is most of the real engineering work.6m
40 XP - L2MechanicsThe pipeline mechanically: extraction, heuristic and classifier-based filtering, MinHash/LSH dedup, decontamination, and domain mixing.11m
70 XP - L3CodeA real MinHash+LSH near-duplicate pipeline and a quality-classifier filter, built end to end.15m
110 XP - L4FoundationsThe math of Jaccard estimation and MinHash collision probability, the LSH threshold curve, and hardware reality of running this at trillions of tokens.14m
180 XP
33 stars in the atlas.