Data Sourcing & Curation

Layer 2 · Mechanics

Data Sourcing & Curation

The pipeline mechanically: extraction, heuristic and classifier-based filtering, MinHash/LSH dedup, decontamination, and domain mixing.

11 min read70 XP

Extraction: HTML is not text

A raw crawled page is HTML: navigation bars, ad slots, cookie banners, comment sections, and — somewhere in there — the actual content. Boilerplate extraction tools (trafilatura, jusText, and similar) try to isolate the main readable content, and the extractor chosen measurably changes downstream corpus quality: an aggressive extractor strips useful context along with the boilerplate, while a lax one leaves navigation menus and cookie notices as 'training data'.

Go deeper: L3 Code