Data Cleaning
Data cleaning removes noise from the training corpus: HTML tags, boilerplate, encoding errors, duplicate content, and low-quality text. Clean data directly improves model quality; a smaller clean corpus often outperforms a larger dirty one.
What is Data Cleaning?
Data cleaning removes noise from the training corpus: HTML tags, boilerplate, encoding errors, duplicate content, and low-quality text. Clean data directly improves model quality; a smaller clean corpus often outperforms a larger dirty one.
Data cleaning removes noise from the training corpus: HTML tags, boilerplate, encoding errors, duplicate content, and low-quality text. Clean data directly improves model quality; a smaller clean corpus often outperforms a larger dirty one.
Where is it used?
LLaMA, GPT-3, and RefinedWeb (Falcon's dataset) all apply aggressive cleaning. RefinedWeb uses deduplication, language filtering, and quality classifiers. Common Crawl raw text is notoriously noisy; cleaning is the most impactful preprocessing step.
How to build it
Use `ftfy` to fix encoding, `bs4` to strip HTML, regex to remove boilerplate. Pipeline: `text = ftfy.fix_text(text); text = BeautifulSoup(text).get_text(); text = re.sub(r'{.*?}', '', text)`. For quality, train a classifier on high-quality vs random web text.