Data Filtering
Data filtering selects high-quality, relevant text and removes harmful, off-topic, or low-information content. This includes language identification, toxicity filtering, PII removal, and quality scoring using classifier models.
What is Data Filtering?
Data filtering selects high-quality, relevant text and removes harmful, off-topic, or low-information content. This includes language identification, toxicity filtering, PII removal, and quality scoring using classifier models.
Data filtering selects high-quality, relevant text and removes harmful, off-topic, or low-information content. This includes language identification, toxicity filtering, PII removal, and quality scoring using classifier models.
Where is it used?
LLaMA-2 filters data using safety classifiers and quality models. OpenAI uses similar filtering for GPT-4 training data. The Pile and RedPajama apply filtering based on perplexity scores from reference models. Filtering complements cleaning.
How to build it
Language filter: `langdetect.detect(text) == 'en'`. Quality: compute perplexity with a small LM and keep low-perplexity text. Toxicity: run a classifier (`unitary/toxic-bert`). PII: regex for emails/phones. Keep text passing all filters.