Text Cleaning
Text cleaning normalizes the extracted text: fixing encoding issues, removing non-printable characters, normalizing whitespace, and optionally lowercasing. This ensures the tokenizer sees consistent input and the model learns clean patterns.
What is Text Cleaning?
Text cleaning normalizes the extracted text: fixing encoding issues, removing non-printable characters, normalizing whitespace, and optionally lowercasing. This ensures the tokenizer sees consistent input and the model learns clean patterns.
Text cleaning normalizes the extracted text: fixing encoding issues, removing non-printable characters, normalizing whitespace, and optionally lowercasing. This ensures the tokenizer sees consistent input and the model learns clean patterns.
Where is it used?
Every LLM dataset pipeline (LLaMA, GPT-3, RefinedWeb) includes a text cleaning step. For small-scale projects, cleaning is simpler but still important: Project Gutenberg texts contain ligatures, smart quotes, and inconsistent line breaks that need normalization.
How to build it
Pipeline: `text = ftfy.fix_text(text); text = unicodedata.normalize('NFKC', text); text = re.sub(r'\s+', ' ', text); text = text.lower()`. Remove non-printable: `text = ''.join(c for c in text if c.isprintable() or c == '\n')`. Save the cleaned `.txt`.