Text Extraction
Text extraction converts raw source files (HTML, PDF, EPUB) into clean plain text suitable for tokenization. It strips formatting, metadata, navigation elements, and encoding artifacts, leaving only the natural language content.
What is Text Extraction?
Text extraction converts raw source files (HTML, PDF, EPUB) into clean plain text suitable for tokenization. It strips formatting, metadata, navigation elements, and encoding artifacts, leaving only the natural language content.
Text extraction converts raw source files (HTML, PDF, EPUB) into clean plain text suitable for tokenization. It strips formatting, metadata, navigation elements, and encoding artifacts, leaving only the natural language content.
Where is it used?
When building datasets from Project Gutenberg (HTML/EPUB), arXiv (PDF), or web crawls, text extraction is the first step. LLaMA's CommonCrawl pipeline uses `resiliparse` for HTML extraction. For books, removing Gutenberg licenses and headers is standard.
How to build it
For HTML: `BeautifulSoup(html).get_text()`. For PDF: `pypdf.PdfReader(file).extract_text()`. For Gutenberg: find start/end markers (`*** START OF ... ***` and `*** END OF ... ***`) and slice between them. Save the extracted text as UTF-8 `.txt`.