Training Dataset
A training dataset is the collection of text (or tokens) used to train the model. For LLM pretraining, it is typically a massive web crawl combined with books, code, and curated sources, ranging from hundreds of billions to trillions of tokens.
What is Training Dataset?
A training dataset is the collection of text (or tokens) used to train the model. For LLM pretraining, it is typically a massive web crawl combined with books, code, and curated sources, ranging from hundreds of billions to trillions of tokens.
A training dataset is the collection of text (or tokens) used to train the model. For LLM pretraining, it is typically a massive web crawl combined with books, code, and curated sources, ranging from hundreds of billions to trillions of tokens.
Where is it used?
GPT-3 was trained on Common Crawl, WebText2, Books1, Books2, and Wikipedia (~500B tokens). LLaMA used CommonCrawl, C4, Github, Books, ArXiv, Wikipedia (~1.4T tokens). The quality and diversity of the dataset strongly determines model capabilities.
How to build it
Load with HuggingFace `datasets.load_dataset('wikitext', 'wikitext-103')` for small experiments. For large-scale: stream from Parquet shards with `datasets.load_dataset(path, streaming=True)`. Pre-tokenize and save as `.bin` files for fast `mmap` loading.