Dataset Design
Dataset design is the curation, cleaning, deduplication, and formatting of training or evaluation data to maximise model quality and minimise bias, toxicity, and contamination.
What is Dataset Design?
Dataset design is the curation, cleaning, deduplication, and formatting of training or evaluation data to maximise model quality and minimise bias, toxicity, and contamination.
Dataset design is the curation, cleaning, deduplication, and formatting of training or evaluation data to maximise model quality and minimise bias, toxicity, and contamination.
Where is it used?
Llama-3's 15T-token pretraining mix and OpenAI's RLHF preference datasets are carefully designed; The Pile and RedPajama are open pretraining datasets; `datasets` and `dolma` tooling help.
How to build it
Use `datasets.load_dataset`, deduplicate with `datasketch.MinHash`, filter toxicity with `transformers.pipeline("text-classification", model="...toxic-bert")`, and log stats with `datasets`.