Book Dataset
A book dataset is a small, clean text corpus (e.g., a single public domain book) used to train a miniature LLM for learning purposes. It provides enough text to learn basic language patterns while being small enough to train on a single GPU.
What is Book Dataset?
A book dataset is a small, clean text corpus (e.g., a single public domain book) used to train a miniature LLM for learning purposes. It provides enough text to learn basic language patterns while being small enough to train on a single GPU.
A book dataset is a small, clean text corpus (e.g., a single public domain book) used to train a miniature LLM for learning purposes. It provides enough text to learn basic language patterns while being small enough to train on a single GPU.
Where is it used?
Karpathy's nanoGPT and 'Build GPT from Scratch' tutorial train on Shakespeare or TinyStories datasets. The original GPT-2 paper used BookCorpus. For students, Project Gutenberg books or the TinyStories corpus are ideal small-scale training datasets.
How to build it
Download a public domain book from Project Gutenberg as plain text. Save as `.txt`. For TinyStories: `datasets.load_dataset('roneneldan/TinyStories')`. For Shakespeare: download `input.txt` from nanoGPT's repo. Keep the raw text for the next preprocessing step.