MDSection 18 of 38 · 10 subtopics
Build a Small LLM
Build a small LLM from scratch. Dataset, tokenizer, GPT architecture, training, and generation.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
01
Book Dataset
Book reader + dataset panel.
book-dataset
02
Text Extraction
Raw-book-to-text pipeline.
text-extraction
03
Text Cleaning
Cleaning preview.
text-cleaning
04
Tokenizer
Tokenizer playground.
tokenizer
05
Training Sequences
Sliding context window.
training-sequences
06
Embeddings
Token embedding flow.
embeddings
07
Transformer
Small GPT architecture builder.
transformer
08
Training Loop
Live training dashboard.
training-loop
09
Text Generation
Interactive local-model chat.
text-generation
10
Model Inspection
Layer/parameter explorer.
model-inspection