LLM Training
The training pipeline: data cleaning, batching, forward pass, loss, backprop, weight updates, checkpoints.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
Pretraining
Full training pipeline.
Training Dataset
Dataset explorer.
Data Cleaning
Before/after document viewer.
Data Filtering
Filter pipeline.
Data Deduplication
Duplicate-document graph.
Training Samples
Sliding-window tokenizer.
Batch Construction
Sample-to-batch animation.
Forward Pass
Model execution animation.
Loss Calculation
Prediction vs target panel.
Backpropagation
Reverse gradient animation.
Weight Updates
Before/after weight matrix.
Learning Rate
Slider + training graph.
Learning Rate Scheduling
Learning-rate curve.
Checkpoints
Training timeline.
Validation
Training vs validation loss graph.