Pretraining vs Fine-Tuning
Pretraining trains a model from scratch on massive unlabelled text to learn general language patterns, while fine-tuning adapts that pretrained model on a small labelled dataset for a specific task or behaviour.
What is Pretraining vs Fine-Tuning?
Pretraining trains a model from scratch on massive unlabelled text to learn general language patterns, while fine-tuning adapts that pretrained model on a small labelled dataset for a specific task or behaviour.
Pretraining trains a model from scratch on massive unlabelled text to learn general language patterns, while fine-tuning adapts that pretrained model on a small labelled dataset for a specific task or behaviour.
Where is it used?
Llama-3-8B-Base is pretrained on 15T tokens; Llama-3-8B-Instruct is fine-tuned on curated instruction data. GPT-4 follows the same base-then-align two-stage pipeline.
How to build it
Compare loss curves: pretrain by training `GPT2LMHeadModel` from random weights on a large corpus, then fine-tune by loading pretrained weights via `from_pretrained` and training on a small JSONL dataset.