Pretraining
Pretraining is the first, largest-scale training phase where a model learns general language patterns from massive unlabeled text. The objective is typically next-token prediction, producing a base model that can then be fine-tuned for specific tasks.
What is Pretraining?
Pretraining is the first, largest-scale training phase where a model learns general language patterns from massive unlabeled text. The objective is typically next-token prediction, producing a base model that can then be fine-tuned for specific tasks.
Pretraining is the first, largest-scale training phase where a model learns general language patterns from massive unlabeled text. The objective is typically next-token prediction, producing a base model that can then be fine-tuned for specific tasks.
Where is it used?
GPT-3, LLaMA-2, BERT, and T5 all underwent pretraining on trillions of tokens. LLaMA-2 was pretrained on 2T tokens. The pretrained 'base' model is what gets released (e.g., `Llama-2-7b-hf`) before instruction tuning produces the 'chat' variant.
How to build it
Collect a large text corpus, tokenize it, pack into fixed-length sequences, and train with cross-entropy loss on next-token prediction. Use `Trainer` from HuggingFace or a custom loop with AdamW, cosine LR schedule, and gradient accumulation.