Training Loop
The training loop iterates over batches of sequences, performing forward pass, loss computation, backpropagation, and weight updates. It runs for many epochs until the model's validation loss stops improving or a target quality is reached.
What is Training Loop?
The training loop iterates over batches of sequences, performing forward pass, loss computation, backpropagation, and weight updates. It runs for many epochs until the model's validation loss stops improving or a target quality is reached.
The training loop iterates over batches of sequences, performing forward pass, loss computation, backpropagation, and weight updates. It runs for many epochs until the model's validation loss stops improving or a target quality is reached.
Where is it used?
Every LLM pretraining uses this loop. nanoGPT trains Shakespeare for ~5000 iterations on a single GPU in minutes. LLaMA-2 trains for ~2T tokens over weeks on thousands of GPUs. The loop is the same; the scale differs.
How to build it
`for step in range(max_steps): x, y = next(loader); logits = model(x); loss = F.cross_entropy(logits.view(-1, V), y.view(-1)); optimizer.zero_grad(); loss.backward(); clip_grad_norm_(model.parameters(), 1.0); optimizer.step(); scheduler.step()`. Log loss every 100 steps.