Training Experiments
Training experiments vary training-time factors (data mixture, learning rate schedule, batch size, optimiser) to study their effect on final model quality and training stability.
What is Training Experiments?
Training experiments vary training-time factors (data mixture, learning rate schedule, batch size, optimiser) to study their effect on final model quality and training stability.
Training experiments vary training-time factors (data mixture, learning rate schedule, batch size, optimiser) to study their effect on final model quality and training stability.
Where is it used?
Llama scaling-law experiments, the Chinchilla compute-optimal study, and learning-rate sweeps in every production training run are training experiments; `wandb` sweeps automate them.
How to build it
Use `wandb.sweep` with a `method: bayes` config over `learning_rate: {min: 1e-5, max: 1e-3}` and `batch_size: [32, 64, 128]`, launch agents, and pick the config with best validation loss.