Checkpoints
A checkpoint is a saved snapshot of the model's weights, optimizer state, and training progress at a specific step. Checkpoints enable resuming interrupted training, evaluating intermediate models, and deploying trained models for inference.
What is Checkpoints?
A checkpoint is a saved snapshot of the model's weights, optimizer state, and training progress at a specific step. Checkpoints enable resuming interrupted training, evaluating intermediate models, and deploying trained models for inference.
A checkpoint is a saved snapshot of the model's weights, optimizer state, and training progress at a specific step. Checkpoints enable resuming interrupted training, evaluating intermediate models, and deploying trained models for inference.
Where is it used?
LLaMA-2, GPT-3, and Mistral save checkpoints every few thousand steps during pretraining. The final checkpoint becomes the released model. HuggingFace Hub stores checkpoints as `.safetensors` files. Fault-tolerant training requires frequent checkpointing.
How to build it
Save: `torch.save({'model': model.state_dict(), 'optimizer': optimizer.state_dict(), 'step': step}, 'ckpt.pt')`. Load: `ckpt = torch.load('ckpt.pt'); model.load_state_dict(ckpt['model'])`. For large models, use `safetensors` or FSDP sharded checkpoints.