Mixed Precision
Mixed precision combines FP32 and FP16/BF16 during training: the forward and backward passes use lower precision for speed and memory savings, while a master copy of weights and optimizer state are kept in FP32 for numerical stability.
What is Mixed Precision?
Mixed precision combines FP32 and FP16/BF16 during training: the forward and backward passes use lower precision for speed and memory savings, while a master copy of weights and optimizer state are kept in FP32 for numerical stability.
Mixed precision combines FP32 and FP16/BF16 during training: the forward and backward passes use lower precision for speed and memory savings, while a master copy of weights and optimizer state are kept in FP32 for numerical stability.
Where is it used?
All modern LLM training (GPT-3, LLaMA, Mistral) uses mixed precision. NVIDIA's automatic mixed precision (AMP) with FP16 was standard pre-Ampere; BF16 AMP is now preferred. DeepSpeed ZeRO and FSDP integrate mixed precision for distributed training.
How to build it
Maintain FP32 master weights: `model_fp32 = Model().float(); model_amp = model_fp32.to(torch.bfloat16)`. Or use autocast: `with torch.autocast('cuda', dtype=torch.bfloat16): loss = model_fp32(x); loss.backward(); optimizer.step()` (PyTorch handles casting internally).