Weight Updates
Weight updates adjust the model's parameters using the gradients from backpropagation. The optimizer (e.g., AdamW) computes the update direction and magnitude, applying it to each parameter to reduce the loss on future examples.
What is Weight Updates?
Weight updates adjust the model's parameters using the gradients from backpropagation. The optimizer (e.g., AdamW) computes the update direction and magnitude, applying it to each parameter to reduce the loss on future examples.
Weight updates adjust the model's parameters using the gradients from backpropagation. The optimizer (e.g., AdamW) computes the update direction and magnitude, applying it to each parameter to reduce the loss on future examples.
Where is it used?
AdamW is the standard optimizer for training GPT-3, LLaMA-2, and Mistral. Updates are applied after gradient accumulation steps. During fine-tuning with LoRA, only the low-rank adapter weights are updated while base weights stay frozen.
How to build it
Initialize: `optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)`. After `loss.backward()`: `optimizer.step(); optimizer.zero_grad()`. Clip gradients first: `torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)`.