Frozen Weights
Frozen weights are the pretrained model parameters that are set to `requires_grad=False` during LoRA training so they receive no gradient updates and consume no optimizer memory.
What is Frozen Weights?
Frozen weights are the pretrained model parameters that are set to `requires_grad=False` during LoRA training so they receive no gradient updates and consume no optimizer memory.
Frozen weights are the pretrained model parameters that are set to `requires_grad=False` during LoRA training so they receive no gradient updates and consume no optimizer memory.
Where is it used?
In every LoRA/QLoRA run via `peft`, the base transformer weights are frozen while only adapter matrices A and B are trained; this enables fine-tuning 70B models on a single 24GB GPU.
How to build it
After `get_peft_model`, iterate `for p in model.base_model.parameters(): p.requires_grad_(False)` and call `model.print_trainable_parameters()` to confirm only LoRA params are trainable.