Low-Rank Adaptation
LoRA (Low-Rank Adaptation) freezes the original weight matrix W and learns a low-rank update ΔW = B·A where A and B are small factor matrices, reducing trainable parameters by orders of magnitude.
What is Low-Rank Adaptation?
LoRA (Low-Rank Adaptation) freezes the original weight matrix W and learns a low-rank update ΔW = B·A where A and B are small factor matrices, reducing trainable parameters by orders of magnitude.
LoRA (Low-Rank Adaptation) freezes the original weight matrix W and learns a low-rank update ΔW = B·A where A and B are small factor matrices, reducing trainable parameters by orders of magnitude.
Where is it used?
LoRA is the dominant PEFT method for Llama, Mistral, and Stable Diffusion fine-tuning; the `peft` library and `unsloth` provide optimised LoRA training pipelines.
How to build it
Use `peft.LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj","v_proj"])`, wrap the model with `get_peft_model`, and train — then inspect the A and B shapes to confirm low rank.
Code
A practical example:
Further reading
Want to go deeper? These resources cover Low-Rank Adaptation in more detail: