Low-Rank Matrices
The low-rank matrices A (down-projection, shape r×d) and B (up-projection, d×r) approximate the full weight update with rank r ≪ d, so ΔW has at most rank r instead of full rank d.
What is Low-Rank Matrices?
The low-rank matrices A (down-projection, shape r×d) and B (up-projection, d×r) approximate the full weight update with rank r ≪ d, so ΔW has at most rank r instead of full rank d.
The low-rank matrices A (down-projection, shape r×d) and B (up-projection, d×r) approximate the full weight update with rank r ≪ d, so ΔW has at most rank r instead of full rank d.
Where is it used?
A typical LoRA on Llama-3-8B with r=8 and target_modules q/v adds ~14M trainable params vs 8B total; used in every `peft.LoraConfig` application across the open-source community.
How to build it
Manually implement `self.lora_A = nn.Parameter(torch.randn(r, d_in)); self.lora_B = nn.Parameter(torch.zeros(d_out, r))` and add `B @ A * scaling` to the frozen linear output in a custom layer.