LoRA Rank
LoRA rank r is the inner dimension of the A and B factor matrices, controlling the expressiveness of the adapter — higher r captures more complex updates but increases parameter count and memory.
What is LoRA Rank?
LoRA rank r is the inner dimension of the A and B factor matrices, controlling the expressiveness of the adapter — higher r captures more complex updates but increases parameter count and memory.
LoRA rank r is the inner dimension of the A and B factor matrices, controlling the expressiveness of the adapter — higher r captures more complex updates but increases parameter count and memory.
Where is it used?
Common ranks are r=8 or r=16 for Llama/Mistral instruction tuning; r=64+ is used for domain-heavy adaptation; the rank vs performance trade-off is empirically studied in the original LoRA paper.
How to build it
Train the same dataset with `LoraConfig(r=4)`, `r=16`, and `r=64`, evaluate loss and task accuracy, and plot parameter count vs performance to find the sweet spot for your task.