Fine-tuning

The low-rank decomposition, precisely

For a pretrained weight ${\mathbf{W}}_0\in {\mathbb{R}}^{d_{out}\times d_{in}}$, full fine-tuning would learn an unconstrained update $\Delta \mathbf{W}\in {\mathbb{R}}^{d_{out}\times d_{in}}$ — as many free parameters as ${\mathbf{W}}_0$ itself. LoRA constrains it to rank $r$: