LoRA makes fine-tuning cheap because it trains two thin matrices instead of the full weight matrix.
What to build
A program that compares the weights of full fine-tuning with LoRA for one layer.
Requirements
- Print
full F, the number of weights of the layer. - Print
lora L, the number of weights LoRA trains with rank r. - Print
percent P, LoRA as a percentage of full, to two decimal places.
Input
One line with d_in d_out r.
Output
Three lines.
Constraints
1 ≤ r ≤ min(d_in, d_out) ≤ 100000.
Example
Input:
4096 4096 8
Output:
full 16777216
lora 65536
percent 0.39
Why it works
The full matrix has d_in times d_out entries, while two thin matrices of sizes d_in by r and r by d_out together have r times d_in plus d_out entries, which is far fewer when r is small.
Examples
Example 1 · Input
4096 4096 8
Expected output
full 16777216 lora 65536 percent 0.39
Example 2 · Input
100 50 2
Expected output
full 5000 lora 300 percent 6.00
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
A full layer has `d_in * d_out` weights.
Hint 2
LoRA of rank r trains `r * (d_in + d_out)` numbers; the percentage is 100 times LoRA over full.
Solution
Show one correct solution (try it yourself first)
d_in, d_out, r = map(int, input().split())
full = d_in * d_out
lora = r * (d_in + d_out)
print(f"full {full}")
print(f"lora {lora}")
print(f"percent {100 * lora / full:.2f}")