LoRA: Low-Rank Adaptation of Large Language Models
Fine-tune a giant model by training two tiny matrices
To make a big pretrained model good at your specific task, the old way was “full fine-tuning”: nudge all of its billions of weights and save a brand-new full-size copy. For a model like GPT-3, that copy is ~350 GB — and you'd need one for every task. Painful and expensive. LoRA offers a beautiful shortcut. Leave the giant model's weights completely frozen, and next to each one add a tiny detour built from two small matrices multiplied together (call it B × A). Only those small matrices get trained. Because the change a model needs to learn a task turns out to be “low-rank” (simple, not requiring a full-size adjustment), these little add-ons are enough. The savings are staggering: on GPT-3, LoRA trains about 10,000× fewer numbers, uses 3× less GPU memory, and shrinks each task's saved file from 350 GB to 35 MB — while matching or beating full fine-tuning. And once trained, you can fold the detour back into the original weights, so it runs with zero extra delay. It's how almost everyone customizes big models today.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al.
10,000×
Fewer trainable params
On GPT-3 175B, LoRA trains ~10,000× fewer parameters than full fine-tuning (about 4.7M vs 175,255.8M).
3×
Less GPU memory
You don't store optimizer state for the frozen weights, cutting training VRAM by up to 2/3.
350GB → 35MB
Saved file per task
With r=4 on the query & value matrices, each task's checkpoint shrinks ~10,000× — small enough to email.
0 ms
Extra inference delay
The two small matrices can be merged back into the original weights, so a deployed LoRA model runs exactly as fast as the base model.
1.2TB → 350GB
GPT-3 training VRAM
Measured drop in VRAM when training GPT-3 175B with LoRA instead of full fine-tuning.
25%
Training speedup
On GPT-3 175B, ~25% faster training than full fine-tuning — no gradients needed for frozen weights.
on-par / better
Quality vs full fine-tuning
Matches or beats full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3 — despite far fewer trainable parameters.
1–8
Typical rank r
Surprisingly tiny. Even r=1 or r=2 often works well, showing the needed update is very low-rank.
The one-line version
Fine-tune a giant model by training two tiny matrices
LoRA is the trick that lets you customize a massive AI model cheaply — on a normal GPU, saving a tiny file instead of a giant one. It's now the default way to fine-tune big models. The whole idea fits in one sentence: freeze the giant model, and learn a small “detour” next to each weight instead of changing the weight itself. Let's build up to why that works — no math background needed.
The problem
Full fine-tuning makes a whole new giant every time
A pretrained model like GPT-3 has 175 billion numbers (weights) inside it. To specialize it for your task the classic method — full fine-tuning — nudges all of those numbers and saves the result as a brand-new model.
The catch: that new model is just as huge as the original. Want ten different specializations? That's ten ~350 GB copies. It needs massive GPUs to train and enormous storage to keep. As models keep growing, this goes from annoying to impossible.
Think of it like…
Owning one beautiful house (the pretrained model). Full fine-tuning is building a whole new house every time you want a different room layout. LoRA is keeping the one house and adding small, removable furniture arrangements — cheap to make, easy to store, and you can swap them in seconds.
Full fine-tuning vs. LoRA — the cost
Trainable parameters
Full checkpoint
350 GB
per task
LoRA checkpoint
35 MB
per task
On GPT-3 175B, LoRA trains ~10,000× fewer parameters (4.7M vs 175,255.8M) and each saved file drops from 350 GB to 35 MB.
The key insight
The change you need to learn is “simple”
Here's the observation LoRA is built on. When you adapt a model to a task, the change to its weights doesn't need to be complicated. In math terms, that change has a low “rank” — it can be captured by something far smaller than a full weight matrix.
So instead of learning a giant full-size change, LoRA learns a compressed version of it: two skinny matrices that, when multiplied, reconstruct the update. Small enough to train cheaply, expressive enough to do the job.
Two skinny matrices ≈ one big one
Full update ΔW
4,096
= d × d numbers
LoRA (B + A)
512
= 2 × d × r numbers
At r = 4, LoRA uses 8.0× fewer numbers than the full update — and the smaller r is, the bigger the saving. Real weight matrices are far larger than 64×64, so the real savings are enormous.
Think of it like…
Describing a huge multiplication table. Writing out every cell is enormous. But if the table is “low-rank,” you can rebuild the whole thing from just a short row and a short column multiplied together — a tiny recipe that reconstructs the big grid.
The mechanism
A frozen weight plus a trainable detour
For each weight matrix the model uses, LoRA keeps the original W₀ frozen (untouched, no training) and adds a parallel path made of two small matrices, B and A. The input goes through both: the frozen weight and the little detour. Their outputs are added together.
Only B and A are trained. And a neat detail makes this safe: A starts as small random numbers while B starts at exactly zero — so at the very beginning the detour adds nothing, and the model behaves exactly like the original until training gently shapes the detour.
h = W₀·x + ΔW·x = W₀·x + B·A·x (scaled by α/r)
The frozen weight W₀ and the low-rank update ΔW = B·A both act on the same input x, and their outputs are summed. B is d×r, A is r×k, with rank r ≪ min(d,k). B starts at 0 and A is a random Gaussian, so ΔW = 0 at the start of training. The detour's output is scaled by α/r, which lets you change r without re-tuning the learning rate.
Watch the forward pass
Input x goes through both paths; the outputs add up to h.
W₀
frozen — never trained
B · A
trainable detour
Only the violet B·A detour learns. Since B starts at zero, the detour adds nothing at first — the model begins identical to the original, then training gently shapes the detour.
Where to put it
Just the attention's query and value matrices
You don't even need to add a detour everywhere. The authors found that applying LoRA to just the query and value matrices inside the Transformer's attention (the mechanism from Attention Is All You Need) was enough to match full fine-tuning.
That frugality is why the rank can be tiny — often r = 4, sometimes even r = 1 — and still work. The needed adaptation really is that low-rank.
The best part for deployment
Zero extra delay when you run it
Some earlier efficiency tricks (called “adapters”) added extra layers that slowed the model down at run time. LoRA avoids this completely. Because the detour B·A has the same shape as the weight it sits beside, you can simply add it into the frozen weight once — computing W = W₀ + B·A — and then run the model normally.
Even better, swapping tasks is instant: subtract the old detour, add a new one. One frozen base model, a library of featherweight adapters.
One base model, many adapters
Frozen base model
Loaded once. Never changes. Shared by every task.
Click an adapter to merge it in (instant, zero latency):
Model output
No adapter merged — plain base model. Pick one above.
Each adapter is a ~35 MB file, not a 350 GB model. You keep one giant base in memory and swap featherweight adapters per task — subtract the old B·A, add the new one. That's why LoRA is so practical in production.
Think of it like…
Clip-on lenses for one pair of glasses. The glasses (base model) never change. You clip on a “legal documents” lens or a “write like a pirate” lens — each tiny and cheap — and pop it off to swap. Once clipped on, you see through them at full speed, no slowdown.
The payoff
Same quality, a tiny fraction of the cost
The results are what made LoRA take over. On GPT-3 175B, full fine-tuning trains all 175,255.8 million parameters; LoRA trains just 4.7 million — and matches or beats it. For example, on WikiSQL it scores 73.4 vs full fine-tuning's 73.8, and on MultiNLI-matched 91.7 vs 89.5 — better, with 10,000× fewer trainable parameters.
Across RoBERTa, DeBERTa, GPT-2, and GPT-3, the story repeats: LoRA holds its own or wins, while slashing memory, storage, and training time.
By the numbers
10,000×
Fewer trainable params
On GPT-3 175B, LoRA trains ~10,000× fewer parameters than full fine-tuning (about 4.7M vs 175,255.8M).
350GB → 35MB
Saved file per task
With r=4 on the query & value matrices, each task's checkpoint shrinks ~10,000× — small enough to email.
0 ms
Extra inference delay
The two small matrices can be merged back into the original weights, so a deployed LoRA model runs exactly as fast as the base model.
on-par / better
Quality vs full fine-tuning
Matches or beats full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3 — despite far fewer trainable parameters.
LoRA vs full fine-tuning on GPT-3
Full fine-tuning trains
175,255.8M
parameters
LoRA trains
4.7M
parameters (~10,000× fewer)
Task scores (higher is better)
LoRA matches full fine-tuning on WikiSQL and beatsit on MultiNLI (91.7 vs 89.5) and SAMSum — using a rounding-error's worth of the trainable parameters.
The lasting idea
You rarely need to change a giant model to specialize it — you just need to learn a small, low-rank “nudge” beside it. That insight made customizing large models cheap, storable, and instant to swap, and it's now the default across the field.
Figure 1 in the original paper
The paper's reparametrization diagram: the frozen pretrained weights W beside the trainable low-rank matrices A and B — only A and B are trained.
Hu et al., 2021 · view on arXiv
Key takeaways
- Full fine-tuning retrains all of a model's weights and saves a full-size copy per task — impractical for huge models.
- LoRA freezes the pretrained weights and learns a small low-rank detour ΔW = B·A beside each one; only B and A are trained.
- It works because the update a model needs to learn a task is low-rank — a tiny rank (often r=4, even r=1) is enough.
- B starts at zero and A is random, so the detour begins as a no-op and the model starts identical to the original.
- On GPT-3: ~10,000× fewer trainable parameters, 3× less GPU memory, checkpoints from 350GB to 35MB, ~25% faster — matching or beating full fine-tuning.
- Because B·A shares the weight's shape, it merges in for zero extra inference latency, and you can hot-swap adapters per task.
References
- Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022
- Aghajanyan et al. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- Vaswani et al. (2017). Attention Is All You Need. NeurIPS
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.