QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer
Read it your way — plain-language or full technical depth.
780GB → 48GB
65B model memory
Fine-tuning a 65B model needed >780 GB of GPU memory in 16-bit; QLoRA fits it in under 48 GB — a single GPU.
1 GPU
Hardware needed
One 48 GB GPU replaces a whole cluster — no quality loss versus full 16-bit fine-tuning.
99.3%
Guanaco vs ChatGPT
Their Guanaco chatbot reached 99.3% of ChatGPT's score on the Vicuna benchmark — trained in 24 hours on one GPU.
16-bit → 4-bit
Weight precision
The frozen base model is stored in 4-bit NF4 instead of 16-bit — roughly a 4× shrink per number.
24 hours
Guanaco 65B training
The best Guanaco model trained in a single day on one GPU (the 33B version in under 12 hours).
~3 GB
Double Quantization saves
Compressing the quantization bookkeeping saves ~0.37 bits per parameter — about 3 GB on a 65B model.
1,000+
Models trained
Over a thousand models across 8 datasets and multiple scales — including 33B and 65B, infeasible with regular fine-tuning.
no loss
Quality vs 16-bit
QLoRA matches full 16-bit fine-tuning performance despite the 4-bit base — the whole point.
The one-line version
Fine-tune a 65B model on a single GPU
This is the paper that let ordinary people fine-tune enormous models at home. LoRA already made fine-tuning cheap by training tiny add-on matrices instead of the whole model. But one wall remained: you still had to load the giant frozen base model into GPU memory — and the biggest models simply don't fit. QLoRA breaks that wall by shrinking the frozen model to 4 bits so it fits on one GPU, with no loss in quality. Let's see how.
The problem
The base model won't fit — even with LoRA
LoRA freezes the big model and only trains small adapters, so you don't need memory for optimizer state on the base. Huge saving. But you still have to hold the frozen model itself in memory to run inputs through it.
A 65-billion-parameter model in the usual 16-bit format needs over 780 GB of GPU memory to fine-tune — that's a rack of datacenter GPUs. For an individual, it's a non-starter. The bottleneck is no longer the training math; it's just storing the frozen weights.
Think of it like…
Moving into a small apartment. LoRA got rid of the furniture you don't need (optimizer state). But the couch — the frozen base model — is still too big for the door. QLoRA vacuum-packs the couch to a quarter of its size so it fits, and once inside it puffs back up to usable form only where you actually sit on it.
Will it fit on your GPU?
At 65B, full fine-tuning needs >780 GB — a whole cluster. Only QLoRA slips under the 48 GB line of a single GPU.
The core idea
Store each number in 4 bits instead of 16
Every weight in a model is a number stored with some precision. The usual format uses 16 bits per number. Quantization means storing them with fewer bits — here, just 4 bits — which is about a 4× shrink. Less precise, but far smaller.
The catch: naive 4-bit rounding loses too much and hurts the model. QLoRA's first innovation fixes this with a smarter 4-bit format called NF4 (NormalFloat), designed around the fact that a model's weights cluster around zero in a bell-curve shape. NF4 places its 16 possible values where the weights actually are, so each 4-bit code carries maximum information.
16-bit vs 4-bit: watch a weight get squashed
NF4 bunches its 16 levels near zero — exactly where the weights crowd — so almost every weight lands close to a level. Each 4-bit code carries maximum information.
Grey curve = how a model's weights are actually distributed (a bell curve centered on 0). Vertical lines = the 16 values a 4-bit number can take. Toggle to feel why NF4's placement is “information-theoretically optimal” for weights like these.
The three innovations
NF4, Double Quantization, and Paged Optimizers
Getting 4-bit to work without hurting quality took three ideas working together:
- 4-bit NormalFloat (NF4) — the smart 4-bit format above, matched to how weights are distributed.
- Double Quantization — quantization needs little “scaling constants” to remember each block's range. Those add up. So QLoRA also quantizes the constants themselves, saving about 0.37 bits per weight — roughly 3 GB on a 65B model.
- Paged Optimizers — training memory occasionally spikes (e.g. a long input). Instead of crashing, QLoRA temporarily “pages” memory out to the CPU and back, like a computer using disk when RAM is full.
For the curious
Double Quantization compresses the per-block constants (blocksize 64) using a second quantization with blocksize 256, cutting their cost from ~0.5 bits/param to ~0.127 bits/param. Paged Optimizers use NVIDIA unified memory to move optimizer state between GPU and CPU automatically, absorbing the gradient-checkpointing spike on long sequences.
Putting it together
Frozen 4-bit base + full-precision LoRA on top
Here's the whole trick combined. The giant base model sits frozen in 4-bit (tiny, just for storage). The small trainable LoRA adapters stay in full precision. When a number flows through, QLoRA briefly un-squashes the 4-bit weight back to normal precision (BF16), does the math, and adds the LoRA adapter's contribution.
So you get the memory savings of 4-bit storage and the training quality of full-precision adapters. Only the little adapters ever learn; the 4-bit base never changes.
Y = X · dequant(W₄ᵇⁱᵗ) + X · L₁ · L₂
The QLoRA forward pass (simplified from Eq. 5). The frozen 4-bit NF4 weight W is de-quantized back to BF16 on the fly (via double-dequantization of its stored constants), multiplied by the input X, and the LoRA adapter path X·L₁·L₂ (in BF16) is added. Storage type = 4-bit; compute type = BF16. Gradients flow only into L₁, L₂ — never into the 4-bit weights.
Watch a number flow through QLoRA
The 4-bit base un-squashes to BF16 just in time; the LoRA adapter runs alongside.
The 4-bit weight is only un-squashed for the instant it's needed, then discarded — so storage stays tiny. Gradients flow only into the LoRA adapter; the 4-bit base never changes.
Think of it like…
A high-res photo saved as a compact JPEG (4-bit base) with a small editable layer on top (LoRA). The JPEG stays compressed on disk to save space; you only decompress it briefly to view, and all your edits live in the tiny, lossless layer above it.
The payoff
Guanaco: 99.3% of ChatGPT, trained in a day
To prove QLoRA doesn't sacrifice quality, the authors fine-tuned a chatbot family called Guanaco. The best one reached 99.3% of ChatGPT's score on the Vicuna benchmark — while training in just 24 hours on a single GPU.
In head-to-head Elo ratings (judged by GPT-4 and humans), Guanaco 65B and 33B ranked just behind GPT-4 and above ChatGPT — remarkable for models trained on one GPU. And they fine-tuned over 1,000 models to study it thoroughly, including 33B and 65B models that plain fine-tuning simply couldn't run.
By the numbers
780GB → 48GB
65B model memory
Fine-tuning a 65B model needed >780 GB of GPU memory in 16-bit; QLoRA fits it in under 48 GB — a single GPU.
99.3%
Guanaco vs ChatGPT
Their Guanaco chatbot reached 99.3% of ChatGPT's score on the Vicuna benchmark — trained in 24 hours on one GPU.
24 hours
Guanaco 65B training
The best Guanaco model trained in a single day on one GPU (the 33B version in under 12 hours).
no loss
Quality vs 16-bit
QLoRA matches full 16-bit fine-tuning performance despite the 4-bit base — the whole point.
The Guanaco leaderboard
Elo ratings from a tournament judged by GPT-4 and humans. Higher is better. Guanaco models were trained with QLoRA on a single GPU.
99.3% of ChatGPT
Guanaco's score on the Vicuna benchmark — trained in 24 hours on a single 48 GB GPU.
Guanaco 65B and 33B land just behind GPT-4 and aboveChatGPT — remarkable for models fine-tuned on one GPU. The right-hand column is each model's GPU memory footprint.
The lasting idea
You don't need to keep a giant model in full precision to fine-tune it. Store it squashed to 4 bits, un-squash on the fly for the math, and learn only tiny full-precision adapters on top. That combination turned “fine-tune a 65B model” from a datacenter job into a single-GPU, overnight one — and opened high-quality customization to everyone.
Key takeaways
- LoRA still needs the frozen base model in memory; a 65B model in 16-bit needs >780 GB — too big for one GPU.
- QLoRA stores the frozen base model in 4-bit (a ~4× shrink), fitting a 65B model under 48 GB on a single GPU with no quality loss.
- Three innovations make 4-bit work: NF4 (a format matched to weight distributions), Double Quantization (compressing the compression constants), and Paged Optimizers (spilling memory spikes to the CPU).
- The 4-bit base is de-quantized to BF16 on the fly for the math; only the full-precision LoRA adapters are trained.
- Guanaco, fine-tuned with QLoRA in 24 hours on one GPU, reached 99.3% of ChatGPT on the Vicuna benchmark and ranked above ChatGPT in Elo.
- It made fine-tuning enormous models accessible to individuals — a major democratization of the field.
References
- Dettmers, Pagnoni, Holtzman, Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
The paper that introduced RAG — a model with two memories
A big language model stores everything it knows inside its own weights — call that its “memory from studying.” That memory is fuzzy: the model can't quote a source, can't easily be corrected, and forgets nothing gracefully. This 2020 paper — the one that actually coined “RAG” — gives a model a *second*, separate memory: a searchable index of all of Wikipedia (about 21 million passages). When you ask a question, a **retriever** finds the most relevant passages, and a **generator** (a seq2seq model, BART) writes the answer using both its own internal knowledge *and* those retrieved passages. The whole thing is trained end-to-end. They test two flavors — one that uses a single retrieved passage for the whole answer (RAG-Sequence), and one that can draw on a different passage for each word (RAG-Token) — and set new records on open-domain question answering, while producing answers that are more specific, more factual, and *traceable to a source*. Best of all, you can update what the model knows just by swapping the index — no retraining.