Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
The paper that introduced RAG — a model with two memories
A big language model stores everything it knows inside its own weights — call that its “memory from studying.” That memory is fuzzy: the model can't quote a source, can't easily be corrected, and forgets nothing gracefully. This 2020 paper — the one that actually coined “RAG” — gives a model a *second*, separate memory: a searchable index of all of Wikipedia (about 21 million passages). When you ask a question, a **retriever** finds the most relevant passages, and a **generator** (a seq2seq model, BART) writes the answer using both its own internal knowledge *and* those retrieved passages. The whole thing is trained end-to-end. They test two flavors — one that uses a single retrieved passage for the whole answer (RAG-Sequence), and one that can draw on a different passage for each word (RAG-Token) — and set new records on open-domain question answering, while producing answers that are more specific, more factual, and *traceable to a source*. Best of all, you can update what the model knows just by swapping the index — no retraining.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, et al.
Read it your way — plain-language or full technical depth.
44.5
Natural Questions (exact match)
RAG-Sequence set a new state of the art on open-domain Natural Questions, beating closed-book T5-11B (34.5), REALM (40.4), and DPR (41.5).
3 QA tasks
New SOTA on
Open-domain Natural Questions, WebQuestions, and CuratedTREC — with one general recipe, beating task-specific architectures.
21M passages
External memory
A dense vector index of all of Wikipedia (~21 million 100-word passages), searched with FAISS. This is the model's “second memory.”
2
Kinds of memory
Parametric (knowledge baked into the model's weights) + non-parametric (the external, searchable index). RAG combines both.
2
RAG formulations
RAG-Sequence uses one retrieved passage for the whole answer; RAG-Token can use a different passage for each word.
swap the index
Update knowledge
Because knowledge lives in the external index, you can update what the model “knows” by replacing the index — no retraining.
BART
Generator
A pre-trained seq2seq model is the parametric memory; the DPR neural retriever provides the non-parametric memory.
end-to-end
Training
The retriever and generator are fine-tuned jointly — the model learns what to retrieve and how to use it together.
The one-line version
The paper that gave AI a second memory
This is the paper that invented the term RAG. Its insight: a language model's knowledge shouldn't all live inside its weights. Give it a second, external memory — a searchable index of Wikipedia — and let it look things up while it writes. The model that studied and keeps an open reference library beats the one relying on memory alone. Let's unpack how.
The problem
Knowledge baked into weights is fuzzy
Big language models store facts inside their parameters (their weights) — the paper calls this parametric memory. It works surprisingly well, but it has real limits:
- You can't easily see where an answer came from (no source, no provenance).
- You can't update it without expensive retraining — its knowledge is frozen.
- It's imprecise: the model half-remembers facts and can blur them together.
On knowledge-intensive tasks (questions whose answers a person would need to look up), this fuzziness shows, and specialized systems used to win.
Think of it like…
Two kinds of memory in your own head. Parametric memory is what you've internalized — fast, but fuzzy and hard to update. Non-parametric memory is looking it up in a reference book — precise, citable, and you can swap in a newer edition anytime. This paper gives the model both.
The core idea
Parametric + non-parametric memory
RAG combines two memories:
- Parametric memory — a pre-trained seq2seq model (BART) that already knows a lot from training and can write fluent text.
- Non-parametric memory — a dense vector index of ~21 million Wikipedia passages, searchable by meaning. This is knowledge the model can look up rather than memorize.
A retriever fetches relevant passages from the index; the generator writes the answer using both its own knowledge and those passages. Crucially, the two are trained end-to-end — the model learns what to fetch and how to use it, together.
The RAG architecture — press play
Query
“When did the middle ear evolve?”
Query Encoder
q(x) → vector
MIPS · Document Index
search 21M Wikipedia passages
Top-K passages
z₁ … z_K retrieved
Generator (BART)
writes using query + passages
Answer
marginalized over passages
Recreated from the paper's Figure 1 (our design). One query flows through the retriever and generator; the same pipeline handles QA, fact verification, and question generation.
The retriever
Finding the right passages by meaning
The retriever is DPR (Dense Passage Retrieval). It turns the query into a vector and every Wikipedia passage into a vector, then finds the closest matches using Maximum Inner Product Search (MIPS) — a fast “find the most similar vectors” operation (built with FAISS). It returns the top-K passages (e.g. the 5 most relevant).
Because it matches by meaning, it finds the right passage even when the question doesn't share the passage's exact words.
Encode the query, search by meaning
Inner-product match against Wikipedia passage vectors (MIPS):
The middle ear includes the tympanic cavity and the three ossicles.
top-2 ✓Barack Obama was born in Honolulu, Hawaii, in 1961.
The ossicles are three tiny bones that transmit sound in the ear.
top-2 ✓The Divine Comedy is divided into three sections by Dante.
Sound waves cause the eardrum to vibrate, moving the middle-ear bones.
Only the top-2 passages go to the generator. Matching is by meaning: “sound transmitted in the ear” pulls the ossicles passage even without shared words. This is DPR + MIPS over ~21M passages.
Two flavors
RAG-Sequence vs. RAG-Token
How much should one retrieved passage influence the answer? The paper tries two options:
RAG-Sequence — pick the retrieved passages once and use the same one to generate the entire answer. Simple and coherent; best when the answer comes from a single source.
RAG-Token — allow the model to draw on a different passage for each word it generates. More flexible; great when different parts of the answer need different sources.
Under the hood, the model considers several retrieved passages and marginalizes — blends their predictions — rather than betting on just one.
One passage for all, or a different one per word?
Question: “Which prize did the author of The Sun Also Rises win?”
Generated answer
RAG-Token: each part of the answer can draw from a different passage — the author from Doc A/B, the prize from Doc C. More flexible when the answer spans multiple sources.
Think of it like…
Writing an answer with reference books open. RAG-Sequence is picking one book and writing the whole answer from it. RAG-Token is keeping several books open and glancing at whichever one has the fact you need for the next sentence.
The payoff
New records — and answers you can trace
RAG set the state of the art on three open-domain QA tasks. On Natural Questions it scored 44.5 (exact match), beating the closed-book T5-11B (34.5), REALM (40.4), and DPR (41.5) — despite those being much more specialized. It also won on WebQuestions and CuratedTREC.
The two flavors split the wins: RAG-Sequence edged ahead on QA, while RAG-Token produced better generation (more specific, diverse Jeopardy questions; stronger fact verification). And because answers come from retrieved passages, they're traceable — and updatable by swapping the index.
By the numbers
44.5
Natural Questions (exact match)
RAG-Sequence set a new state of the art on open-domain Natural Questions, beating closed-book T5-11B (34.5), REALM (40.4), and DPR (41.5).
3 QA tasks
New SOTA on
Open-domain Natural Questions, WebQuestions, and CuratedTREC — with one general recipe, beating task-specific architectures.
21M passages
External memory
A dense vector index of all of Wikipedia (~21 million 100-word passages), searched with FAISS. This is the model's “second memory.”
swap the index
Update knowledge
Because knowledge lives in the external index, you can update what the model “knows” by replacing the index — no retraining.
RAG vs. the specialists
Natural Questions — exact-match score (higher is better)
One general RAG recipe (teal) matches or beats closed-book giants like T5-11B and task-specific systems like REALM and DPR — setting the state of the art on Natural Questions, WebQuestions, and CuratedTREC.
A neat trick
Update the world by editing the index
Because the model's factual knowledge lives in the external index, you can change what it knows without retraining. The authors demonstrate this: swap the Wikipedia index for a newer one, and the model's answers update to match — for example, naming the current world leader for a given year. That's something a purely parametric model simply can't do without expensive retraining.
The lasting idea
Split a model's knowledge in two: what it internalized (parametric) and what it can look up (non-parametric). Let a retriever fetch real passages and a generator write from them, trained together. That single recipe — introduced here — made AI more accurate, more honest about its sources, and updatable. It's the foundation of modern RAG.
Key takeaways
- This is the paper that introduced “RAG” — combining a model's internal (parametric) memory with an external, searchable (non-parametric) memory.
- The non-parametric memory is a dense index of ~21M Wikipedia passages; a DPR retriever finds the top-K by meaning (MIPS), and a BART generator writes the answer using them.
- Retriever and generator are fine-tuned end-to-end — the model learns what to retrieve and how to use it together.
- Two formulations: RAG-Sequence (one passage for the whole answer) and RAG-Token (a different passage per word); RAG marginalizes over several retrieved passages.
- RAG set the state of the art on open-domain Natural Questions (44.5 EM), WebQuestions, and CuratedTREC — beating far larger closed-book models and task-specific systems.
- Answers are traceable to sources, and you can update the model's knowledge by swapping the index — no retraining.
References
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
- Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.