Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, et al.
Read it your way — plain-language or full technical depth.
3
Core steps
Indexing → Retrieval → Generation — the “Retrieve-Read” pipeline at the heart of every RAG system.
3
RAG paradigms
The survey organizes the field into three generations: Naive RAG, Advanced RAG, and Modular RAG.
3
Core components
Every RAG system rests on three pillars: Retrieval (what to fetch), Generation (how the model answers), and Augmentation (how they combine).
hallucination + staleness
Problems RAG targets
RAG directly attacks LLM hallucination, outdated knowledge, and untraceable reasoning by grounding answers in retrieved sources.
Top-k by meaning
Retrieval
Documents and the query are turned into vectors; RAG retrieves the k chunks most similar in meaning to the question.
add a document
Knowledge updates
Because knowledge lives in an external library, updating what the model “knows” is as easy as adding or editing documents — no retraining.
The one-line version
Give the model an open book
A language model only knows what it memorized during training. Ask it about last week's news, or your company's internal docs, and it either guesses (badly) or makes something up. Retrieval-Augmented Generation (RAG) fixes this with one elegant move: before the model answers, fetch relevant text from an external library and hand it over with the question. It turns a closed-book exam into an open-book one. This survey is the field's map of how RAG works and how it has evolved.
The problem
A model that memorized everything — and nothing new
Large language models are impressive, but they have three stubborn weaknesses:
- Hallucination — when unsure, they produce confident, plausible-sounding falsehoods.
- Outdated knowledge — they're frozen at their training cutoff; they can't know anything since.
- No private knowledge — they've never seen your documents, your database, or your domain.
And when they are wrong, you often can't tell why — there's no source to check. RAG targets all of these at once.
Think of it like…
A closed-book vs. open-book exam. A plain LLM takes the exam from memory alone — great for what it studied, shaky on anything else, and prone to bluffing. RAG lets it bring the textbook: for each question it flips to the relevant pages first, then answers from what's actually written there.
Same question, with and without RAG
“Most companies offer a 30-day refund window.”
✗ made up — it never saw your policy
retrieved: policy.pdf — “Refunds are accepted within 14 days of purchase.”
“Your refund window is 14 days from purchase.”
✓ grounded in a real, citable source
Without RAG the model answers from memory alone — so it guesses, hallucinates, or admits it can't know. With RAG it fetches the relevant source first, then answers from it. Same model, very different trustworthiness.
How it works
Three steps: Index, Retrieve, Generate
The basic RAG recipe — the survey calls it “Retrieve-Read” — has three steps:
1) Indexing. Take your documents (PDFs, web pages, notes), clean them, and chop them into bite-sized chunks. Turn each chunk into a vector (a list of numbers capturing its meaning) with an embedding model, and store them in a vector database.
2) Retrieval. When a question arrives, turn it into a vector too, and pull the Top-k chunks whose meaning is closest to the question.
3) Generation. Hand those chunks plus the original question to the model, and let it write an answer grounded in the retrieved text.
Watch a question flow through RAG
Step 1
Indexing
Documents are cleaned, chopped into bite-sized chunks, turned into vectors (numbers that capture meaning), and stored in a vector database — done once, ahead of time.
Retrieve by meaning, not keywords
Retrieving for: “How tall is the landmark in Paris?”
The Eiffel Tower is 330 m tall and located in Paris.
Top-2 ✓Photosynthesis converts sunlight into chemical energy in plants.
Paris is the capital of France and its largest city.
Top-2 ✓The French Revolution began in 1789.
Water boils at 100°C at sea level.
Notice matching is by meaning, not exact words: “landmark in Paris” pulls the Eiffel Tower chunk even though neither shares those words. Only the Top-2 chunks get sent to the model.
A real example
RAG on a question the model couldn't know
The survey's own worked example makes it concrete: someone asks about Sam Altman's sudden dismissal and rehiring at OpenAI — an event after the model's training. Without RAG, the model draws a blank. With RAG, it retrieves the fresh news and answers. Step through it below.
RAG in action — the 3 steps on a real question
User query
“How do you evaluate the fact that OpenAI's CEO, Sam Altman, went through a sudden dismissal by the board in just three days, and then was rehired by the company?”
Step 1
Indexing
Documents about the event are split into chunks, encoded into vectors, and stored in a vector database — ready to be searched by meaning.
Recreated from the survey's worked example (Fig. 2): a question about a very recent event the model was never trained on. RAG retrieves the news first, so it can answer instead of drawing a blank.
The evolution
Naive → Advanced → Modular RAG
The survey's central contribution is organizing RAG into three generations:
Naive RAG — the original “Retrieve-Read.” Simple and effective, but it has rough edges: retrieval can pull irrelevant or incomplete chunks, the model can still hallucinate or ignore the context, and stitching multiple chunks together can get repetitive or incoherent.
Advanced RAG — adds polish before and after retrieval. Before: rewrite or expand the question and index smarter. After: rerank the retrieved chunks so the best ones sit where the model pays most attention, and compress away the noise.
Modular RAG — treats RAG as swappable building blocks you can rearrange: add a Search module, a query-Rewrite step, a Generate-then-Read trick, and so on — tailoring the flow to the task.
The three generations of RAG
Naive RAG
“Retrieve-Read”The original recipe: index, retrieve Top-k, generate. Simple and effective.
Retrieval can miss or pull irrelevant chunks; the model may still hallucinate; stitched chunks can be repetitive.
Each generation keeps the core retrieve-then-generate idea and adds more control: Advanced RAG polishes a fixed pipeline; Modular RAG turns it into reconfigurable blocks you arrange per task.
For the curious — the fixes Advanced RAG adds
Pre-retrieval: query rewriting, query expansion, sliding-window chunking, and richer metadata to improve what gets found. Post-retrieval: re-ranking the retrieved chunks (relocating the most relevant to the edges of the prompt, where models attend most) and context compression to remove redundant or distracting text before generation.
The anatomy
Every RAG system has three parts
Underneath the paradigms, the survey frames every RAG system as three components you can improve independently:
- Retrieval — what to fetch: the embedding model, the chunking strategy, the vector database, how many chunks to pull.
- Generation — how the model answers given the context: the LLM and how it's prompted to stay faithful to the sources.
- Augmentation — how retrieved text and the question are combined: once, iteratively, or adaptively (e.g. deciding whether to retrieve again).
Thinking in these three pillars is how researchers reason about where a RAG system is weak and what to upgrade.
RAG vs. fine-tuning
A textbook to consult, not knowledge to memorize
How does RAG compare to fine-tuning (retraining the model on new data)? The survey offers a memorable distinction: RAG is like giving the model a tailored textbook to look things up in — ideal when you need precise, current, or private facts. Fine-tuning is like a student internalizing knowledge over time — better for teaching the model a particular style, structure, or format.
They're complementary. RAG shines in dynamic settings because you update knowledge just by editing the library — no retraining — and every answer can cite its source.
RAG vs. fine-tuning at a glance
| RAG | Fine-tuning | |
|---|---|---|
| Best for | precise, current, private facts | style, structure, format |
| Update knowledge | edit the library — instant | retrain the model |
| Traceable to a source | yes | no |
| Analogy | open-book exam | studying it into memory |
Where each method sits: knowledge vs. adaptation
Add relevant retrieved passages to the prompt. High external knowledge, little model change.
Recreated from the survey's Fig. 4. The higher a method sits, the more it leans on external knowledge; the further right, the more it adapts the model itself. RAG climbs the knowledge axis; fine-tuning moves right; combining them reaches the top-right.
The lasting idea
Don't force a model to memorize everything — let it look things up. By retrieving relevant text and grounding the answer in it, RAG makes AI more accurate, current, and traceable, without retraining. That's why “retrieve, then generate” is now a default building block of real-world AI systems.
Figures 1 & 5 in the original paper
For more, see the survey's technology tree of RAG research (Fig. 1) and its map of retrieval-augmentation processes — iterative, recursive, and adaptive retrieval (Fig. 5) — in the original paper.
Gao et al., 2024 · view on arXiv
Key takeaways
- LLMs hallucinate, go out of date, and can't see private data because all their knowledge is frozen in their weights.
- RAG fixes this by retrieving relevant text from an external library and feeding it to the model with the question — an open-book exam.
- The core pipeline is three steps: Indexing (chunk + embed + store), Retrieval (Top-k by meaning), Generation (answer from the chunks).
- The survey names three generations: Naive RAG (Retrieve-Read), Advanced RAG (pre/post-retrieval tricks like reranking), and Modular RAG (swappable building blocks).
- Every RAG system has three components to improve independently: Retrieval, Generation, and Augmentation.
- RAG vs. fine-tuning: RAG is a tailored textbook (precise, updatable, traceable facts); fine-tuning internalizes style and format. They're complementary.
References
- Gao et al. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS
- Vaswani et al. (2017). Attention Is All You Need. NeurIPS
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
The paper that introduced RAG — a model with two memories
A big language model stores everything it knows inside its own weights — call that its “memory from studying.” That memory is fuzzy: the model can't quote a source, can't easily be corrected, and forgets nothing gracefully. This 2020 paper — the one that actually coined “RAG” — gives a model a *second*, separate memory: a searchable index of all of Wikipedia (about 21 million passages). When you ask a question, a **retriever** finds the most relevant passages, and a **generator** (a seq2seq model, BART) writes the answer using both its own internal knowledge *and* those retrieved passages. The whole thing is trained end-to-end. They test two flavors — one that uses a single retrieved passage for the whole answer (RAG-Sequence), and one that can draw on a different passage for each word (RAG-Token) — and set new records on open-domain question answering, while producing answers that are more specific, more factual, and *traceable to a source*. Best of all, you can update what the model knows just by swapping the index — no retraining.