The glossary
Concepts, defined once
Every recurring idea across our papers, explained in plain language — no prior knowledge required.
[CLS]
A special token at the start of every sequence; its final hidden state is used as the summary for classification tasks.
Adapter
The small trainable matrices (kept in full precision in QLoRA) that actually learn the task, on top of the frozen 4-bit base.
Advanced RAG
RAG with pre-retrieval (query rewriting, better indexing) and post-retrieval (reranking, compression) improvements.
Alignment
Making a model actually do what the user wants — be helpful, honest, and harmless — instead of just predicting plausible text.
Alignment tax
The small drop in performance on some standard tasks that can come from aligning a model — largely fixable with PPO-ptx.
Attention
A mechanism that lets the model weigh how much every other token matters to the current one, all at once.
Augmentation
How retrieved text and the question are combined and fed to the model — once, repeatedly, or adaptively.
Autoregressive
Generating output one token at a time, each conditioned on the tokens generated before it.
Auxiliary objective
A secondary training goal (here, keeping language modeling during fine-tuning, weighted by λ) that improves generalization and speed.
B and A matrices
The two small trainable matrices in LoRA. B is d×r, A is r×k, and r (the rank) is tiny. B starts at zero, A is random.
BERT
Bidirectional Encoder Representations from Transformers — a model pre-trained on raw text that can be fine-tuned for many language tasks.
BF16
A 16-bit number format used for computation. QLoRA stores in 4-bit but computes in BF16 after un-squashing.
Bidirectional
Using context from both sides of a word (left and right) at once, rather than only the words before it.
Bit / precision
How many bits are used to store each number. 16-bit is standard; 4-bit uses a quarter of the space per number.
BLEU
A score (0–100) measuring how close a machine translation is to human reference translations; higher is better.
BooksCorpus
The pre-training dataset: 7,000+ unpublished books, valued for long, contiguous passages that teach long-range structure.
Byte-pair encoding (BPE)
A way of splitting text into subword tokens (here, 40,000 merges) so the model handles rare and novel words gracefully.
Capability
An optional feature each side declares during initialize; both must respect what was negotiated for the whole session.
Chain of thought
A series of intermediate reasoning steps a model writes out before its final answer — like showing your work.
Chain-of-thought prompting
Including step-by-step reasoning in a prompt's few-shot examples so the model imitates that reasoning on new problems.
Checkpoint
A saved copy of a model's trained weights. LoRA's checkpoints are tiny (megabytes) instead of hundreds of gigabytes.
Chunk
A small, digestible piece of a document. RAG retrieves chunks (not whole documents) so it can fetch just the relevant bits.
Client
A connector inside the host that keeps a private 1:1 connection to one server.
Common Crawl
A giant public archive of web pages; a filtered version was the bulk of GPT-3's training text.
Comparison / ranking
Instead of scoring one answer, a person orders several answers best-to-worst; each pair becomes a training signal for the reward model.
Context window
How much text the model can read at once. Few-shot examples have to fit inside it.
Cross-attention
Attention where the decoder's words attend to the encoder's representation of the input sentence.
Cross-loop position offset (CLP)
PLT's trick of shifting the previous loop's hidden states one token to the right before the next loop, which breaks the sequential dependency so loops can overlap.
De-quantize
Un-squashing a 4-bit weight back to higher precision (BF16) just in time to do the math, then discarding it.
Decoder
The stack that generates the output sentence one token at a time, using the input and what it has written so far.
Decoder-only Transformer
A Transformer that keeps only the decoder stack, using causal masking so each token sees only earlier tokens. The GPT shape.
Delimiter token
A special token (like $) inserted between parts of an input to mark boundaries, e.g. between a premise and hypothesis.
dₖ (key dimension)
The length of the Query and Key vectors; attention scores are divided by √dₖ to keep the softmax stable.
Double Quantization
Also quantizing the small “scaling constants” that quantization itself needs — saving ~0.37 bits per weight (~3 GB on a 65B model).
DPR
Dense Passage Retrieval — encodes queries and passages as vectors so relevant passages can be found by vector similarity.
Effective rank
A measure of how diverse the token representations are. It peaks at loop 2 and shrinks afterward, meaning later loops narrow rather than enrich the representation.
Elicitation
A client primitive: a server asks the user for structured input mid-task; the client renders a form. Never for sensitive data.
Elo rating
A score from head-to-head match-ups (like chess rankings) used here to compare chatbots, judged by GPT-4 and humans.
Embedding / vector
A list of numbers that captures the meaning of a piece of text, so similar meanings sit close together and can be searched.
Emergent ability
A capability absent in small models that appears only past a certain scale, and can't be predicted by extrapolating from smaller ones.
Encoder
The stack that reads the input sentence and builds a rich, context-aware representation of it.
Exact match (EM)
A QA metric: the fraction of questions where the model's answer exactly matches the correct answer.
Exemplar
One worked example placed in the prompt. Chain-of-thought used about eight per task.
FastMCP
The high-level Python API for building MCP servers: @mcp.tool() turns a typed function into a tool automatically.
Feed-forward network (FFN)
A per-token network (two linear layers with a ReLU between) applied after attention to each word independently.
Few-shot prompting
Steering a model by giving it a few examples in the prompt, with no weight updates — also called in-context learning.
Fine-tuning
Updating a model's weights by training on task data — the expensive alternative that chain-of-thought's prompt-only method beat on GSM8K.
Fine-tuning
Adapting a pretrained model to a specific task or behavior by training it further. QLoRA makes this cheap for huge models.
Frozen weights
The base model's numbers, left unchanged during fine-tuning. QLoRA stores them in 4-bit since they never need to be updated.
Full fine-tuning
The classic approach: update ALL of a model's weights and save a full-size copy per task — expensive for large models.
G-SWA (gated sliding-window attention)
An attention scheme where each loop blends a small local window with a shared frozen memory from the first loop, keeping total memory nearly constant.
Gain–cost trade-off
The paper's framing: each loop's refinement benefit vs. the fixed positional-mismatch cost it incurs. The best loop count is where gain still beats cost.
GELU
Gaussian Error Linear Unit — the smooth activation function GPT uses in place of ReLU.
Generation
The step where the model writes its answer — in RAG, using the retrieved text as grounding.
Generative pre-training
Training a model to generate text (predict the next word) on lots of unlabeled data, before adapting it to specific tasks.
Generator
The component that writes the answer. Here it's BART, a pre-trained seq2seq model, conditioned on the query and retrieved passages.
GPU memory (VRAM)
The fast memory on a graphics card. Fitting a model in it is the main constraint QLoRA solves — a 65B model now fits in 48 GB.
GSM8K
A benchmark of ~8,000 grade-school math word problems requiring multi-step arithmetic reasoning.
Guanaco
The chatbot family the authors fine-tuned with QLoRA to prove it works — reaching 99.3% of ChatGPT on the Vicuna benchmark.
Hallucination
When a model confidently produces information that sounds right but isn't true or supported by any source.
Host
The AI app you use (e.g. Claude Desktop, an AI IDE). It holds the conversation, manages clients, and handles consent.
Human feedback
Signals from people — written examples and rankings of answers — used to teach the model what good behavior looks like.
In-context learning
A model learning a task purely from examples in its prompt at inference time, without changing its weights.
Indexing
Preparing documents for retrieval: cleaning them, splitting into chunks, turning each into a vector, and storing them for fast search.
Inference latency
The delay when the model generates output. LoRA adds none, because the detour merges into the original weights.
initialize / handshake
The first exchange when a client and server connect: they agree on a version and declare their capabilities.
Input transformation
Reshaping a structured task input into a single token sequence with delimiters, so one unchanged model can process it.
Instruction following
Doing what a request actually asks — including explicit constraints like “answer in two sentences.”
JSON-RPC 2.0
The simple message format MCP uses — requests, results, errors, and one-way notifications, all as JSON.
KL penalty
A term that keeps the new policy from drifting too far from the original model — a “leash” that prevents reward-hacking.
Knowledge-intensive task
A task whose answers a person couldn't produce without looking things up — e.g. open-domain question answering.
KV-cache
The stored keys and values a transformer keeps so it doesn't recompute attention over earlier tokens; its size drives memory cost.
Labeler
A person hired to write example answers and rank model outputs, providing the human feedback the method relies on.
LaMDA
A Google dialogue model family (up to 137B parameters), one of the three model families evaluated in the paper.
Language model
A model that predicts likely next words in text. Trained well, this simple skill turns into answering, translating, and writing.
Language modeling
The task of predicting the next word given the previous words. Doing it well requires learning grammar, facts, and context.
Latent chain-of-thought
Reasoning that happens inside the model's hidden representations across loops, without writing out explicit reasoning text.
Layer normalization
Rescaling a token's feature values to a stable range at each step to keep training smooth.
Logging
A client primitive: a server sends structured log messages up to the client.
Loop count (R)
How many times the shared block is applied. R=1 is a normal (non-looped) model; this paper studies R = 1, 2, 3, 4.
Looped transformer
A model that runs its input through one shared block of layers multiple times, reusing the same weights to get deeper computation without more parameters.
LoRA
Low-Rank Adaptation — training tiny add-on matrices (adapters) beside a frozen model instead of changing the whole model. QLoRA builds on it.
Low-rank update
A change to the weights simple enough to be written as two small matrices multiplied together (B·A) instead of one big one.
Marginalize
Blending the generator's predictions across several retrieved passages, weighted by how relevant each is, instead of betting on one.
Masked (causal) self-attention
Attention restricted so a token can only look at earlier tokens, never future ones — required for honest next-word prediction.
Masked Language Model (MLM)
BERT's main pre-training task: hide some words and train the model to predict them from surrounding context.
Masking
Hiding future tokens from the decoder so it can't cheat while predicting the next word.
MCP
Model Context Protocol — an open standard for connecting AI applications to external tools and data.
Merging
Adding B·A back into the frozen weight (W = W₀ + B·A) so the model runs with no extra layers and zero added delay.
MIPS
Maximum Inner Product Search — a fast way to find the vectors most similar to the query vector (implemented with FAISS).
Modular RAG
RAG built from swappable, reconfigurable modules (search, rewrite, generate-read, fusion) tailored to the task.
Multi-head attention
Running attention several times in parallel with different learned lenses, then combining the results.
Naive RAG
The original, straightforward RAG: retrieve relevant chunks, then read/generate. Simple but with rough edges.
Next Sentence Prediction (NSP)
A pre-training task where the model judges whether one sentence actually follows another.
NF4 (NormalFloat)
QLoRA's 4-bit number format, with its 16 levels placed where model weights actually cluster (a bell curve), so it keeps maximum information.
Non-monotonic
A trend that goes up then down (not steadily in one direction). Here, performance rises at R=2 then falls at R=3+.
Non-parametric memory
Knowledge stored outside the model — here, a searchable index of Wikipedia the model can look things up in. Precise, citable, swappable.
Notification
A one-way JSON-RPC message with no id and no reply, e.g. notifications/initialized.
Offset cost Ω(r)
How much the CLP shift distorts a token's input, measured as the average distance between neighboring tokens' representations. It stays roughly constant across loops.
Open-domain QA
Answering questions that could be about anything, without being given the relevant passage — the model must find it first.
Paged Optimizers
Temporarily moving optimizer memory to the CPU during brief spikes (via NVIDIA unified memory) so training doesn't crash.
PaLM
Google's 540-billion-parameter language model, the largest tested here and the one that set the GSM8K record with chain-of-thought.
Parallel Loop Transformer (PLT)
A looped-transformer design that lets loops run in parallel and share memory, so adding loops barely costs extra time or memory.
Parameter
One of the tiny adjustable numbers a model learns during training. GPT-3 has 175 billion of them — more parameters usually means more capacity.
Parametric memory
Knowledge stored inside a model's weights (learned during training). Fast but fuzzy, hard to update, and can't cite sources.
Policy
In RL, the model that produces outputs (here, the summarizer). It's the thing being optimized.
Positional encoding
Numbers added to embeddings that tell the model the position/order of each token, since attention itself is order-blind.
PPO
Proximal Policy Optimization — the reinforcement-learning method the model uses to practice and improve its reward score.
PPO-ptx
A version of PPO that mixes in the original next-word training during practice, to avoid losing general skills (paying down the alignment tax).
Preference comparison
Showing a human two outputs and recording which one they judge better — the training signal for the reward model.
Prompt
A user-controlled, reusable message template a server exposes, often as a slash command. Fetched with prompts/get.
Provenance
Being able to trace an answer back to its source. RAG provides it because answers come from retrieved passages.
Proxy objective
A stand-in goal you can measure (like ROUGE) that only approximates what you truly care about.
Quantization
Storing numbers with fewer bits (here, 4 instead of 16) to shrink a model's memory footprint. Less precise, but much smaller.
Query / Key / Value
Three vectors per token: the request (Query) is matched against descriptions (Keys) to decide how much of each token's content (Value) to use.
Query & value matrices
Two of the weight matrices inside attention. Applying LoRA to just these was enough to match full fine-tuning.
RAG
Retrieval-Augmented Generation — fetching relevant external text and giving it to the model with the question so its answer is grounded in real sources.
RAG-Sequence
The formulation that uses the same retrieved passage to generate the entire answer.
RAG-Token
The formulation that can use a different retrieved passage for each generated word.
Rank
A measure of how “complex” a matrix is. A low-rank matrix can be rebuilt from a few skinny pieces — the key to LoRA's savings.
Reference summary
A human-written summary used as the training target and comparison baseline.
ReLU
A simple function that keeps positive numbers as-is and turns negative numbers into zero.
Reranking
Reordering retrieved chunks so the most relevant ones are placed where the model pays the most attention.
Residual connection
A shortcut that adds a layer's input back onto its output so information and learning signals flow easily through deep stacks.
Resource
Application-controlled data a server exposes for context, addressed by a URI. Read with resources/read.
Retrieval
Finding the most relevant pieces of text for a question from an external knowledge library.
Retriever
The component that finds relevant passages for a query. In this paper it's DPR (Dense Passage Retrieval).
Reward hacking
When a policy finds outputs that score high on the reward model without actually being good — a failure the KL penalty guards against.
Reward model
A second model (the “judge”) trained to score any answer by predicting how much a human would prefer it.
RLHF
Reinforcement Learning from Human Feedback — the three-step recipe in this paper for training a model on what people prefer.
RNN
Recurrent neural network — processes a sequence one step at a time, carrying a hidden memory forward.
Roots
A client capability: the client tells a server which folders/files it's allowed to work within (roots/list).
ROUGE
A summarization metric based on word overlap with a reference summary. A rough proxy for quality, not quality itself.
Sampling
A client primitive: a server asks the host's LLM to run a completion on its behalf (no server-side API keys), with user approval.
Scale
Making the model (and its training data) much bigger. This paper's central finding is that scale unlocks new abilities.
Segment embedding
An added vector telling BERT whether a token belongs to sentence A or sentence B.
Self-attention
Attention within one sequence — words attending to other words in the same sentence.
Self-supervised
Training where the labels come from the data itself (e.g. hidden words), so no human annotation is needed.
Semantic search
Searching by meaning rather than exact words — so a query finds relevant text even if the wording differs.
Seq2seq
A sequence-to-sequence model that maps an input sequence to an output sequence — the “workhorse” generator RAG builds on.
Server
An independent program that exposes tools, resources, and prompts to AI apps via MCP. Can be local or remote.
Softmax
A function that turns a list of raw scores into positive percentages that add up to 100%.
stdio transport
The host runs the server as a local subprocess and exchanges JSON over stdin/stdout. Simplest, local.
Streamable HTTP
The transport for remote servers: one HTTP endpoint (POST + optional SSE stream). Replaced the older HTTP+SSE transport.
Supervised fine-tuning (SFT)
Step 1: train the model to imitate high-quality example answers written by people.
SWE-bench Verified
A benchmark that tests whether a model can resolve real software-engineering issues from GitHub repositories.
Symbolic reasoning
Rule-following tasks like concatenating last letters or tracking coin flips — used to test whether the model learned a procedure that generalizes.
Test-time compute
Spending extra computation when the model answers (rather than making the model bigger) to improve quality.
Textual entailment
Deciding whether one sentence (the hypothesis) logically follows from another (the premise).
TL;DR dataset
A dataset of Reddit posts paired with author-written “too long; didn't read” summaries, used for training and evaluation.
Token
A chunk of text — a word or word-piece — that the model treats as a single unit.
Tool
A model-controlled action a server exposes (e.g. get_weather). Discovered with tools/list, run with tools/call.
Top-k
The k most relevant chunks retrieved for a question (e.g. the top 3 or 5).
Trainable parameters
The numbers actually updated during training. LoRA trains only B and A — thousands of times fewer than full fine-tuning.
Transfer learning
Reusing knowledge learned on one task (here, next-word prediction) to boost performance on another.
Transformer encoder
The encoder half of the Transformer architecture — the bidirectional stack BERT is built from.
Transport
The pipe MCP messages travel through (stdio or Streamable HTTP). The JSON-RPC messages are the same either way.
TruthfulQA
A benchmark that tests whether a model gives truthful answers to questions people often get wrong.
Unlabeled data
Raw text with no human annotations. Abundant and free — the fuel for pre-training.
Vector database
A store built to quickly find the vectors (and their text) most similar in meaning to a query vector.
Verifier
A separate model trained to score/rank a base model's candidate answers. The prior GSM8K record paired a fine-tuned model with one.
Weight matrix
A grid of numbers a neural network multiplies inputs by. Large models have billions of these numbers.
WordPiece
A tokenization method that splits text into subword units from a fixed vocabulary (30,000 tokens for BERT).
Zero-shot
Performing a task with no task-specific fine-tuning at all — a capability the pre-trained model already showed hints of.
α/r scaling
A constant that scales the detour's output, letting you change the rank r without re-tuning the learning rate.