LoopCoder-v2
Only loop once for efficient test-time computation scaling
Looped transformers make a model “think longer” by running the same block of layers over and over — reusing the same weights to get deeper computation for free. But each extra loop normally makes inference slower and hungrier for memory. The Parallel Loop Transformer (PLT) fixes that cost, so the real question becomes: how many loops actually help? Training a family of 7B code models from scratch (on 18 trillion tokens) with 1, 2, 3, and 4 loops, the authors find a surprising, sharply non-monotonic answer: two loops is the sweet spot. One extra loop lifts SWE-bench Verified from 43.0% to 64.4% — competitive with far larger models — but a third loop makes things worse, dropping to 27.6%. Their diagnostics explain why: the second loop does the real refinement, while later loops mostly add noise, and PLT's parallelism trick charges a small fixed “tax” every loop that eventually outweighs the shrinking benefit.
Jian Yang, Shawn Guo, Wei Zhang, Tianyu Zheng, et al.
43.0 → 64.4
SWE-bench Verified
The non-looped baseline (R=1) vs. the two-loop model (R=2) — a +21.4-point jump on real software-engineering tasks.
27.6
…then it regresses
SWE-bench Verified at R=3. A third loop drops well below even the baseline — looping too much actively hurts.
14.0 → 31.0
Multi-SWE (R=1 → R=2)
SWE-bench Multilingual more than doubles with the single extra loop.
7B
Model size
A 7-billion-parameter dense transformer — yet the 2-loop variant rivals 30B–72B open models on SWE-bench Verified.
18T
Pretraining tokens
Trained from scratch on 18 trillion tokens, balanced 1:1 between text and code across 100+ programming languages.
1M
Compute
GPU-hours to train the full family of loop-count variants in this study.
R = 2
Optimal loop count
One extra loop past the baseline. The benefit saturates immediately; more loops regress.
30–45×
Offset cost vs. gain
Past loop 2, PLT's fixed positional-mismatch cost outweighs each loop's shrinking refinement gain by 30–45×.
The one-line version
What if a model could just re-read its own thoughts?
A normal transformer has a fixed stack of layers — the input flows through once and out comes an answer. A looped transformer does something clever: it takes one block of layers and runs the input through it several times, reusing the exact same weights each pass. That gives you a deeper computation — more “thinking” — without adding a single new parameter.
This paper asks a deceptively simple question about that idea: how many times should you loop? The answer turns out to be surprising, and the reason why is the interesting part.
The setup
Looping is powerful — but normally it's expensive
Reusing one block of layers R times gives you the depth of an R-times-bigger model on a small parameter budget. That's great. But standard looping has a catch: each loop must wait for the previous one to finish (it's sequential), and each loop stores its own memory of the conversation (its “KV-cache”).
So with R loops, you pay roughly R times the latency and R times the memory. That makes deep looping impractical where speed or memory is tight.
h⁽⁰⁾ = Embed(x) · h⁽ʳ⁾ = f_θ( h⁽ʳ⁻¹⁾ ) · logits = Head( h⁽ᴿ⁾ )
A looped transformer applies the same shared block f_θ repeatedly: the output of loop r−1 becomes the input to loop r. Effective depth grows as R·L (loops × layers) while the parameter count stays fixed. The catch: h⁽ʳ⁾ can't be computed until h⁽ʳ⁻¹⁾ is done — strictly sequential.
Think of it like…
Re-reading a dense paragraph. The first read gets you the gist; a second read catches what you missed. But a fifth and sixth read rarely add anything — and if each re-read costs you real time, you'll want to stop early. This paper is about finding exactly where “stop early” is.
The efficiency trick
The Parallel Loop Transformer removes the cost
The Parallel Loop Transformer (PLT) — the framework this paper builds on — makes loop count almost free with two mechanisms:
- Shared-KV gated sliding-window attention (G-SWA): instead of every loop keeping its own full memory, all loops share the first loop's frozen memory, plus a small local window (width 64). This keeps total memory nearly constant no matter how many loops you run.
- Cross-loop position offset (CLP): this breaks the “each loop must wait for the last” chain, so loops can run concurrently inside a single forward pass — giving near-single-pass speed.
Sequential looping vs. PLT
| Sequential loop | PLT | |
|---|---|---|
| Execution | sequential | parallel, single pass |
| Latency | O(R · C_block) | ≈ C_block |
| KV-cache memory | O(R · L · S · d) | O(L · S · d) |
Sequential looping vs. PLT — drag the loop count
Sequential loop
Each loop waits for the last and stores its own cache.
PLT (parallel)
Loops overlap in one pass and share the first loop's frozen cache.
Drag R up: the sequential model's latency and memory climb with every loop, while PLT stays essentially flat. That's what makes loop count a free design choice — and turns the real question into “how many loops actually help?”
The clever part (with a catch)
How PLT breaks the waiting chain — the offset
Normally loop r for a token needs that token's own result from loop r−1. PLT sidesteps this: before each loop, it takes the previous loop's hidden states, shifts them one position to the right, and adds them back to the original word embeddings. So token i at loop r is fed a mix of its own embedding and its neighbor's (token i−1) previous-loop state.
That one-position shift is what lets loops overlap and run in parallel. But it comes at a price: each token is now refining using slightly mismatched context — its neighbor's, not its own. This positional mismatch is the core constraint PLT introduces, and it's central to everything that follows.
B⁽ʳ⁾ = Embed(x) + shift( h⁽ʳ⁻¹⁾ ) , h⁽ʳ⁾ = f_θ( B⁽ʳ⁾ )
The cross-loop position offset (CLP). shift(·) moves the previous loop's hidden states right by one token — so shift(h)ᵢ = hᵢ₋₁ (with the first slot zeroed). This removes the same-token dependency between consecutive loops, allowing loop r of token xᵢ to run concurrently with loop r+1 of token xᵢ₋₁ in one pass.
See the cross-loop offset in action
Input to loop 2 = Embed(x) + shift(h from loop 1)
h from loop 1
Notice each token is fed its left neighbor'sprevious-loop state, not its own. That mismatch is what lets the loops overlap and run in parallel — but it's also the fixed “positional tax” Ω(r) the paper charges to every loop.
The central idea
A tug-of-war: refinement gain vs. offset cost
The paper frames loop-count selection as a gain–cost trade-off:
- Gain — an extra loop is only worth it if it meaningfully refines the representation: coherently updating hidden states, changing how attention routes information, and shifting the output toward the right answer.
- Cost — the CLP offset injects a positional mismatch at every loop boundary. They measure this directly as the intrinsic offset cost Ω(r): the average distance between neighboring tokens' representations (if neighbors are very different, the shift is very lossy).
The punchline: the gain shrinks fast with each added loop, but the cost stays roughly constant. So beyond a couple of loops, the fixed tax dominates the vanishing benefit.
Ω(r) = (1/S) · Σᵢ ‖ h⁽ʳ⁻¹⁾ᵢ − h⁽ʳ⁻¹⁾ᵢ₋₁ ‖₂
The intrinsic offset cost: the mean distance between adjacent tokens' hidden states at the previous loop. Because the CLP shift feeds each token its neighbor's state, this distance is exactly how much information the shift distorts. Empirically Ω(r) is nearly constant across loops — a roughly fixed “positional tax” per loop.
The gain–cost scissors
The gain collapses after loop 2 while the offset cost holds steady — a “scissors” whose crossing point sets the optimal loop count at R=2. Drag across the chart to inspect each loop.
Think of it like…
Paying a flat $10 delivery fee for a meal. The first meal is worth it, the second maybe. But if each additional meal is smaller than the last while the $10 fee never changes, there's a point where you're paying mostly for delivery. That crossover is loop 2.
The evidence
Two loops wins — decisively and non-monotonically
The authors trained four matched 7B models — at 1, 2, 3, and 4 loops — from scratch on the same 18T tokens, then instruction-tuned and evaluated them identically. The result is strongly non-monotonic: performance jumps at two loops, then falls off a cliff.
On SWE-bench Verified (fixing real GitHub issues), the baseline scores 43.0%, the two-loop model leaps to 64.4% — competitive with 30B–72B open models and approaching 480B-scale systems — and the three-loop model regresses to 27.6%, below the baseline. The same shape holds across code generation, code reasoning, agentic software engineering, and tool-use benchmarks.
By the numbers
43.0 → 64.4
SWE-bench Verified
The non-looped baseline (R=1) vs. the two-loop model (R=2) — a +21.4-point jump on real software-engineering tasks.
27.6
…then it regresses
SWE-bench Verified at R=3. A third loop drops well below even the baseline — looping too much actively hurts.
R = 2
Optimal loop count
One extra loop past the baseline. The benefit saturates immediately; more loops regress.
30–45×
Offset cost vs. gain
Past loop 2, PLT's fixed positional-mismatch cost outweighs each loop's shrinking refinement gain by 30–45×.
Explore the loop-count results
SWE-bench Verified: Resolving real GitHub issues. The headline result: +21.4 at R=2, then a collapse. Peak at R=2, then regression — the core non-monotonic finding.
The diagnosis
Why exactly two? Looking inside the loops
The authors don't just report the curve — they open up the model to explain it, using per-loop “lenses”:
- Loop 2 does the real work. It produces the biggest change in attention routing, the biggest shift in the output distribution, and the peak in representational diversity (effective rank).
- Later loops spin their wheels. Beyond loop 2, updates become oscillatory — successive refinement directions reverse (their alignment goes negative), attention patterns freeze and heads grow redundant, and representational diversity shrinks. In the 4-loop model, a middle loop is nearly a dead pass-through.
- The offset tax never goes away. Because Ω(r) stays roughly constant while gains vanish, extra loops become net-negative — a clean mechanistic story for the saturation at two loops.
Looping as “latent” chain-of-thought
The looped refinement can be read as a form of chain-of-thought that happens inside the representation space — the model reasons across loops without ever writing out reasoning tokens. The paper suggests explicit (written) and latent (looped) reasoning are complementary, not substitutes.
The takeaway for practitioners
Because the gain–cost signature is visible in the model's own internals, you can pick a good loop count from interpretability diagnostics — hidden-state dynamics, attention change, output shift, and the offset cost Ω(r) — instead of expensively training and benchmarking every option. For PLT-style models today, that answer is: loop once extra, no more.
Key takeaways
- Looped transformers reuse one block of layers R times for deeper computation at no extra parameter cost — a lever for test-time compute.
- PLT makes loop count nearly free on latency and memory via a shared frozen KV-cache and a one-token cross-loop position offset (CLP).
- But CLP feeds each token its neighbor's context, creating a fixed per-loop “positional tax” — the offset cost Ω(r), which stays roughly constant.
- Refinement gain collapses after loop 2 while the offset cost holds steady, so performance is sharply non-monotonic and peaks at R=2.
- A 7B two-loop model hits 64.4% on SWE-bench Verified (up from 43.0%), rivaling far larger models; a third loop regresses to 27.6%.
- You can choose the loop count from the model's internal diagnostics, without exhaustively benchmarking every option.
References
- Yang et al. (2026). LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling. arXiv:2606.18023
- Dehghani et al. (2018). Universal Transformers
- (2026). Parallel Loop Transformer (PLT). arXiv (ref. [16] in the paper)
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.