Transformers & Attention
The architecture behind modern AI — self-attention, multi-head attention, and the models that grew out of them.
Attention Is All You Need
The Transformer — the architecture behind modern AI
Before this paper, the best translation systems read sentences one word at a time, carrying a running memory forward (recurrent networks). That was slow — you can't process word 5 until word 4 is done — and forgetful over long sentences. The 2017 Transformer threw out that sequential machinery entirely and replaced it with attention: a mechanism that lets every word look directly at every other word and decide which ones matter. Because nothing waits its turn, the model trains fully in parallel on GPUs, captures long-range relationships in a single step, and set new translation records while training in a fraction of the time. This design became the foundation of GPT, BERT, Claude, and essentially all modern AI.
LoopCoder-v2
Only loop once for efficient test-time computation scaling
Looped transformers make a model “think longer” by running the same block of layers over and over — reusing the same weights to get deeper computation for free. But each extra loop normally makes inference slower and hungrier for memory. The Parallel Loop Transformer (PLT) fixes that cost, so the real question becomes: how many loops actually help? Training a family of 7B code models from scratch (on 18 trillion tokens) with 1, 2, 3, and 4 loops, the authors find a surprising, sharply non-monotonic answer: two loops is the sweet spot. One extra loop lifts SWE-bench Verified from 43.0% to 64.4% — competitive with far larger models — but a third loop makes things worse, dropping to 27.6%. Their diagnostics explain why: the second loop does the real refinement, while later loops mostly add noise, and PLT's parallelism trick charges a small fixed “tax” every loop that eventually outweighs the shrinking benefit.
BERT
Pre-training deep bidirectional transformers for language understanding
Before BERT, the strongest language models read text in one direction — left to right — so when understanding a word they could only use the words before it. BERT's insight was to read in both directions at once, letting every word draw on its full context, left and right. To train a model that way, the authors invented a clever fill-in-the-blank game: randomly hide 15% of the words and have the model guess them from everything around them (the Masked Language Model), plus a second task of judging whether one sentence naturally follows another. After this self-supervised “pre-training” on 3.3 billion words of books and Wikipedia — no human labels needed — the same model can be lightly “fine-tuned” for almost any language task by adding just one small output layer. This one recipe set new records on eleven benchmarks at once, pushing the GLUE score to 80.5% and SQuAD question-answering to a 93.2 F1.
Improving Language Understanding by Generative Pre-Training
GPT-1 — pre-train once, fine-tune for anything
In 2018, most language AI was built one task at a time: to do sentiment analysis you built a sentiment model, for question answering a question-answering model, each needing its own pile of hand-labeled examples — which are scarce and expensive. This paper (the original GPT) showed a better recipe. First, take a single Transformer and train it on a mountain of ordinary unlabeled text with one dead-simple goal: predict the next word, over and over, across 7,000+ books. That teaches it a surprising amount about grammar, facts, and reasoning — for free, no labels needed. Then, to specialize it for a real task, just keep that same model almost entirely unchanged and fine-tune it briefly on a small labeled set, feeding structured inputs (like a premise and hypothesis) as one plain sequence with a divider token. This one general model beat purpose-built, task-specific architectures on 9 of 12 benchmarks. It's the blueprint every GPT since has followed.
GPT-3
Language Models are Few-Shot Learners
People can learn a new task from just a couple of examples. If someone shows you two or three sentences and their French translations, you can usually translate the next one — no long training needed. Computers used to be bad at this. To teach a model a new task, you normally had to feed it thousands of labeled examples and retrain it. This paper asks a simple question: what if we just make the model really, really big? The authors built GPT-3, a text model with 175 billion “knobs” (parameters) — about 10 times bigger than anything like it before. Then they did something surprising: they gave it new tasks with only a few examples typed right into the prompt, and changed nothing inside the model. No retraining. It just read the examples and did the task. This is called few-shot learning, and GPT-3 turned out to be shockingly good at it — even at things like simple arithmetic and writing news articles people couldn't tell were fake.
LoRA: Low-Rank Adaptation of Large Language Models
Fine-tune a giant model by training two tiny matrices
To make a big pretrained model good at your specific task, the old way was “full fine-tuning”: nudge all of its billions of weights and save a brand-new full-size copy. For a model like GPT-3, that copy is ~350 GB — and you'd need one for every task. Painful and expensive. LoRA offers a beautiful shortcut. Leave the giant model's weights completely frozen, and next to each one add a tiny detour built from two small matrices multiplied together (call it B × A). Only those small matrices get trained. Because the change a model needs to learn a task turns out to be “low-rank” (simple, not requiring a full-size adjustment), these little add-ons are enough. The savings are staggering: on GPT-3, LoRA trains about 10,000× fewer numbers, uses 3× less GPU memory, and shrinks each task's saved file from 350 GB to 35 MB — while matching or beating full fine-tuning. And once trained, you can fold the detour back into the original weights, so it runs with zero extra delay. It's how almost everyone customizes big models today.