Attention Is All You Need

GPT, Claude, Gemini, BERT, AlphaFold — they are all, under the hood, Transformers. This is the paper that introduced them. Its radical idea was to delete the machinery everyone assumed you needed to understand language, and replace it with one elegant mechanism: attention. Let's build up to what that means, one idea at a time — no math background required.

Before 2017, the best translators read a sentence one word at a time, left to right, keeping a running summary of everything seen so far. This caused two problems.

First, it's slow by design: you can't start processing word 5 until words 1–4 are done, so you can't speed up a single sentence just by adding more computers. Second, it's forgetful: by the time the machine reaches the end of a long sentence, the beginning has faded, so it loses connections between far-apart words.

The Transformer's boldest move was to dispense with both of the era's dominant designs — not just recurrent networks, but also the convolutional sequence models (like ConvS2S and ByteNet) that were the other way to parallelize. It relies on attention alone.

Attention is a mechanism where, to understand one word, the model looks at all the other words and decides how much each one matters right now. Take the sentence “The animal didn't cross the street because it was too tired.” To resolve what “it” refers to, attention lets the model glance at “animal” and “street” and figure out that “it” means the animal.

The magic: in the encoder, every word does this looking at the same time, in one step. (In the decoder, a word can only look at earlier words — more on that when we reach masking.)

How does a word decide what to look at? Each word generates three things:

• a Query — “what am I looking for?”
• a Key — “how do I describe myself?”
• a Value — “the actual information I'll hand over.”

A word's Query is compared against every word's Key to produce match scores, and those scores decide how much of each word's Value gets mixed into the final answer for that position. Separating what decides relevance (Query·Key) from what information is passed on (Value) is the single most clarifying idea in the whole paper.

The match scores can get very large when the Query/Key vectors are long. When scores are too large, the softmax that turns them into percentages becomes overconfident — it dumps almost all the weight onto a single word. The problem isn't that numbers overflow; it's that softmax saturates, and in that regime its gradients shrink toward zero, so the model can barely learn. To prevent this, every score is divided by the square root of the vector length, keeping softmax in a healthy, learnable range.

Self-attention is a sentence looking at itself — every word gathering context from its own neighbors. Cross-attention is the translation half looking back at the original: as the model writes each German word, it looks at all the English words to decide what to say next. Same machinery — the difference is only where the three vectors come from.

In self-attention, Q, K, and V all derive from the same sequence. In cross-attention, the Queries come from the decoder while the Keys and Values come from the encoder's final output.

Instead of doing attention just once, the model does it several times in parallel using different learned “lenses.” One head might track which word is the grammatical subject, another which words are related in meaning, another long-distance references like pronouns. Their findings are then combined, so a single layer can capture many kinds of relationships simultaneously.

With 8 heads and d_model = 512, each head works in a 64-dimensional subspace — splitting the dimension keeps the total compute close to a single full-size attention while giving the model multiple distinct viewpoints.

Because attention looks at all words at once, it has no built-in sense of order — “dog bites man” and “man bites dog” would look identical to it. So the model adds a special pattern of numbers to each word's embedding that encodes its position, like a subtle timestamp, letting attention recover who came before whom. (Note it's added, not glued on the side.)

The Transformer has two towers. The encoder reads the whole input sentence and builds a rich, context-aware representation of it. The decoder writes the output one word at a time, consulting both what it has written so far and the encoder's representation. Each tower stacks 6 identical layers, so understanding gets refined step by step.

Each encoder layer = multi-head self-attention + a feed-forward network. Each decoder layer adds a third sub-layer: masked self-attention, then cross-attention into the encoder, then feed-forward.

Stacking many layers can cause the signal to fade away or blow up. Two fixes work together. A residual connection adds each sub-layer's input back onto its output, giving information (and the learning signal) a shortcut straight through. Layer normalization then rescales the numbers to a consistent range at each step. Together they let a deep stack train smoothly. In the original design each sub-layer computes LayerNorm(x + Sublayer(x)) — the norm comes after the residual add.

After each attention step, every word passes through the same wide two-layer network by itself — no looking at neighbors here. It expands each word's 512-dim vector to 2048, applies a ReLU, then contracts back to 512. Attention mixes information between words; this feed-forward step then does extra per-word processing on the mixed result. Far from a minor detail, this sublayer actually holds the majority of each layer's parameters.

While learning to write a translation, the model must predict each next word using only the words before it — not by cheating and looking at the answer that comes later. Masking blocks each position in the decoder from attending to any word that comes after it, so generation stays honest and strictly left-to-right.

Concretely: in the decoder's self-attention, the pre-softmax scores for all future positions are set to −∞, so after softmax their weights are exactly 0. The encoder uses no such mask — it may attend to the entire input.

The Transformer beat the best previous systems while training dramatically faster because everything runs in parallel. On WMT 2014 English→German it reached 28.4 BLEU, improving over all prior results — including ensembles — by more than 2 BLEU. On English→French it reached 41.8 BLEU, outperforming all previously published single models, at under a quarter of the prior state-of-the-art training cost. The big model trained in 3.5 days on 8 P100 GPUs. Efficiency this good is a major reason the whole field pivoted to this design.