BERT

In 2018, getting a machine to do a new language task usually meant building a new model for it. BERT changed that. It's a single model pre-trained on a mountain of raw text, that you can then fine-tune — with one small added layer — to state-of-the-art performance on question answering, sentiment, inference, and much more.

Two ideas made it work: reading text bidirectionally (both left and right context at once), and a self-supervised training trick that makes bidirectionality possible. Let's build up to both.

The strong language models before BERT (like GPT) were unidirectional: they read left to right, so when representing a word they could only use the words that came before it. That's a real limitation — to understand a word, the words after it often matter just as much.

Take “bank” in “I made a deposit at the ____ ” versus “I sat on the river ____ ”. You need the right-hand context to know which “bank” is meant. A left-to-right model reading up to the blank is missing exactly that.

Here's the catch: you can't just train a deep bidirectional model the normal way. If every word could see every other word (including itself), predicting the next word becomes trivial — the model would just “see” the answer. BERT's solution is the Masked Language Model (MLM).

Randomly hide 15% of the words in a sentence, then train the model to guess them from all the surrounding context. Because the target words are hidden, the model can safely look in both directions. There's one subtlety: the special [MASK] token used in training never appears when you later fine-tune. So of the chosen 15%, the model replaces them with [MASK] 80% of the time, a random word 10% of the time, and leaves them unchanged 10% of the time — keeping training and real use aligned.

Many tasks — question answering, natural-language inference — hinge on the relationship between two sentences, which word-level prediction alone doesn't capture. So BERT adds a second, simple pre-training task: Next Sentence Prediction (NSP).

Give the model two sentences, A and B. Half the time B really is the sentence that followed A (labeled IsNext); half the time B is a random sentence from the corpus (NotNext). The model learns to tell which. It's trivially easy to generate this data from any text, and the final model reaches 97–98% accuracy on it.

BERT packs its input as a sequence that can hold one or two sentences. A special [CLS] token starts every sequence — its final hidden state becomes the summary used for classification. A [SEP] token separates the two sentences.

Each token's input is the sum of three embeddings: the token itself (from a 30,000-word WordPiece vocabulary), a segment embedding (is this sentence A or B?), and a position embedding (where in the sequence). Adding them lets one vector carry what the word is, which sentence it's in, and where it sits.

This is the paradigm BERT popularized. Pre-training is expensive and done once: the model learns MLM + NSP on 3.3 billion words of unlabeled text (BooksCorpus + Wikipedia), absorbing general language understanding. Fine-tuning is cheap and done per task: start from the pre-trained weights, add a single output layer, and train briefly on the labeled task data.

The beauty is how little changes between tasks — the same pre-trained model becomes a question-answerer, a sentiment classifier, or an entailment judge just by swapping the tiny top layer and fine-tuning.

With the same pre-trained model and minimal per-task changes, BERT set new state-of-the-art results across the board. It pushed the GLUE score to 80.5% (a 7.7-point jump), MultiNLI to 86.7%, SQuAD v1.1 question-answering to a 93.2 test F1, and the harder SQuAD v2.0 to 83.1 F1. Larger was better: BERT-Large (340M parameters) consistently beat BERT-Base (110M), even on small datasets.