Deep Learning Foundations
The building blocks — the ideas and training tricks that make deep neural networks work.
Attention Is All You Need
The Transformer — the architecture behind modern AI
Before this paper, the best translation systems read sentences one word at a time, carrying a running memory forward (recurrent networks). That was slow — you can't process word 5 until word 4 is done — and forgetful over long sentences. The 2017 Transformer threw out that sequential machinery entirely and replaced it with attention: a mechanism that lets every word look directly at every other word and decide which ones matter. Because nothing waits its turn, the model trains fully in parallel on GPUs, captures long-range relationships in a single step, and set new translation records while training in a fraction of the time. This design became the foundation of GPT, BERT, Claude, and essentially all modern AI.
BERT
Pre-training deep bidirectional transformers for language understanding
Before BERT, the strongest language models read text in one direction — left to right — so when understanding a word they could only use the words before it. BERT's insight was to read in both directions at once, letting every word draw on its full context, left and right. To train a model that way, the authors invented a clever fill-in-the-blank game: randomly hide 15% of the words and have the model guess them from everything around them (the Masked Language Model), plus a second task of judging whether one sentence naturally follows another. After this self-supervised “pre-training” on 3.3 billion words of books and Wikipedia — no human labels needed — the same model can be lightly “fine-tuned” for almost any language task by adding just one small output layer. This one recipe set new records on eleven benchmarks at once, pushing the GLUE score to 80.5% and SQuAD question-answering to a 93.2 F1.