Attention Is All You Need
A reader's summary of the 2017 Transformer paper — its history, authors, main ideas, critiques, and afterlife. Editorial synthesis, not a substitute for the paper.
The paper at a glance
Eight Google researchers proposed a sequence model built from attention alone — no recurrence, no convolutions — and named it the Transformer. Trained on machine translation, it set new BLEU records on English–German and English–French at a fraction of the previous training cost (3.5 days on eight GPUs), and its architecture turned out to be the substrate of essentially everything since: BERT, GPT and its descendants, vision Transformers, AlphaFold. At its mathematical core sit a few lines of linear algebra — three learned projections and a softmax — which is part of why it belongs in this section: rarely has so much rested on so small a formula.
The authors
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin — listed with a footnote unusual for the field: “equal contribution, listing order is random.” Gomez was a twenty-year-old intern; the title riffs on the Beatles. Just as unusual is the epilogue: within a few years all eight had left Google, founding or leading Cohere (Gomez), Character.AI (Shazeer, since re-acquired into Google), Essential AI (Vaswani, Parmar), Sakana AI (Jones), NEAR (Polosukhin) and Inceptive (Uszkoreit) — a diaspora that itself became a story about where the value of research accrues.
History and context
By 2016, sequence-to-sequence learning ran on recurrent networks: LSTMs read a sentence token by token into a vector, and attention — introduced by Bahdanau et al. in 2014 — had been bolted on as a fix for the bottleneck. Google Translate's own GNMT was an eight-layer LSTM system. Recurrence had a structural flaw nothing could patch: each token waits for the previous one, so training cannot parallelize along the sequence. Convolutional challengers (ByteNet, ConvS2S) attacked this but paid with depth for long-range dependencies. The Transformer's bet — championed inside Google Brain over months of tinkering later recounted in Wired's oral history — was that the bolt-on was the machine: drop recurrence entirely and keep only attention. The paper appeared at NeurIPS 2017 — without winning any award there.
Main ideas
Self-attention
Every token computes a query, a key and a value; each position attends to all others via softmax(QKᵀ/√d)V. Information routes by content, not by adjacency — any two words connect in one step, however far apart, where recurrent networks needed a relay chain.
Multi-head attention
Run several attentions in parallel over projected subspaces, then concatenate. Different heads specialize — syntax here, coreference there — giving the model many simultaneous notions of 'relevant'.
Positional encodings
Attention is blind to order — a set operation. Sinusoidal position signals added to embeddings restore sequence, cheaply and without learned recurrence; successors (like rotary embeddings) refined but kept the idea.
The stack
Six identical blocks of attention + feed-forward layers, wrapped in residual connections and layer normalization; a causal mask keeps the decoder from peeking at the future. The recipe proved so stable that today's frontier models are still recognizably this diagram.
Parallelism as the real product
Recurrence processes tokens one after another; attention processes them all at once. Constant sequential depth is what let GPUs and TPUs saturate — making compute, not architecture, the binding constraint. Scaling laws, and everything they bought, follow from this property.
Engineering care
The unglamorous half of the result: the √d scaling that keeps softmax gradients alive, warmup learning-rate schedules, label smoothing, dropout, byte-pair encoding. The paper is also a masterclass in ablation — each trick measured, none mystical.
Critique
- The quadratic wall. Attending everyone to everyone costs O(n²) in sequence length. Long documents, code bases and video strain it — spawning a decade of efficient-attention variants, the FlashAttention systems line, and state-space challengers (Mamba) that revive recurrence; so far the Transformer has absorbed rather than lost these fights.
- Weak inductive bias. The architecture assumes almost nothing about structure, which is why it generalizes across modalities — and why it is data- and compute-hungry, losing to convolutions and trees on small datasets.
- Is attention really all you need? Later analysis located much of a model's stored knowledge in the feed-forward layers, and interpretability work (“Attention is not Explanation,” Jain & Wallace 2019) cautioned against reading attention weights as reasons. The title was marketing as much as theorem.
- Novelty in synthesis. Attention (Bahdanau, Luong) and self-attention precursors (Cheng, Parikh, Lin, 2016–17) existed; the contribution was the ruthless simplification, the recipe, and the evidence — which is a real contribution, but a different one than the folklore suggests.
Impact
Arguably no paper this century has moved more of the world per page. BERT (2018) took the encoder and rewrote search; GPT (2018–2020) took the decoder and, scaled by the laws the architecture's parallelism made measurable, produced ChatGPT and the current AI wave — including the model writing this summary. Vision Transformers showed images are “worth 16×16 words”; AlphaFold 2's Evoformer carried attention into biology's biggest result. Citations passed one hundred thousand within seven years, making it one of the most-cited scientific papers of the century — while the architecture itself has changed remarkably little: normalizations moved, positions became rotary, experts mixed (Shazeer again), but a 2017 author would recognize today's frontier model at a glance. For the operational consequences of that wave, see the writings on AI in factories.
Notable engagements
- Wired's oral history (2024) — Steven Levy's “8 Google Employees Invented Modern AI” reconstructed the paper's making and its authors' exodus.
- Nvidia GTC 2024 — Jensen Huang hosted seven of the eight authors on stage: the hardware industry saluting the workload that made it the world's most valuable.
- Andrej Karpathy — the field's best-known teacher popularized the view of the Transformer as a general-purpose differentiable computer, remarkably stable since 2017.
- The challengers — Albert Gu and Tri Dao's state-space models (Mamba) are the most serious documented attempt to answer the title in the negative; the verdict, so far, is hybrid architectures.