Attention Is All You Need
| Author | Ashish Vaswani et al. |
|---|---|
| Published | 2017 |
| Format | Research paper |
| Original | Read the original |
TL;DR — The 2017 Google paper that introduced the Transformer, a network built only on attention with no recurrence, which became the architecture behind modern large language models.
Before this paper, the best models for sequence tasks such as translation were recurrent or convolutional encoder–decoders with an attention mechanism attached. The authors drop the recurrence entirely and build both encoder and decoder from stacked layers of self-attention and feed-forward networks. The paper defines scaled dot-product attention, multi-head attention and sinusoidal positional encodings, and its “Why Self-Attention” section argues the design is more parallel and connects distant positions in fewer steps than recurrent layers.
On the WMT 2014 benchmarks the big model scored 28.4 BLEU on English-to-German, more than 2 points above the previous best results including ensembles, and 41.8 BLEU on English-to-French after 3.5 days of training on eight P100 GPUs, a fraction of earlier models’ training cost. Ablations vary the number of heads and the model size, and a final experiment applies the model to English constituency parsing.
The eight authors, from Google Brain, Google Research and the University of Toronto, are listed in random order with a footnote crediting each one’s contribution. The paper was presented at NIPS 2017. This copy is arXiv version 7, revised August 2023.
What it is: A peer-reviewed conference paper and one of the most cited and consequential papers in machine learning. It is short and technical, and it is the primary source for the Transformer.
Where this came from
- Will Manidis’s “most underrated PDF” thread, July 2023 — routing record: The Most Underrated PDF on the Internet (July 2023)
15 pages. A copy is archived locally against link rot; the header links the original source.