Skip to content
kwharrison13.comEssays, books and notes by Kyle Harrison

Attention Is All You Need

From kwharrison13.com · Library of Alexandria
Attention Is All You Need
AuthorAshish Vaswani et al.
Published2017
FormatResearch paper
OriginalRead the original

TL;DR — The 2017 Google paper that introduced the Transformer, a network built only on attention with no recurrence, which became the architecture behind modern large language models.

Before this paper, the best models for sequence tasks such as translation were recurrent or convolutional encoder–decoders with an attention mechanism attached. The authors drop the recurrence entirely and build both encoder and decoder from stacked layers of self-attention and feed-forward networks. The paper defines scaled dot-product attention, multi-head attention and sinusoidal positional encodings, and its “Why Self-Attention” section argues the design is more parallel and connects distant positions in fewer steps than recurrent layers.

On the WMT 2014 benchmarks the big model scored 28.4 BLEU on English-to-German, more than 2 points above the previous best results including ensembles, and 41.8 BLEU on English-to-French after 3.5 days of training on eight P100 GPUs, a fraction of earlier models’ training cost. Ablations vary the number of heads and the model size, and a final experiment applies the model to English constituency parsing.

The eight authors, from Google Brain, Google Research and the University of Toronto, are listed in random order with a footnote crediting each one’s contribution. The paper was presented at NIPS 2017. This copy is arXiv version 7, revised August 2023.

What it is: A peer-reviewed conference paper and one of the most cited and consequential papers in machine learning. It is short and technical, and it is the primary source for the Transformer.

Where this came from

15 pages. A copy is archived locally against link rot; the header links the original source.

Curated by Kyle Harrison. Organized by Claude. Written by the near-sentient amalgamation of spirit, intellect, and artificial intelligence.