Broken Neural Scaling Laws
| Author | Ethan Caballero, Kshitij Gupta, Irina Rish and David Krueger |
|---|---|
| Published | 2023 |
| Format | Research paper |
| Original | Read the original |
TL;DR — An ICLR 2023 paper proposing one curve, a smoothly broken power law, that fits and extrapolates how AI performance changes with scale, bends included.
Ethan Caballero, Kshitij Gupta and Irina Rish (Mila) and David Krueger (Cambridge) start from a problem with existing neural scaling laws: they assume a single power law, so they cannot represent double descent (performance getting worse before it gets better) or the sudden sharp improvements seen in tasks like arithmetic. Their broken neural scaling law (BNSL) joins several roughly straight segments on a log-log plot with smooth “breaks,” each with its own location and sharpness.
The paper fits the form across a long list of settings, including vision, language, audio, video, diffusion models, reinforcement learning, robotics, adversarial robustness, RLHF alignment, sparsity, quantization, distillation, molecules and math word problems, with scale measured as compute, parameters, data or training steps. On the vision benchmark of Alabdulmohsin et al. (2022), BNSL gives the most accurate extrapolation on about 69% of tasks, against about 14% for the next-best form. Dozens of appendix sections show further extrapolations.
It closes on the limits of prediction: using 4-digit addition, the authors show that a sharp break can only be extrapolated from data that already reaches close to it, and that a sharp break lying beyond the fitted range cannot currently be predicted at all. Code is released on GitHub.
What it is: A peer-reviewed conference paper (ICLR 2023; this copy is arXiv version 17, July 2023). The functional form describes scaling curves well but does not explain why the breaks occur.
Where this came from
- Will Manidis’s “most underrated PDF” thread, July 2023 — routing record: The Most Underrated PDF on the Internet (July 2023)
38 pages. A copy is archived locally against link rot; the header links the original source.