Scaling Laws for Neural Language Models

A reader's summary of the 2020 OpenAI paper that made scaling a science — its history, authors, main ideas, critiques, and the debate it started. Editorial synthesis, not a substitute for the paper.

The paper at a glance

Published in January 2020, the paper measured how the test loss of Transformer language models depends on model size, dataset size, and training compute, and found each relationship to be a clean power law. The practical upshot was blunt: within the range tested, there was no evidence that making the model bigger, feeding it more data, or training it longer would stop helping. That single empirical result — more than any specific architectural trick — is what convinced a generation of labs to spend hundreds of millions of dollars on the bet that “bigger” was not a slogan but a measurable, fittable curve.

The authors

Jared Kaplan, a Johns Hopkins theoretical physicist consulting for OpenAI, led the work with Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei. The physicist's eye shows: the paper reads like an exercise in finding the empirical law that governs a system, the way one might fit a gas law, rather than a machine-learning ablation study. Several authors — Brown and Radford among them — moved directly from this paper into building GPT-3 later that same year; Amodei left in 2021 to co-found Anthropic on a related but distinct set of bets about scale and safety.

History and context

The idea that deep learning performance scales predictably with data and model size had been observed before — most notably by Hestness et al. at Baidu in 2017, across vision, speech and language tasks — but it remained a curiosity rather than a strategy. Kaplan et al. gave it the specificity a research lab could act on: exact exponents, an irreducible loss term, and a rule for how to split a fixed compute budget between model size and data. Within months, OpenAI trained GPT-3 at 175 billion parameters largely as a test of the curve, and the results — few-shot capabilities nobody had explicitly trained for — reframed scaling from an efficiency question into the field's central research program.

Main ideas

Critique

• The Chinchilla correction. Hoffmann et al. (DeepMind, 2022) showed the original compute-optimal recipe was skewed toward parameters and starved of data: their 70-billion parameter Chinchilla, trained on far more tokens, beat DeepMind's own 280-billion parameter Gopher. Most large models trained under the 2020 recipe, GPT-3 included, were quietly undertrained.
• Loss is not capability. The law describes next-token prediction loss, not downstream task performance. Wei et al. (2022) catalogued abilities that seemed to “emerge” abruptly at scale rather than improve smoothly; Schaeffer et al. (2023) countered that most of this is a mirage created by discontinuous metrics, not a discontinuity in the underlying loss. The argument is unresolved.
• The data wall. The power law wants ever more tokens, and the internet's stock of high-quality text is finite; Epoch AI's estimates put full depletion sometime in the second half of the 2020s. That constraint, more than any algorithmic limit, is what pushed the frontier toward synthetic data and toward scaling test-time compute instead of pretraining compute.
• Extrapolation is a bet, not a theorem. A power law fit across seven orders of magnitude is a strong empirical regularity, not a law of physics. The 2024–2025 chatter about frontier models “hitting a wall” — smaller headline gains between successive flagship releases than the curve implied — is best read as evidence that the extrapolation, not the paper, had limits.

Impact

This paper is the empirical backbone of the scaling race: the capital expenditure on GPU clusters and data centers, the trillion-dollar buildout in AI infrastructure, and the industry's working assumption that the next order of magnitude buys the next capability. It is routinely paired with Rich Sutton's 2019 essay “The Bitter Lesson” — the argument that general methods leveraging computation beat hand-crafted human knowledge over the long run — as the doctrine's two founding texts, one philosophical, one measured. When pretraining gains began to look less dramatic relative to their cost, the same logic simply moved: reasoning models that scale test-time compute (OpenAI's o1 and its successors) are a bet that a new axis obeys a similar curve, not an abandonment of the original one. It also supplies the demand side of the argument made under Jevons Paradox: a predictable return to more compute is exactly what gives cheaper inference somewhere new to go, rather than simply shrinking total spend.

Notable engagements

• Hoffmann et al., “Training Compute-Optimal Large Language Models” (2022) — the direct successor and correction, now the more commonly cited allocation rule.
• Richard Sutton, “The Bitter Lesson” (2019) — the philosophical companion piece, cited by scaling's advocates and critics alike.
• Wei et al. vs. Schaeffer et al. — the emergent- abilities debate, run largely as a rebuttal-and-reply in the open literature.
• Epoch AI — the research group that has most publicly tracked the scaling frontier and the data-wall timeline the paper's logic implies.