Model Collapse
A reader's summary of the phenomenon formalized in 2023–2024: what happens, mathematically, when generative models are trained on data increasingly produced by earlier generations of models like themselves. Editorial synthesis of the papers, not a claim about any specific product.
The idea at a glance
Model collapse describes a degenerative feedback loop: a generative model trained partly or wholly on the outputs of previous models (rather than on original human-generated data) tends, across successive generations, to lose information about the tails of the true data distribution first, and then to converge toward a narrower, more repetitive, lower-variance version of that distribution. Colloquially it has been nicknamed “Habsburg AI,” after the inbreeding of the Spanish royal house — models trained on their own descendants' output, with the genetic defects standing in for statistical ones.
Origins
The term and its formal treatment come chiefly from a 2023 preprint by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson, “The Curse of Recursion: Training on Generated Data Makes Models Forget,” later revised and published in Nature in 2024 as “AI models collapse when trained on recursively generated data.” The paper demonstrated the effect across language models, variational autoencoders, and Gaussian mixture models, arguing the mechanism is general to generative modeling rather than specific to any one architecture.
History and context
The concern predates the formal papers: practitioners had informally worried since the GPT-3 era (2020–2021) about a future internet increasingly populated by AI-generated text, and about what happens when the next generation of models is trained on a web crawl that includes the previous generation's output. The 2023–2024 papers gave the worry a name, a mechanism, and controlled experiments rather than leaving it as speculation — timed to arrive just as generative text and image tools began producing a measurable share of newly published web content, which is the exact precondition the theory requires.
The mechanism
The core mechanism: recursive training on synthetic data
Model collapse names what happens when a generative model is trained on data increasingly produced by earlier generations of similar models rather than by the original human-generated distribution — each generation inherits and compounds the errors and narrowed variance of the one before it.
Early collapse: the tails go first
Rare events and minority patterns in the training data — the long tail of any real distribution — are the first casualties, since a finite sample from a model's own outputs under-represents low-probability content relative to the true distribution, and each re-training pass compounds the under-representation.
Late collapse: convergence to a low-variance point
Continued recursive training pushes the model toward outputs that increasingly resemble a narrow, repetitive, low-variance approximation of the original distribution — in the limit, textual or visual mush that has lost most of the diversity and detail present in the human-generated data it started from.
Two compounding error sources
The Shumailov et al. (2023/2024, published in Nature) analysis identifies statistical approximation error (finite samples never perfectly represent a distribution) and functional approximation error (models are imperfect estimators even with infinite data) as the two sources that compound across generations, on top of whatever error the training and optimization process itself introduces.
It is a property of the training loop, not of any one model
Collapse is not a claim that any specific model is bad; it is a claim about what happens to a *lineage* of models across generations when later generations' training data is contaminated by earlier generations' synthetic outputs, without a compensating supply of fresh, human-generated data.
Mitigations exist and are actively used
Provenance tracking (watermarking, metadata tagging of synthetic content), preserving access to earlier human-generated data snapshots, and curated mixing ratios between synthetic and organic data are the main proposed defenses — none of which fully solves the problem, but all of which slow it, and all of which are now standard practice at frontier labs.
Critique
- Real deployments are not pure recursion. The cleanest experimental demonstrations train exclusively on prior generations' synthetic output; frontier labs in practice mix synthetic data deliberately (for distillation and curriculum design — see distillation) with large, curated stores of human-generated and filtered data, which is a meaningfully different and less collapse-prone regime than the papers' worst case.
- Synthetic data is not always harmful. Curated, verified synthetic data — model outputs checked against ground truth, as in math and code generation with unit tests, or reinforcement-learning self-play — has been shown toimprove models rather than degrade them, since verification removes the errors that recursion would otherwise compound. The collapse result applies specifically to unfiltered, uncurated recursive training.
- Scale and mixing ratios matter more than the headline. Follow-up work has shown that collapse can be slowed or avoided entirely by keeping even a modest, constant proportion of fresh human-generated data in each training generation — the dramatic total-collapse scenarios require a near-total absence of fresh data, an assumption stronger than what most real training pipelines make.
- Detection is getting harder as the effect matters more. The mitigations that would most reliably prevent collapse — reliable provenance tagging of AI-generated content across the open web — are exactly the capability that critics of Dead Internet Theory note is weakest today, creating tension between how serious the problem is claimed to be and how tractable its main proposed fix actually is.
Impact
Model collapse gave the AI research community a formal, citable reason to prioritize data provenance, licensing deals for human-generated corpora, and synthetic-data curation pipelines — concerns that previously circulated as informal worry. It has become a standard citation in discussions of AI training-data scarcity alongside the broader “data wall” debate: estimates that the supply of unused, high-quality human-generated text is finite and may already be substantially exhausted by frontier models, making the quality of whatever synthetic data replaces it a first-order question rather than an afterthought.
Notable engagements
- Shumailov et al., Nature (2024) — “AI models collapse when trained on recursively generated data,” the peer-reviewed version of the founding result.
- “Habsburg AI” (informal coinage, 2023) — the nickname that carried the concept from researchers to general tech commentary.
- Data-licensing deals (2023–2025) — publisher and platform licensing agreements with major AI labs are frequently framed in coverage as a direct response to the scarcity of uncontaminated human-generated training data.
- Curated-synthetic-data research (ongoing) — verified self-play and rejection-sampling approaches in math and code domains are the most cited counter-examples showing synthetic data can improve rather than degrade a model.