Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; 2 of the 7 sources its claims cite were checked for support.

AI models collapse when trained on recursively generated data

Open access

Abstract

) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as 'model collapse' and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.

Overview

CoChat Summary

TL;DR

Indiscriminate use of model-generated content in training causes “model collapse,” a degenerative process in which generative models forget the true data distribution, beginning with the tails and eventually converging to a low-variance point estimate. The authors demonstrate this theoretically and empirically for Gaussian mixture models, variational autoencoders, and large language models.

Key findings

  • Model collapse occurs across generative model families: in discrete distributions with exact approximation, the process converges to a delta function with probability 1; in the Gaussian setting, Theorem 3.1 states that the Wasserstein-2 distance between the true distribution and its approximation diverges arbitrarily far while variance collapses to zero as generations increase.
  • In LLM experiments, an OPT-125m model fine-tuned on wikitext2 reaches 34 mean perplexity from a zero-shot baseline of 115. With five epochs and no original training data, performance degrades “from 20 to 28 perplexity points”; with ten epochs and 10% of original training data preserved, there is “only minor degradation of performance.”
  • Generated data over generations contain more sequences the original model would produce with higher likelihood, but also much longer tails of sequences the original model would never produce, indicating accumulated errors.
  • An ablation with a repeating penalty of 2.0 causes models to produce lower-score continuations to avoid repeats, and “causes the perplexity to double compared with the original”; models remain as susceptible to model collapse, if not more.

Methods

  • Theoretical analysis of “learning with generational data”: a discrete distribution case with exact approximation, analyzed as a Markov chain, and a multidimensional Gaussian approximation using unbiased sample mean and variance estimators, with Wasserstein-2 distance as the divergence measure.
  • LLM experiments fine-tune the OPT-125m causal language model from Meta via Hugging Face on wikitext2. Data generation uses five-way beam search; training sequences are blocked to 64 tokens, and for each token sequence the model predicts the next 64 tokens, producing an artificial dataset of the same size. Each experiment is run five times with different randomness seeds, and the best-performing model on the original task is used as the base for subsequent generations.
  • Two LLM settings are tested: five epochs with no original training data retained, and ten epochs with a random 10% of original data points sampled for each new generation. VAEs and GMMs are covered in the Supplementary Materials.

Limitations

  • The paper states that theoretical claims are mostly presented in the limit of generations going to infinity, whereas experiments use a finite, usually small, number of generations; complete collapse may or may not occur even after a small number of steps, and convergence rates are highly related to the specifics of the problem.
  • The paper states that larger language models in non-fine-tuning settings were not run because training a single moderately large model produces twice the American lifetime’s worth of CO2; the LLM experiments are a fine-tuning proof of concept, and even those took weeks to run.
  • Evident from the text: the LLM evidence is limited to OPT-125m, wikitext2, five runs, and fine-tuning; no full retraining from scratch is tested, and VAE/GMM details are relegated to the Supplementary Materials.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41586-024-07566-y

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 The high-level description of the feedback mechanism in the learning process. a, Model collapse refers to a degenerative learning process in which models start forgetting improbable events over time, as the model becomes poisoned with its own projection of reality. Here data are assumed to be human-curated and start off clean; then model 0 is trained and data are sampled from it; at step n, data are added to the overall data from step n − 1 and this combination is used to train model n. Data obtained with Monte Carlo sampling should ideally be statistically close to the original, provided that fitting and sampling procedures are perfect. This process depicts what happens in real life with the Internet: model-generated data become pervasive. b,c, Performance of OPT-125m models of different generations evaluated using the original wikitext2 test dataset. Shown on the left are the histograms of perplexities of each individual data training sequence produced by different generations as evaluated by the very first model trained with the real data. Over the generations, models tend to produce samples that the original model trained with real data is more likely to produce. At the same time, a much longer tail appears for later generations. Later generations start producing samples that would never be produced by the original model, that is, they start misperceiving reality based on errors introduced by their ancestors. The same plots are shown in 3D in the Supplementary Materials. On the right, average perplexity and its standard deviation are shown for each independent run. The x axis refers to the generation of the model. ‘Real’ refers to the ‘model 0’ trained on the original wikitext2 dataset; model 1 was trained on the data produced by model 0, model 2 was trained on data produced by model 1 and so on, with all generated datasets equal in size. We find that models trained on generated data are able to learn some of the original task, but with errors, as seen from the increase in perplexity.
Fig. 1 The high-level description of the feedback mechanism in the learning process. a, Model collapse refers to a degenerative learning process in which models start forgetting improbable events over time, as the model becomes poisoned with its own projection of reality. Here data are assumed to be human-curated and start off clean; then model 0 is trained and data are sampled from it; at step n, data are added to the overall data from step n − 1 and this combination is used to train model n. Data obtained with Monte Carlo sampling should ideally be statistically close to the original, provided that fitting and sampling procedures are perfect. This process depicts what happens in real life with the Internet: model-generated data become pervasive. b,c, Performance of OPT-125m models of different generations evaluated using the original wikitext2 test dataset. Shown on the left are the histograms of perplexities of each individual data training sequence produced by different generations as evaluated by the very first model trained with the real data. Over the generations, models tend to produce samples that the original model trained with real data is more likely to produce. At the same time, a much longer tail appears for later generations. Later generations start producing samples that would never be produced by the original model, that is, they start misperceiving reality based on errors introduced by their ancestors. The same plots are shown in 3D in the Supplementary Materials. On the right, average perplexity and its standard deviation are shown for each independent run. The x axis refers to the generation of the model. ‘Real’ refers to the ‘model 0’ trained on the original wikitext2 dataset; model 1 was trained on the data produced by model 0, model 2 was trained on data produced by model 1 and so on, with all generated datasets equal in size. We find that models trained on generated data are able to learn some of the original task, but with errors, as seen from the increase in perplexity.CC BY · PubMed Central

Verification

Claims the paper makes

1 Valid · 0 Not valid · 3 Not checked · 40 Own results and interpretations

11 more claims

Create a free account to see all 44 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 22
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 10

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

19 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 11 references

Research works printed but not linked: 7

  1. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
  2. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. in Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds Burstein, J., Doran, C. & Solorio, T.) 4171–4186 (Association for Computational Linguistics, 2019).
  3. Carlini, N. & Terzis, A. in Proc. Tenth International Conference on Learning Representations (ICLR, 2022).
  4. Carlini, N. et al. in Proc. 2024 IEEE Symposium on Security and Privacy (SP) 179 (IEEE, 2024).
  5. Mousavi-Hosseini, A., Park, S., Girotti, M., Mitliagkas, I. & Erdogdu, M. A. in Proc. Eleventh International Conference on Learning Representations (ICLR, 2023).
  6. Gu, Y., Dong, L., Wei, F. & Huang, M. in Proc. Twelfth International Conference on Learning Representations (ICLR, 2024).
  7. Strubell, E., Ganesh, A. & McCallum, A. in Proc. 57th Annual Meeting of the Association for Computational Linguistics (eds Korhonen, A., Traum, D. & Màrquez, L.) 3645–3650 (Association for Computational Linguistics, 2019).

Cited by

See all 793

Metadata

Authors
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y.
Affiliations
  • University of Oxford (GB)
  • University of Cambridge (GB)
  • Imperial College London (GB)
  • University of Toronto (CA)
Corresponding author
  • Shumailov, Ilia
  • Shumaylov, Zakhar
  • Gal, Yarin
Venue
Nature
Published
2024-07-24
Volume
631
Issue
8022
Pages
755-759
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W4400949264
Cited by
793
References
11
Citations by year
2026: 407 · 2025: 330 · 2024: 56
FWCI
116.53
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
model collapse, large language models, self-training, generative AI training data, variational autoencoder, Gaussian mixture model, distribution shift, web scraping, ChatGPT, generative models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AI models collapse when trained on recursively generated data. https://app.cochat.ai/papers/10.1038/s41586-024-07566-y
The paper
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature. https://doi.org/10.1038/s41586-024-07566-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.