The Curse of Recursion: Training on Generated Data Makes Models Forget
Open accessAbstract
Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The evidence for the effectiveness of training on generated (synthetic) data is currently mixed.
“However, the evidence for the effectiveness of training on generated (synthetic) data is currently mixed.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Shumailov et al., 2023
Section 5.2 (page not available) reports that models trained on generated data could learn some of the task, but both training regimes degraded performance. That supports a qualified view of effectiveness in the settings studied; the paper does not establish that the broader body of evidence is currently mixed. (LLM, conf=0.78)
Experimental — accuracy not yet measuredRepeated synthetic training carries distribution drift risk.
“Repeated synthetic training carries distribution drift risk [27]; we mitigate this through aggressive regularization and phase-gated transition to real feedback data.”
From SuperLocalMemory: Privacy-Preserving Multi-Agent Memory with Bayesian Trust Defense Against Memory Poisoning (2026) · cites this paper as [27]
Challenges of training on synthetic data can be mitigated to some extent by training on a mixture of human-generated and synthetic data.
“or by training on a mixture of human-generated and synthetic data (Gunasekar et al., 2023; Shumailov et al., 2023; Gerstgrasser et al., 2024; Alemohammad et al., 2023).”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Shumailov et al., 2023
Iteratively training on model outputs can cause models to lose information about the original human data distribution, resulting in increasingly homogeneous and unrealistic outputs.
“One challenge is that models might lose information about the original human data distribution, such that iteratively training on model outputs results in increasingly homogeneous and unrealistic outputs (Shumailov et al., 2023).”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Shumailov et al., 2023
Model collapse is a critical challenge when training models on self-generated data, especially in multi-round iterations due to limited quality and diversity of self-generated data.
“Model collapse is one of the critical challenges when training models on self-generated data [35, 36], especially in multi-round iterations due to the limited quality and diversity of self-generated data.”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [35]
It is estimated that many supposedly human-written texts on crowdsourcing platforms are already augmented by machines.
“For instance, it is estimated that many supposedly human-written texts on crowdsourcing platforms are already augmented by machines 163 .”
From Machine culture (2023) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 13
- Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
- Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254-11263, 2019.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- Adrian Fischer, Robert E. Gaunt, and Andrey Sarantsev. The variance-gamma distribution: A review, 2023.
- Matthias Gelbrich. On a formula for the l2 wasserstein metric between measures on euclidean and hilbert spaces. Mathematische Nachrichten, 147(1):185-203, 1990.
- Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahra- mani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Cur- ran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/ e034fb6b66aacc1d48f445ddfb08da98-Paper.pdf.
- James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526, 2017.
- Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
- Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
- Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. Sponge examples: Energy-latency attacks on neural networks. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 212-231. IEEE, 2021.
- Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019. · arXiv
- Nassim Nicholas Taleb. Black swans and the domains of statistics. The American Statistician, 61(3):198-200, 2007.
Cited by
Sorted by citations
AI models collapse when trained on recursively generated data
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07566-y · Cited by 793 · see citing works
Self-citationArtificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707 · see citing works
Artificial intelligence risk management framework :
National Institute of Standards and Technology (US) (2024) · NIST Trustworthy and Responsible AI · doi:10.6028/nist.ai.600-1 · Cited by 274 · see citing works
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136 · see citing works
“The most common mechanism is to use text formatting such”
Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80 · see citing works
“One challenge is that models might lose information about the original human data distribution, such that iteratively training on model outputs results in increasingly homogeneous and unrealistic outputs (Shumailov et al., 2023).”
BackgroundInfluentialSelf-consuming generative models go MAD
Casco-Rodriguez, J., Alemohammad, S., Luzi, L., Imtiaz, A., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. (2023) · LatinX in AI at Neural Information Processing Systems Conference 2023 · doi:10.52591/lxai202312101 · Cited by 83 · see citing works
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10903-2 · Cited by 94 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Veselovsky, V., Ribeiro, M. H., & West, R. (2023) · arXiv · doi:10.48550/arxiv.2306.07899 · Cited by 73 · see citing works
“This may degrade the performance of “recursive” LLMs, i.e., those trained on synthetically generated data (Shumailov et al., 2023), and magnify the impact of the values and ideologies encoded by these models (Santurkar et al., 2023; Liu et al., 2022).”
Hoskins, A. (2024) · Memory, Mind & Media · doi:10.1017/mem.2024.16 · Cited by 64 · see citing works
“They argue, therefore, that ‘ the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet ’ (Shumailov et al. 2023, 1).”
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57 · see citing works
Ashkinaze, J., Mendelsohn, J., Qiwei, L., Budak, C., & Gilbert, E. (2025) · Proceedings of the ACM Collective Intelligence Conference · doi:10.1145/3715928.3737481 · Cited by 33 · see citing works
Human innovation and the creative agency of the world in the age of generative AI
Peschl, M. F. (2024) · Possibility Studies & Society · doi:10.1177/27538699241238049 · Cited by 23 · see citing works
“Infiltrating sound and creative human knowledge into the global knowledge base : How can we escape both model collapse through data poisoning and recursive application of cognitive technologies (Brinkmann et al., 2023; Shumailov et al., 2023) and the decline of human creativity?”
BackgroundMethodInfluentialWatermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B. (2023) · arXiv · doi:10.48550/arxiv.2311.04378 · Cited by 12 · see citing works
Leveraging Large Language Models to Detect npm Malicious Packages
Zahan, N., Burckhardt, P., Lysenko, M., Aboukhadijeh, F., & Williams, L. (2024) · arXiv · doi:10.48550/arxiv.2403.12196 · Cited by 3 · see citing works
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3 · see citing works
“Feedback can also amplify bias (Taori & Hashimoto, 2022) or eliminate the tails of the distribution (Shumailov et al., 2023) if a model is trained on its own outputs.”
BackgroundEmpowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
Zhai, Y., Liu, H., Zhang, Z., Lin, T., Xu, K., Yang, C., Feng, D., Ding, B., & Wang, H. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3729965 · Cited by 3 · see citing works
Testing the limits of large language models in debating humans
Flamino, J., Modi, M. S., Szymanski, B. K., Cross, B., & Mikolajczyk, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-98378-1 · Cited by 1 · see citing works
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., & Choi, Y. (2025) · arXiv · doi:10.48550/arxiv.2510.22954 · Cited by 1 · see citing works
- Authors
- Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R.
- Affiliations
- University of Oxford (GB)
- University of Cambridge (GB)
- Imperial College London (GB)
- University of Toronto (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.17493
- OpenAlex
- W4378765257
- Cited by
- 151
- References
- 0
- Citations by year
- 2026: 17 · 2025: 36 · 2024: 71 · 2023: 27
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- world models, counter-worlds, grid world, reproducibility, audit logs, cross-domain transfer, hash chains, ablation study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
