Large language models reduce public knowledge sharing on online Q&A platforms
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) are a potential substitute for human-generated data and knowledge resources. This substitution, however, can present a significant problem for the training data needed to develop future models if it leads to a reduction of human-generated content. In this work, we document a reduction in activity on Stack Overflow coinciding with the release of ChatGPT, a popular LLM. To test whether this reduction in activity is specific to the introduction of this LLM, we use counterfactuals involving similar human-generated knowledge resources that should not be affected by the introduction of ChatGPT to such extent. Within 6 months of ChatGPT’s release, activity on Stack Overflow decreased by 25% relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, and to similar forums for mathematics, where ChatGPT is less capable. We interpret this estimate as a lower bound of the true impact of ChatGPT on Stack Overflow. The decline is larger for posts related to the most widely used programming languages. We find no significant change in post quality, measured by peer feedback, and observe similar decreases in content creation by more and less experienced users alike. Thus, LLMs are not only displacing duplicate, low-quality, or beginner-level content. Our findings suggest that the rapid adoption of LLMs reduces the production of public data needed to train them, with significant consequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In programming, the most affected clusters are related to JavaScript, HTML, and CSS, which are also among the most affected programming languages in collaborative programming platforms such as Stack Overflow (del Rio-Chanona et al., 2024).”
From Winners and losers of generative AI: Early Evidence of Shifts in Freelancer Demand (2025) · cites this paper as del Rio-Chanona et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
“A Model of Growth: Through Creative Destruction”
Aghion, P., & Howitt, P. (1992) · Econometrica · doi:10.2307/2951599 · Cited by 7,529
Competing Technologies, Increasing Returns, and Lock-In by Historical Events
Arthur, W. B. (1989) · The Economic Journal · doi:10.2307/2234208 · Cited by 7,489
Automation and New Tasks: How Technology Displaces and Reinstates Labor
Acemoglu, D., & Restrepo, P. (2019) · Journal of Economic Perspectives · doi:10.1257/jep.33.2.3 · Cited by 2,612
Some Simple Economics of Open Source
Lerner, J., & Tirole, J. (2002) · The Journal of Industrial Economics · doi:10.1111/1467-6451.00174 · Cited by 2,355
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150 · cited 2×
“While early evidence shows that these models enhance productivity especially among new and inexperienced workers (Noy and Zhang, 2023; Brynjolfsson et al., 2023), there are other ways in which they may contribute to inequality between people and firms (Rock, 2019), for instance via potential negative side effects of a…”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
AI models collapse when trained on recursively generated data
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07566-y · Cited by 793
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449
Welcome to the Era of ChatGPT et al.: The Prospects of Large Language Models
Teubner, T., Flath, C. M., Weinhardt, C., van der Aalst, W., & Hinz, O. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00795-x · Cited by 384 · cited 2×
“The remarkable effectiveness of ChatGPT is reflected in its quick adoption (Teubner et al., 2023) and application across diverse fields including auditing (Gu et al.”
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 377 · cited 1×
“Given the growing evidence that data generated by LLMs are un - likely to effectively train new LLMs (22, 23), modelers face the real problem of running out of useful data.”
AI and the transformation of social science research
Grossmann, I., Feinberg, M., Parker, D. C., Christakis, N. A., Tetlock, P. E., & Cunningham, W. A. (2023) · Science · doi:10.1126/science.adi1778 · Cited by 377 · cited 1×
“poised to change how we do research (Grossmann et al., 2023) and present a strong competitor to search engines (Xu et al.”
Electronic Publication and the Narrowing of Science and Scholarship
Evans, J. A. (2008) · Science · doi:10.1126/science.1150473 · Cited by 364
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287 · cited 1×
“Because ChatGPT performs relatively well on software programming tasks (15), we study Stack Overflow, the largest online Q&A platform for software development and programming.”
OpenAssistant Conversations - Democratizing Large Language Model Alignment
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07327 · Cited by 218 · cited 1×
“Unlike synthetic data, data on user interactions with LLMs can be used to significantly improve and tune their performance (47).”
ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience
Xu, R., Feng, Y., & Chen, H. (2023) · arXiv · doi:10.48550/arxiv.2307.01135 · Cited by 129 · cited 1×
“, 2023) and present a strong competitor to search engines (Xu et al., 2023), we need to understand what LLM efficiency implies for our contact with diverse sources of information and incentives to try new things.”
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125 · cited 1×
“And though watermarks may help humans and models identify data creators (44), the general problem of determining whether, for example, a text is written by a human or LLM is difficult at scale (45).”
Nothing to See Here? A non-inferiority approach to parallel trends
Bilinski, A., & Hatfield, L. A. (2018) · arXiv · doi:10.48550/arxiv.1805.03273 · Cited by 104
The consequences of generative AI for online knowledge communities
Burtch, G., Lee, D., & Chen, Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-61221-0 · Cited by 83 · cited 1×
“Recent work by Burtch et al. (37) studying the evolution of activity on Stack Overflow and Reddit found similar results to ours.”
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80
Veselovsky, V., Ribeiro, M. H., & West, R. (2023) · arXiv · doi:10.48550/arxiv.2306.07899 · Cited by 73 · cited 1×
“…Overflow and Reddit demonstrate the value of this kind of data for the continued training of LLMs. h The proliferation of LLMs has already impacted other forms of data creation: many Amazon Mechanical Turk workers now generate content (i.e. respond to surveys, evaluate texts) using ChatGPT (43).”
Research works printed but not linked: 25
- Henzinger, 2004, Extracting knowledge from the world wide web, Proc Natl Acad Sci U S A, 101, 5186, 10.1073/pnas.0307528100 · DOI
- Hess, 2003, Ideas, artifacts, and facilities: information as a common-pool resource, Law Contemp Probl, 66, 111
- Smith, 2023, Astronomia ex machina: a history, primer and outlook on neural networks in astronomy, R Soc Open Sci, 10, 221454, 10.1098/rsos.221454 · DOI
- Guo, 2023, 10.48550/arXiv.2305.18365 · DOI · Paper page
- Delile, 2023, 10.48550/arXiv.2306.11174 · arXiv · doi:10.48550/arxiv.2306.11174
- Widjojo, 2023, 10.48550/arXiv.2307.10793 · DOI
- Chiang, 2023, ChatGPT is a blurry JPEG of the web
- David, 1985, Clio and the economics of QWERTY, Am Econ Rev, 75, 332
- Stojkoski, 2024, Estimating digital product trade through corporate revenue data, Nat Commun, 15, 5262, 10.1038/s41467-024-49141-z · DOI
- Kreitmeir, 2023, 10.48550/arXiv.2304.09339 · DOI
- Eloundou, 2024, GPTs are GPTs: labor market impact potential of LLMs, Science, 384, 1306, 10.1126/science.adj0998 · doi:10.1126/science.adj0998 · Paper page
- Schumpeter, 1942, Capitalism, socialism, and democracy
- Gaessler, 2023, Training with AI: evidence from chess computers, Strat Manag J, 44, 2724, 10.1002/smj.3512 · DOI · Paper page
- Taleb, 2012, Antifragile: how to live in a world we don’t understand
- Wyllie, 2024, 10.48550/arXiv.2403.07857 · arXiv · doi:10.48550/arxiv.2403.07857
- Tian-Zheng Wei, 2024, 10.48550/arXiv.2402.10892 · DOI
- Page, 2006, Path dependence, Quart J Polit Sci, 1, 87, 10.1561/100.00000006 · DOI · Paper page
- Vincent, 2021, A deeper investigation of the importance of Wikipedia links to search engine results, Proc ACM Hum-Comput Inter, 5, 1
- Greenstein, 2014, Digital dark matter and the economic contribution of Apache, Res Policy, 43, 623, 10.1016/j.respol.2014.01.003 · DOI
- Xu, 2020, What makes geeks tick? A study of stack overflow careers, Manage Sci, 66, 587, 10.1287/mnsc.2018.3264 · DOI
- Sandrini, 2023, Generative ai and deceptive news consumption, Econ Lett, 232, 111317, 10.1016/j.econlet.2023.111317 · DOI
- Johnson, 2023, Power and progress: our thousand-year struggle over technology and prosperity
- Lehdonvirta, 2022, Cloud empires: how digital platforms are overtaking the state and how we can regain control, 10.7551/mitpress/14219.001.0001 · DOI
- Kummer, 2020, Unemployment and digital public goods contribution, Inform Syst Res, 31, 801, 10.1287/isre.2019.0916 · DOI
- Héder, 2022, The past, present and future of the ELKH cloud, Inform Társadalom, 22, 128, 10.22503/inftars.XXII.2022.2.8 · DOI
Cited by
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71
Winners and losers of generative AI: Early Evidence of Shifts in Freelancer Demand
Teutloff, O., Einsiedler, J., Kässi, O., Braesemann, F., Mishkin, P., & del Rio-Chanona, R. M. (2025) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2024.106845 · Cited by 39
Introducing AI to an Online Petition Platform Changed Outputs but not Outcomes
Corpus, I., Gilbert, E., Koenecke, A., & Naaman, M. (2025) · arXiv · doi:10.48550/arxiv.2511.13949 · Cited by 2
Metadata
- Authors
- del Rio-Chanona, R. M., Laurentsyeva, N., & Wachs, J.
- Affiliations
- Department of Computer Science, University College London , London WC1E 6EA ,
- Faculty of Economics, LMU Munich , Munich 80539 ,
- Department of Network Science, Corvinus University of Budapest , Budapest 1093 ,
- Venue
- PNAS Nexus
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-09
- Volume
- 3
- Issue
- 9
- ISSN
- 2752-6542
- Cited by
- 52
- References
- 21
- Language
- en
- Funders
- Hungarian National Scientific Fund (OTKA FK 145960)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.