AI model GPT-3 (dis)informs us better than humans
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) is changing the way we create and evaluate information, and this is happening during an infodemic, which has been having marked effects on global health. Here, we evaluate whether recruited individuals can distinguish disinformation from accurate information, structured in the form of tweets, and determine whether a tweet is organic or synthetic, i.e., whether it has been written by a Twitter user or by the AI model GPT-3. The results of our preregistered study, including 697 participants, show that GPT-3 is a double-edge sword: In comparison with humans, it can produce accurate information that is easier to understand, but it can also produce more compelling disinformation. We also show that humans cannot distinguish between tweets generated by GPT-3 and written by real Twitter users. Starting from our results, we reflect on the dangers of AI for disinformation and on how information campaigns can be improved to benefit global health.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, AI-generated misinformation influences attitudes and decisions (Kim et al. 2023; Spitale et al. 2023).”
From Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation (2026) · cites this paper as Spitale et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As such, challenges arise as, by now, outputs are indistinguishable from authentic content and may present misinformation or deceive users (Spitale et al. 2023).”
From Generative AI (2024) · cites this paper as Spitale et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“it was revealed that GPT-3 is a double-edged sword: it produces accurate information that is easier to understand than human-written content but also generates disinformation that is more compelling and harder to detect.”
From Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation (2026) · cites this paper as Spitale et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“and vast amounts of compelling mis/disinformation with little human involvement (Buchanan et al., 2021;Spitale et al., 2023).”
From Anatomy of an AI-powered malicious social botnet (2024) · cites this paper as Spitale et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731 · cited 1×
“An alternative view is proposed by the Dunning-Kruger effect, which can also help in interpreting our findings ( 25 , 26 ).”
Cook, J., Lewandowsky, S., & Ecker, U. K. H. (2017) · PLOS ONE · doi:10.1371/journal.pone.0175799 · Cited by 971
Dale, R. (2021) · Natural Language Engineering · doi:10.1017/s1351324920000601 · Cited by 432 · cited 2×
“GPT-3, the latest iteration of the generative pretrained trans-formers developed by OpenAI, is arguably the most advanced system of pretrained language representations ( 2 ).”
Plagiarism in the age of massive generative pre-trained transformers (GPT-3)
Dehouche, N. (2021) · Ethics in Science and Environmental Politics · doi:10.3354/esep00195 · Cited by 296
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · cited 1×
“In line with previous research ( 21 ), we found that both respondents and GPT-3 were not able to distinguish whether a tweet was organic or synthetic (data on GPT-3 ’ s assessment are available in the study ’ s repository) ( 15 ).”
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290 · cited 1×
“In line with previous research ( 21 ), we found that both respondents and GPT-3 were not able to distinguish whether a tweet was organic or synthetic (data on GPT-3 ’ s assessment are available in the study ’ s repository) ( 15 ).”
Cabanac, G., Labbé, C., & Magazinov, A. (2021) · arXiv · doi:10.48550/arxiv.2107.06751 · Cited by 224 · cited 1×
“The system relies on statistical representations of language for how it is used in real-life by real humans or “ a simulacrum of the interaction between people and the world ” ( 4 ).”
Mahmood, K., & University of the Punjab (2016) · Comminfolit · doi:10.15760/comminfolit.2016.10.2.24 · Cited by 212
Can GPT-3 pass a writer’s turing test?
Elkins, K., & Chun, J. (2020) · Journal of Cultural Analytics · doi:10.22148/001c.17212 · Cited by 202 · cited 1×
“…tool for machine translations, text classification, dialogue/chatbot systems, knowledge summarizing, question answering, creative writing ( 2 , 5 , 6 ), detecting hate speech ( 7 ), and automatic code writing ( 2 , 8 ), it can also be used to produce “ misinformation, spam, phishing, abuse of…”
Research works printed but not linked: 1
- T. Rosenstiel J. Sonderman K. Loker M. Ivancin N. Kjarval “How people use Twitter in general.” American Press Institute (2015); www.americanpressinstitute.org/publications/reports/survey-research/how-people-use-twitter-in-general/.
Cited by
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 2,267
“content and may present misinformation or deceive users (Spitale et al., 2023).”
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 596
“Simply put, it is a piece of software that can create lines on its own that are so distinctive they almost sound like they were written by a human [25].”
BackgroundGenerative artificial intelligence, human creativity, and art
Zhou, E., & Lee, D. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae052 · Cited by 459
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial
Salvi, F., Ribeiro, M. H., Gallotti, R., & West, R. (2024) · doi:10.21203/rs.3.rs-4429707/v1 · Cited by 285
Metadata
- Authors
- Spitale, G., Biller-Andorno, N., & Germani, F.
- Affiliations
- University of Zurich (CH)
- Institute of Biomedical Ethics and History of Medicine, University of Zurich, Zurich, Switzerland.
- Corresponding author
- Germani, Federico
- Venue
- Science Advances
- Published
- 2023-06-30
- Volume
- 9
- Issue
- 26
- Pages
- eadh1850-eadh1850
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC10306283
- OpenAlex
- W4382361534
- Cited by
- 241
- References
- 23
- Citations by year
- 2026: 64 · 2025: 89 · 2024: 65 · 2023: 23
- FWCI
- 103.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Locations
- Science Advances (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- GPT-3, disinformation, synthetic content detection, AI-generated text, infodemic, information campaigns, global health communication
- MeSH terms
- Artificial Intelligence, Humans, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.