Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; 44 of the 64 sources its claims cite were checked for support.
Abstract
The rapid advancement of generative artificial intelligence (AI) has introduced both opportunities and challenges in the fight against misinformation. This scoping review synthesizes recent empirical studies to explore the dual role of generative AI—particularly large language models (LLMs)—in the generation, detection, mitigation, and impact of misinformation. Analyzing 24 empirical studies, our review suggests that LLMs can generate highly convincing misinformation, often exploiting cognitive biases and ideological leanings of the audiences, while also demonstrating the ability to detect false claims and enhance users’ resistance to misinformation. Mitigation efforts show mixed results, with personalized corrections proving effective but safeguards inconsistently applied. Additionally, exposure to AI-generated misinformation was found to reduce trust and influence decision-making. This review underscores the need for standardized evaluation metrics, interdisciplinary collaboration, and stronger regulatory measures to ensure the responsible use of generative AI in the information ecosystem.
Overview
CoChat Summary
TL;DR
This scoping review synthesizes 24 empirical studies from 16 articles on generative AI’s dual role in misinformation: LLMs can generate convincing false content, but also show promise for detection and mitigation. The review finds mixed and context-dependent performance, with calls for standardized metrics, stronger safeguards, and broader geographic and longitudinal evidence.
Key findings
- LLMs can generate highly credible misinformation, especially when personalized or tailored: ChatGPT-4.0-generated personalized false headlines were difficult to detect when aligned with participants’ demographics; Menz et al. found ChatGPT, Google Bard, and HuggingChat generated over 40,000 words across 113 unique health misinformation outputs without jailbreaking.
- Detection performance was inconsistent: ChatGPT-4.0 generally outperformed GPT-3.5 on vaccine myths, but Węcel et al. found ChatGPT’s fake-news accuracy remained low, only slightly better than random guessing, with slight-to-fair agreement with human fact-checkers; Makhortykh et al. found more than 25% of chatbot responses failed expert-verified standards.
- Mitigation showed mixed results: GPT-4 explanation-based interventions improved accuracy by up to 47.6%, and personalized explanations were rated more helpful when aligned with users, but misaligned personalization reduced credibility; safeguards were fragile, with Claude 2 consistently refusing disinformation while Copilot later generated it in 12-week follow-up tests.
- Exposure to AI-generated misinformation reduced trust and influenced decisions: in travel experiments, incorrect ChatGPT information lowered visit intentions even with error reminders; Spitale et al. found participants were no better than random guessing at distinguishing AI-generated from human-written tweets, and AI disinformation was more compelling and harder to detect.
Methods
- Scoping review following PRISMA-ScR guidelines; Google Scholar search from January 2018 to September 19, 2024, combining generative AI terms (e.g., LLM, GPT, ChatGPT) and misinformation terms (e.g., misinformation, disinformation, fake news), retaining the first 100 results per query.
- Screening yielded 2,000 records; after removing 1,026 duplicates, 974 articles were title/abstract screened, 918 excluded, 56 full-text reviewed, 40 excluded, and 16 articles retained, containing 24 empirical studies.
- Both authors independently coded studies using an iteratively refined codebook; intercoder agreement was 100% for most variables, 81.25% for Study Type (average kappa = 0.822), and 75% for Evaluation Method (k = 0.50), with discrepancies resolved by consensus.
Limitations
- The paper states: inclusion focused on publicly available generative AI systems, excluding highly customized or proprietary models; the review does not quantify effect sizes or aggregate findings statistically; it was limited to English-language studies retrieved primarily through Google Scholar; most studies were from North America, Europe, and Australia, introducing Western-centric bias; and impact claims rely on a narrow set of experimental studies, many in travel contexts.
- Evident from the text: the scoping review design limits comparative or evaluative judgments about intervention effectiveness or model performance, as the authors also acknowledge.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Valid · 0 Not valid · 17 Not checked · 39 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“They found that ChatGPT-4.0-generated personalized misinformation was difficult to detect, especially when aligned with participants' demographics.”
What the sources say
- Supportedcites Gabriel et al. 2024 source
The Introduction reports an early finding that GPT-4-generated personalized disinformation became harder to identify as false when viewed by users aligned with its intended audience. Section 6.1 says the generated headlines targeted groups with specific demographic attributes. The citing sentence accurately paraphrases that finding. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“with ChatGPT achieving over 81% accuracy in classifying conspiracy theories.”
What the sources say
- Supportedcites Kuznetsova et al. 2025 source
The “Accuracy of detecting conspiratorial statements” section reports that ChatGPT gave 81–86% correct responses across all languages tested. The Discussion and conclusion likewise describes its performance as 81% and above in all three languages; the study notes that its statement selection was relatively small and focused on well-established conspiratorial claims. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“can verify factual errors based on extensive pretrained data (Augenstein et al. 2024; Kuznetsova et al. 2025)”
What the sources say
- Not checked — no full text of the sourcecites Augenstein et al. 2024 source
- Partly supportedcites Kuznetsova et al. 2025 sourceExperimental — accuracy not yet measured
The Detection accuracy of false, true, and borderline statements section (page unavailable) reports that the tested chatbots classified 70% of prompts in accordance with the veracity baseline, supporting their ability to identify factual errors in some cases. The paper discusses training-data cutoffs and dependence on training data, but does not establish that verification works *because of extensive pretrained data*. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“LLMs possess strong reasoning ability (Huang and Chang 2022)”
What the sources say
- Partly supportedcites Huang and Chang 2022 sourceExperimental — accuracy not yet measured
The Introduction reports strong performance on certain reasoning tasks, but says it remains unclear whether LLMs are actually reasoning or how far their abilities extend. Section 6 likewise says the evidence is insufficient to conclude that LLMs can truly reason. The claim presents a qualified finding about task performance as an established general ability. (LLM, conf=0.88)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“McMahon and McMahon (2024) identified significant misinformation from ChatGPT regarding self-managed abortion, notably overstating risks and contradicting established medical guidelines.”
What the sources say
- Supportedcites McMahon and McMahon 2024 source
The Results section says ChatGPT frequently overstated the risks of self-managed medication abortion. The Discussion says those responses contradicted evidence of its safety and effectiveness, and cites WHO guidance recognizing self-managed medication abortion as safe, effective, and acceptable. (LLM, conf=0.93)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
48 more claims · 17 with their cited sources checked
Create a free account to see all 95 claims and every source verdict.
- Probably not a substantial claim. 13
- It may state two separate claims as one. 32
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research on academic integrity highlights that AI-generated content may appear coherent and authoritative while lacking verifiable sources or containing fabricated information, thereby increasing the cognitive burden placed on students' evaluative judgment (Park and Nan, 2025; Slutskiy, 2025).”
From Information competence and critical thinking in digital environments evidence from university students in Kazakhstan (2026) · cites this paper as Park and Nan, 2025
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Adopting collaborative, multi-actor approaches to AI use and risk management further supports their ethical and strategic application (Park and Nan 2025).”
From Digital civics in action: the promise and emerging practices of generative AI in grassroots community advocacy (2026) · cites this paper as Park and Nan 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3.1 Misinformation generation via generative AI · p. 7
“Menz et al. (2024) evaluated the ability of five LLMs to generate health-related misinformation.”
§ 4.3.3 Mitigation of AI-generated misinformation · p. 10
“The researchers tested four widely used LLMs by prompting them to generate misinformation about two controversial health topics”
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565 · cited 1×
“This scoping review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines (Tricco et al. 2018) to ensure a structured and transparent methodology.”
The spreading of misinformation online
Del Vicario, M., Bessi, A., Zollo, F., Petroni, F., Scala, A., Caldarelli, G., Stanley, H. E., & Quattrociocchi, W. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1517441113 · Cited by 2,407 · cited 1×
“…has become a critical concern in communication research, particularly as digital platforms facilitate the rapid dissemination of false or mis-leading content (Del Vicario et al. 2016; Muhammed and Mathew 2022; Nan et al. 2022b; Suarez-Lledo and Alva-rez-Galvez 2021; Wang et al. 2019).”
Systematic Literature Review on the Spread of Health-related Misinformation on Social Media
Wang, Y., McKee, M., Torbica, A., & Stuckler, D. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.112552 · Cited by 1,806 · cited 1×
“…has become a critical concern in communication research, particularly as digital platforms facilitate the rapid dissemination of false or mis-leading content (Del Vicario et al. 2016; Muhammed and Mathew 2022; Nan et al. 2022b; Suarez-Lledo and Alva-rez-Galvez 2021; Wang et al. 2019).”
Prevalence of health misinformation on social media: systematic review
Suarez-Lledo, V., & Alvarez-Galvez, J. (2021) · Journal of Medical Internet Research · doi:10.2196/17187 · Cited by 1,564
The psychological drivers of misinformation belief and its resistance to correction
Ecker, U. K. H., Lewandowsky, S., Cook, J., Schmid, P., Fazio, L. K., Brashier, N., Kendeou, P., Vraga, E. K., & Amazeen, M. A. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-021-00006-y · Cited by 1,372 · cited 1×
“…understood as false or misleading information shared without malicious intent (Ireton and Posetti 2018), and, more broadly, as an umbrella term that includes all forms of false or misleading information, including fake news, conspiracy theories, pseudoscience, and propaganda (Ecker et al. 2022).”
Deepfakes and beyond: A Survey of face manipulation and fake detection
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237 · cited 1×
“The term “deepfake,” once narrowly associated with face-swapping technology, now encompasses a wide range of AI-generated media, including voice imitation and synthetic videos (Tolosana et al. 2020; Verdoliva 2020).”
Public health and online misinformation: challenges and recommendations
Swire-Thompson, B., & Lazer, D. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094127 · Cited by 1,170 · cited 3×
“In the health and science context, misinformation typically refers to content that contradicts expert consensus (Nan et al. 2023; Swire-Thompson and Lazer 2020; Vraga and Bode 2020).”
The Emergence of Deepfake Technology: A Review
Westerlund, M. (2019) · Technology Innovation Management Review · doi:10.22215/timreview/1282 · Cited by 1,138 · cited 1×
“These highly realistic fabrica-tions have been increasingly used in disinformation cam-paigns, particularly targeting political figures (Westerlund 2019), raising concerns about their impact on public perception and trust (Dobber et al. 2021; Vaccari and Chadwick 2020; Verma 2024).”
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120 · cited 1×
“Unlike earlier forms of AI, generative AI systems can autonomously produce human-like outputs from input prompts, powered by LLMs, deep learning models pre-trained on a vast corpus of textual data to predict and generate coherent language (Cascella et al. 2023).”
Inoculating the Public against Misinformation about Climate Change
van der Linden, S., Leiserowitz, A., Rosenthal, S., & Maibach, E. (2017) · Global Challenges · doi:10.1002/gch2.201600008 · Cited by 1,058 · cited 1×
“Psychological inoculation or prebunking, which offers preemptive warnings and refutations, has also shown promise in building resistance to misinformation (Van der Linden et al. 2017; Traberg et al. 2022).”
Research works printed but not linked: 2
- Bandara C (2024) Hallucination as disinformation: the role of LLMs in amplifying conspiracy theories and fake news. J Appl Cybersecur Anal Intell Decis-Mak Syst 14:65–76
- Sundar SS (2008) The MAIN model: A heuristic approach to understanding technology effects on credibility. In: Metzger MJ, Flanagin AJ (eds) Digital media, youth, and credibility. The MIT Press, Cambridge, pp 73–100. https://doi.org/10.1162/dmal.9780262562324.073 · doi:10.1162/dmal.9780262562324.073
Cited by
Evaluating the accuracy and reliability of AI content detectors in academic contexts
Hadra, M., Cambridge, K., & Mesbah, M. (2026) · International Journal for Educational Integrity · doi:10.1007/s40979-026-00213-1 · Cited by 27
“Dissemination of fake news to redirect public opinions and generate deceptive product reviews is also expected to increase many folds with the invention of GenAI tools, particularly those capable of producing not only text, but also synthetic images and videos (Park and Nan 2025; Saheb et al. 2024).”
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
“Misinformation, once generated by an AI system, can propagate rapidly and at scale, causing social harm that far exceeds the original data quality failure [268,269].”
Providing feedback without verification: A hidden risk of trust in generative AI
Hu, P., Cao, C., & Li, D. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103370 · Cited by 12
Pum, M. (2025) · Interdisciplinary Educational Technology · doi:10.71176/interedtech/17573 · Cited by 7
“Lastly, the misinformation generated by AI has a demonstrable influence on users' attitudes and decisions (Park & Nan, 2025).”
Agarwal, A., Sebastian, M. P., & Krishnan, S. (2026) · Internet Research · doi:10.1108/intr-01-2025-0050 · Cited by 6
Metadata
- Authors
- Park, S., & Nan, X.
- Affiliations
- University of Maryland, College Park (US)
- Corresponding author
- Park, Seyeon
- Venue
- AI & SOCIETY
- Published
- 2025-09-30
- Volume
- 41
- Issue
- 2
- Pages
- 1501-1515
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- OpenAlex
- W4414644854
- Cited by
- 58
- References
- 62
- Citations by year
- 2026: 55 · 2025: 3
- FWCI
- 99.66
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI & Society (publishedVersion, cc-by)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- generative AI, misinformation, large language models, AI-generated misinformation, misinformation detection, misinformation mitigation, cognitive biases, ideological bias, trust in AI, decision-making, AI regulation, responsible AI use
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation. https://app.cochat.ai/papers/10.1007/s00146-025-02620-3
- The paper
- Park, S., & Nan, X. (2026). Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation. AI & SOCIETY. https://doi.org/10.1007/s00146-025-02620-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


