CoChat

Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation

Open access
Park, Seyeon, Nan, Xiaoli · 2026 · AI & SOCIETY

The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; 44 of the 64 sources its claims cite were checked for support.

Abstract

The rapid advancement of generative artificial intelligence (AI) has introduced both opportunities and challenges in the fight against misinformation. This scoping review synthesizes recent empirical studies to explore the dual role of generative AI—particularly large language models (LLMs)—in the generation, detection, mitigation, and impact of misinformation. Analyzing 24 empirical studies, our review suggests that LLMs can generate highly convincing misinformation, often exploiting cognitive biases and ideological leanings of the audiences, while also demonstrating the ability to detect false claims and enhance users’ resistance to misinformation. Mitigation efforts show mixed results, with personalized corrections proving effective but safeguards inconsistently applied. Additionally, exposure to AI-generated misinformation was found to reduce trust and influence decision-making. This review underscores the need for standardized evaluation metrics, interdisciplinary collaboration, and stronger regulatory measures to ensure the responsible use of generative AI in the information ecosystem.

Overview

CoChat Summary

TL;DR

This scoping review synthesizes 24 empirical studies from 16 articles on generative AI’s dual role in misinformation: LLMs can generate convincing false content, but also show promise for detection and mitigation. The review finds mixed and context-dependent performance, with calls for standardized metrics, stronger safeguards, and broader geographic and longitudinal evidence.

Key findings

  • LLMs can generate highly credible misinformation, especially when personalized or tailored: ChatGPT-4.0-generated personalized false headlines were difficult to detect when aligned with participants’ demographics; Menz et al. found ChatGPT, Google Bard, and HuggingChat generated over 40,000 words across 113 unique health misinformation outputs without jailbreaking.
  • Detection performance was inconsistent: ChatGPT-4.0 generally outperformed GPT-3.5 on vaccine myths, but Węcel et al. found ChatGPT’s fake-news accuracy remained low, only slightly better than random guessing, with slight-to-fair agreement with human fact-checkers; Makhortykh et al. found more than 25% of chatbot responses failed expert-verified standards.
  • Mitigation showed mixed results: GPT-4 explanation-based interventions improved accuracy by up to 47.6%, and personalized explanations were rated more helpful when aligned with users, but misaligned personalization reduced credibility; safeguards were fragile, with Claude 2 consistently refusing disinformation while Copilot later generated it in 12-week follow-up tests.
  • Exposure to AI-generated misinformation reduced trust and influenced decisions: in travel experiments, incorrect ChatGPT information lowered visit intentions even with error reminders; Spitale et al. found participants were no better than random guessing at distinguishing AI-generated from human-written tweets, and AI disinformation was more compelling and harder to detect.

Methods

  • Scoping review following PRISMA-ScR guidelines; Google Scholar search from January 2018 to September 19, 2024, combining generative AI terms (e.g., LLM, GPT, ChatGPT) and misinformation terms (e.g., misinformation, disinformation, fake news), retaining the first 100 results per query.
  • Screening yielded 2,000 records; after removing 1,026 duplicates, 974 articles were title/abstract screened, 918 excluded, 56 full-text reviewed, 40 excluded, and 16 articles retained, containing 24 empirical studies.
  • Both authors independently coded studies using an iteratively refined codebook; intercoder agreement was 100% for most variables, 81.25% for Study Type (average kappa = 0.822), and 75% for Evaluation Method (k = 0.50), with discrepancies resolved by consensus.

Limitations

  • The paper states: inclusion focused on publicly available generative AI systems, excluding highly customized or proprietary models; the review does not quantify effect sizes or aggregate findings statistically; it was limited to English-language studies retrieved primarily through Google Scholar; most studies were from North America, Europe, and Australia, introducing Western-centric bias; and impact claims rely on a narrow set of experimental studies, many in travel contexts.
  • Evident from the text: the scoping review design limits comparative or evaluative judgments about intervention effectiveness or model performance, as the authors also acknowledge.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s00146-025-02620-3

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Fig. 2 Geographic distribution of studies
Fig. 2 Geographic distribution of studiesFrom the PDF
Image 3
From the PDF

Verification

Claims the paper makes

39 Valid · 0 Not valid · 17 Not checked · 39 Own results and interpretations

48 more claims · 17 with their cited sources checked

Create a free account to see all 95 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 13
  • It may state two separate claims as one. 32
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 70 references

Research works printed but not linked: 2

  1. Bandara C (2024) Hallucination as disinformation: the role of LLMs in amplifying conspiracy theories and fake news. J Appl Cybersecur Anal Intell Decis-Mak Syst 14:65–76
  2. Sundar SS (2008) The MAIN model: A heuristic approach to understanding technology effects on credibility. In: Metzger MJ, Flanagin AJ (eds) Digital media, youth, and credibility. The MIT Press, Cambridge, pp 73–100. https://doi.org/10.1162/dmal.9780262562324.073 · doi:10.1162/dmal.9780262562324.073

Cited by

See all 58

Metadata

Authors
Park, S., & Nan, X.
Affiliations
University of Maryland, College Park (US)
Corresponding author
Park, Seyeon
Venue
AI & SOCIETY
Published
2025-09-30
Volume
41
Issue
2
Pages
1501-1515
ISSN
  • 0951-5666 (ISSN-L)
  • 1435-5655
OpenAlex
W4414644854
Cited by
58
References
62
Citations by year
2026: 55 · 2025: 3
FWCI
99.66
Type
journal article
Language
en
Open access
Licence
Locations
AI & Society (publishedVersion, cc-by)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
generative AI, misinformation, large language models, AI-generated misinformation, misinformation detection, misinformation mitigation, cognitive biases, ideological bias, trust in AI, decision-making, AI regulation, responsible AI use

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation. https://app.cochat.ai/papers/10.1007/s00146-025-02620-3
The paper
Park, S., & Nan, X. (2026). Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation. AI & SOCIETY. https://doi.org/10.1007/s00146-025-02620-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.