Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index; none of the 8 sources its claims cite is checked for support yet.

Dialogues with large language models reduce conspiracy beliefs even when the AI is perceived as human

Open access
Boissin, Esther, Costello, Thomas H, Spinoza-Martín, Daniel, Rand, David G, Pennycook, Gordon · 2025 · PNAS Nexus

Abstract

Although conspiracy beliefs are often viewed as resistant to correction, recent evidence shows that personalized, fact-based dialogues with a large language model (LLM) can reduce them. Is this effect driven by the debunking facts and evidence, or does it rely on the messenger being an AI? In other words, would the same message be equally effective if delivered by a human? To answer this question, we conducted a preregistered experiment (N = 955) in which participants reported either a conspiracy belief or a nonconspiratorial but epistemically unwarranted belief and interacted with a LLM that argued against that belief using facts and evidence. We randomized whether the debunking LLM was characterized as an AI tool or a human expert and whether the model used human-like conversational tone. The conversations significantly reduced participants' confidence in both conspiracies and epistemically unwarranted beliefs, with no significant differences across conditions. Thus, AI persuasion is not reliant on the messenger being an AI model: it succeeds by generating compelling messages.

Overview

CoChat Summary

TL;DR

In a preregistered experiment (N = 955), two-round GPT-4o dialogues reduced confidence in conspiracy and other epistemically unwarranted beliefs. The debunking effect did not differ whether the model was labeled an AI or an expert, or whether it used a human-like tone.

Key findings

  • Across all conditions, the conversation significantly reduced belief (all ps < 0.018). The decrease was larger for conspiracy beliefs (10 points, 11.81% decrease) than for general “epistemically weak” beliefs (5 points, 5.96% decrease); b = 5.11, 95% CI (2.75, 7.48), P < 0.001.
  • No significant effects of debunker identification (AI vs. Expert), prompt type (Neutral vs. Human-like), or their interactions (all ps > 0.25). A Bayes factor analysis favored the null model, BF01 = 518,181.1 ± 6.43%.
  • Among participants told they were conversing with an expert, 45% used language explicitly indicating they thought the interlocutor was human, while 21% suspected they were talking to an AI. Controlling for AI detection or excluding detectors still yielded no significant differences across conditions.
  • Exploratory NLP analyses found participants in the AI condition used less lexically diverse language than those in the Expert condition (type–token ratio: b = −0.02, P = 0.032; unique word types: b = −4.58, P = 0.032), with no significant differences for syntactic complexity, epistemic markers, or argument connectives (all P > 0.084).

Methods

  • Preregistered experiment; 8,274 individuals were recruited from Lucid’s participant pool, 1,997 completed the full study, and the final sample included 955 participants (506 in the conspiracy-belief condition, 449 in the general epistemically suspect belief condition) after preregistered exclusions.
  • Participants reported either a conspiracy belief or a nonconspiratorial but epistemically unwarranted belief, rated confidence (0–100), had a two-round chat with GPT-4o prompted to reduce confidence in that belief, and then re-rated confidence. They were randomized to debunker identification (AI vs. Expert) and prompt type (Neutral vs. Human-like).
  • Main analysis used linear regression predicting postconversation belief with contrast-coded predictors for debunker identification, prompt type, belief type, all two-way and three-way interactions, and a control for preconversation belief; post hoc Bayes factor and exploratory NLP analyses were also reported.

Limitations

  • Stated: the model was trained primarily on large-scale Internet data dominated by English and Western sources, likely shaping its tone, assumptions, and argumentative style; debunking may not generalize across cultural contexts.
  • Stated: analyses controlling for AI detection or excluding AI detectors condition on post-treatment variables, creating causal inference caveats.
  • Stated/evident: attrition was significantly higher in the Expert condition (47.50%) than the AI condition (42.69%; t(3629) = 2.99, P = 0.003), though results remained robust under conservative imputation; the final sample came from Lucid after exclusions, which may limit generalizability (evident, not framed as a limitation by the authors).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/pnasnexus/pgaf325

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Reduction in belief (pretreatment belief—post-treatment belief) as a function of speaker (Expert vs. AI) and Prompt Type (Human-like vs. Neutral) for the conspiracy belief debunking condition (left panel) and the epistemically unwarranted belief debunking condition (right panel). Error bars indicate 95% CI.
Fig. 1. Reduction in belief (pretreatment belief—post-treatment belief) as a function of speaker (Expert vs. AI) and Prompt Type (Human-like vs. Neutral) for the conspiracy belief debunking condition (left panel) and the epistemically unwarranted belief debunking condition (right panel). Error bars indicate 95% CI.CC BY · PubMed Central

Verification

Claims the paper makes

0 Valid · 0 Not valid · 6 Not checked · 22 Own results and interpretations

10 more claims

Create a free account to see all 28 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 7
  • It may state two separate claims as one. 7

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentt(3629) = 2.99reported p = .003 · recomputed p = .003

    “t(3629) = 2.99, p = .003”

1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 13 references

Research works printed but not linked: 1

  1. Sundar, 2008, Digital Media, youth, and credibility, 73

Cited by

See all 18

Metadata

Authors
Boissin, E., Costello, T. H., Spinoza-Martín, D., Rand, D. G., & Pennycook, G.
Affiliations
  • Cornell University (US)
  • American University (US)
  • Massachusetts Institute of Technology (US)
  • University of Regina (CA)
Corresponding author
Boissin, Esther
Venue
PNAS Nexus
Published
2025-10-14
Volume
4
Issue
11
Pages
pgaf325-pgaf325
ISSN
2752-6542 (ISSN-L)
OpenAlex
W4415222261
Cited by
18
References
13
Citations by year
2026: 17 · 2025: 1
FWCI
30.93
Type
journal article
Language
en
Open access
Locations
  • PNAS Nexus (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Social Sciences and Humanities Research Council of Canada
  • Fyssen Foundation
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Psychology › Social Psychology › Psychology of Social Influence
  • Social Sciences › Psychology › Social Psychology › Paranormal Experiences and Beliefs
Keywords
conspiracy beliefs, large language models, AI persuasion, debunking, belief confidence, misinformation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Dialogues with large language models reduce conspiracy beliefs even when the AI is perceived as human. https://app.cochat.ai/papers/10.1093/pnasnexus/pgaf325
The paper
Boissin, E., Costello, T. H., Spinoza-Martín, D., Rand, D. G., & Pennycook, G. (2025). Dialogues with large language models reduce conspiracy beliefs even when the AI is perceived as human. PNAS Nexus. https://doi.org/10.1093/pnasnexus/pgaf325

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.