Artificial Influence: Comparing the Effects of AI and Human Source Cues in Reducing Certainty in False Beliefs
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
People often resist updating their beliefs, even when confronted with strong evidence. While some research suggests artificial intelligence (AI) could be a solution to this problem, its persuasive capacity remains under-explored. This pre-registered study examines whether large language models (LLMs) can reduce belief certainty among a sample of N=1,730 Americans, all of whom held at least one false or unsupported belief. Treated participants engaged in up to five rounds of conversation with ChatGPT-4o, with the treatment manipulating who participants were told they were speaking with: ChatGPT, an expert on the topic, or a fellow survey respondent who disagreed with them. Across all conditions, we find that the conversation with AI reduced participants' certainty in their false or unsupported beliefs, with 29\% of participants even persuaded to switch to an accurate belief. However, ChatGPT as a source label did not contribute to this persuasive capacity. The reduction in belief certainty was not significantly greater for the ChatGPT label compared to a fellow survey taker label, though it was for the expert label. Our findings point largely to message effects—even when brief, multi-round conversations with AI have a clear influence, emphasizing the significance of both the content and interactive nature of these conversations. However, the role of source cues remains critical, with the appeal of a human expert source label over a ChatGPT label raising important questions wit…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The Nature and Origins of Misperceptions: Understanding False and Unsupported Beliefs About Politics
Flynn, D. J., Nyhan, B., & Reifler, J. (2017) · Political Psychology · doi:10.1111/pops.12394 · Cited by 1,183
Durably reducing transphobia: A field experiment on door-to-door canvassing
Broockman, D., & Kalla, J. (2016) · Science · doi:10.1126/science.aad9713 · Cited by 798
Cited by
Boissin, E., Costello, T. H., Spinoza-Martín, D., Rand, D. G., & Pennycook, G. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf325 · Cited by 18
Metadata
- Authors
- Goel, N., Bergeron, T., Lee-Whiting, B., galipeau, T., Bohonos, D., Islam, M., Lachance, S., Savolainen, S., Treger, C., & Merkley, E.
- ORCID
- Publisher
- Center for Open Science
- Published
- 2024-11-04
- OpenAlex
- W4404031955
- Cited by
- 9
- References
- 2
- Citations by year
- 2026: 2 · 2025: 7
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › Psychology › Social Psychology › Psychology of Social Influence
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Strategies and Epistemologies
- Keywords
- large language models, ChatGPT, belief updating, belief certainty, false belief, source cues, message effects, belief revision
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.