CoChat

Artificial Influence: Comparing the Effects of AI and Human Source Cues in Reducing Certainty in False Beliefs

Open access

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

People often resist updating their beliefs, even when confronted with strong evidence. While some research suggests artificial intelligence (AI) could be a solution to this problem, its persuasive capacity remains under-explored. This pre-registered study examines whether large language models (LLMs) can reduce belief certainty among a sample of N=1,730 Americans, all of whom held at least one false or unsupported belief. Treated participants engaged in up to five rounds of conversation with ChatGPT-4o, with the treatment manipulating who participants were told they were speaking with: ChatGPT, an expert on the topic, or a fellow survey respondent who disagreed with them. Across all conditions, we find that the conversation with AI reduced participants' certainty in their false or unsupported beliefs, with 29\% of participants even persuaded to switch to an accurate belief. However, ChatGPT as a source label did not contribute to this persuasive capacity. The reduction in belief certainty was not significantly greater for the ChatGPT label compared to a fellow survey taker label, though it was for the expert label. Our findings point largely to message effects—even when brief, multi-round conversations with AI have a clear influence, emphasizing the significance of both the content and interactive nature of these conversations. However, the role of source cues remains critical, with the appeal of a human expert source label over a ChatGPT label raising important questions wit…

Overview

Integrity signals

Registry

Crossref · 10.31219/osf.io/2vh4k

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Cited by

See all 9

Metadata

Authors
Goel, N., Bergeron, T., Lee-Whiting, B., galipeau, T., Bohonos, D., Islam, M., Lachance, S., Savolainen, S., Treger, C., & Merkley, E.
Published
2024-11-04
OpenAlex
W4404031955
Cited by
9
References
2
Citations by year
2026: 2 · 2025: 7
Type
posted content
Language
en
Open access
Locations
OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Psychology › Social Psychology › Psychology of Social Influence
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Strategies and Epistemologies
Keywords
large language models, ChatGPT, belief updating, belief certainty, false belief, source cues, message effects, belief revision

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.