CoChat
Sign Up
Cited by 38Misinformation and Its ImpactsInformation Retrieval and Search BehaviorEducational Strategies and Epistemologies

Conversational presentation mode increases credibility judgements during information search with ChatGPT

Open access
Anderl, Christine, Klein, Stefanie H., Sarigül, Büsra, Schneider, Frank M., Han, Junyi, Fiedler, Paul L., Utz, Sonja·2024·Scientific Reports
DOI 10.1038/s41598-024-67829-6PMC PMC11272919

Published version of a preprint

Open this paper in CoChat — free account

Abstract

People increasingly use large language model (LLM)-based conversational agents to obtain information. However, the information these models provide is not always factually accurate. Thus, it is critical to understand what helps users adequately assess the credibility of the provided information. Here, we report the results of two preregistered experiments in which participants rated the credibility of accurate versus partially inaccurate information ostensibly provided by a dynamic text-based LLM-powered agent, a voice-based agent, or a static text-based online encyclopedia. We found that people were better at detecting inaccuracies when identical information was provided as static text compared to both types of conversational agents, regardless of whether information search applications were branded (ChatGPT, Alexa, and Wikipedia) or unbranded. Mediation analysis overall corroborated the interpretation that a conversational nature poses a threat to adequate credibility judgments. Our research highlights the importance of presentation mode when dealing with misinformation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-024-67829-6

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The polished appearance of AI-generated content may make it more difficult for learners to distinguish readability from reliability, encouraging them to equate expressive quality with evidential quality and reducing their motivation to verify.

    “Its polished appearance may make it more difficult for learners to distinguish readability from reliability (Kasneci et al., 2023; Lee et al., 2025), encouraging them to equate expressive quality with evidential quality and reducing their motivation to verify (Anderl et al., 2024; Martín-Moncunill and Alonso Martínez, 2025).”

    From Responsible use of generative AI in Chinese vocational education: eye-tracking evidence on source verification and immediate near-transfer across three information-seeking conditions (2026) · cites this paper as Anderl et al., 2024

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. [6]Niszczota, P. & Rybicka, I. The credibility of dietary advice formulated by ChatGPT: Robo-diets for people with food allergies. Nutrition 112, 112076 (2023). 37269717 10.1016/j.nut.2023.112076 · DOI
  2. [7]Ali, R. et al. Performance of ChatGPT, GPT-4, and google bard on a neurosurgery oral boards preparation question bank. Neurosurgery 10.1227/neu.0000000000002551 (2023). 37306460 10.1227/neu.0000000000002551 · DOI
  3. [8]Gaiser, F. & Utz, S. Is hearing really believing? The importance of modality for perceived message credibility during information search with smart speakers. J. Media Psychol. Theor. Methods Appl.10.1027/1864-1105/a000384 (2023). · DOI
  4. [9]Sundar, S. S. The MAIN model: A heuristic approach to understanding technology effects on credibility. In Digital media, youth, and credibility (eds. Metzger, M. J. & Flanagin, A. J.) 73–100 (MIT Press Cambridge, MA, 2008).
  5. [11]Qiu, L. & Benbasat, I. Online consumer trust and live help interfaces: The effects of text-to-speech voice and three-dimensional avatars. Int. J. Hum. Comput. Interact. 19, 75–94 (2005). 10.1207/s15327590ijhc1901_6 · DOI
  6. [12]Huschens, M., Briesch, M., Sobania, D. & Rothlauf, F. Do you trust ChatGPT?--Perceived credibility of human and AI-generated content. arXiv Prepr. arXiv:2309.02524 (2023). · arXiv
  7. [13]Grice, H.P. Logic and conversation. Syntax and Semantics: Speech Acts (ed. Cole, P. & Morgan, J. L.) 3, 41–58 (Academic Press, 1975).
  8. [14]Guay, B., Berinsky, A. J., Pennycook, G. & Rand, D. How to think about whether misinformation interventions work. Nat. Hum. Behav. 7 (8), 1231–1233 (2023). 37563304 10.1038/s41562-023-01667-w · DOI
  9. [15]Guess, A. M. et al. A digital media literacy intervention increases discernment between mainstream and false news in the United States and India. Proc. Natl. Acad. Sci. 117 (27), 15536–15545 (2020). 32571950 10.1073/pnas.1920498117 PMC7355018 · DOI · Paper page
  10. [17]Graf, L. K. M., Mayer, S. & Landwehr, J. R. Measuring processing fluency: One versus five items. J. Consum. Psychol. 28, 393–411 (2018). 10.1002/jcpy.1021 · DOI
  11. [18]Schwarz, N., Jalbert, M., Noah, T. & Zhang, L. Metacognitive experiences as information: Processing fluency in consumer judgment and decision making. Consum. Psychol. Rev. 4, 4–25 (2021). 10.1002/arcp.1067 · DOI
  12. [19]Song, H. & Schwarz, N. Fluency and the detection of misleading questions: Low processing fluency attenuates the Moses illusion. Soc. Cogn. 26, 791–799 (2008). 10.1521/soco.2008.26.6.791 · DOI
  13. [20]Scholl, S. G., Greifeneder, R. & Bless, H. When fluency signals truth: Prior successful reliance on fluency moderates the impact of fluency on truth judgments. J. Behav. Decis. Mak. 27, 268–280 (2014). 10.1002/bdm.1805 · DOI
  14. [21]Schwede, M., Zierau, N., Janson, A., Hammerschmidt, M. & Leimeister, J. M. ’I Will Follow You!’–How recommendation modality impacts processing fluency and purchase intention. In How Recommendation Modality Impacts Processing Fluency and Purchase Intention (December 9, 2022). Forty-Third International Conference on Information Systems, Copenhagen (2022).
  15. [22]Chaiken, S. & Maheswaran, D. Heuristic processing can bias systematic processing: Effects of source credibility, argument ambiguity, and task importance on attitude judgment. J. Pers. Soc. Psychol. 66, 460–473 (1994). 8169760 10.1037/0022-3514.66.3.460 · DOI
  16. [24]Short, J., Williams, E. & Christie, B. The social psychology of telecommunications. (Wiley, 1976).
  17. [25]Lee, K. M., Jung, Y., Kim, J. & Kim, S. R. Are physically embodied social agents better than disembodied social agents?: The effects of physical embodiment, tactile interaction, and people’s loneliness in human–robot interaction. Int. J. Hum. Comput. Stud. 64, 962–973 (2006). 10.1016/j.ijhcs.2006.05.002 · DOI · Paper page
  18. [27]Keil, M. & Johnson, R. D. Feedback channels: Using social presence theory to compare voice mail to e-mail. J. Inf. Syst. Educ. 13 (4), 295–302 (2002).
  19. [31]Hassanein, K. & Head, M. Manipulating perceived social presence through the web interface and its impact on attitude towards online shopping. Int. J. Hum. Comput. Stud. 65, 689–708 (2007). 10.1016/j.ijhcs.2006.11.018 · DOI · Paper page
  20. [32]Rzepka, C. Examining the use of voice assistants: A value-focused thinking approach. In Twenty-fifth Americas Conference on Information Systems 1–10 (2019).
  21. [33]Xu, R., Feng, Y. & Chen, H. Chatgpt vs. google: A comparative study of search performance and user experience. arXiv Prepr. arXiv:2307.01135 (2023). · arXiv
  22. [34]Spatharioti, S. E., Rothschild, D. M., Goldstein, D. G. & Hofman, J. M. Comparing traditional and LLM-based search for consumer choice: A randomized experiment. arXiv Prepr. arXiv:2307.03744 (2023). · arXiv
  23. [35]Metzger, M. J. & Flanagin, A. J. Psychological approaches to credibility assessment online. In The handbook of the psychology of communication technology (ed. Sundar, S. S.) 445–466 (Wiley, 2015).
  24. [39]Dragojevic, M. & Giles, H. I don’t like you because you’re hard to understand: The role of processing fluency in the language attitudes process. Hum. Commun. Res. 42, 396–420 (2016). 10.1111/hcre.12079 · DOI
  25. [40]Levine, T. R. & Markowitz, D. M. The role of theory in researching and understanding human communication. Hum. Commun. Res. 50 (2), 154–161 (2024). 10.1093/hcr/hqad037 · DOI
  26. [41]Bullock, J. G., & Green, D. P. The failings of conventional mediation analysis and a design-based alternative. Adv. Methods Pract. Psychol. Sci.4(4). 10.1177/25152459211047227 (2021). · DOI
  27. [42]Bullock, J. G., Green, D. P. & Ha, S. E. Yes, but what’s the mechanism? (Don’t expect an easy answer). J. Pers. Soc. Psychol. 98 (4), 550–558. 10.1037/a0018933 (2010). 20307128 10.1037/a0018933 · DOI · Paper page
  28. [43]Green, D. P., Ha, S. E. & Bullock, J. G. Enough already about “black box” experiments: Studying mediation is more difficult than most scholars suppose. Ann. Am. Acad. Polit. Soc. Sci. 628, 200–208. 10.1177/0002716209351526 (2010). 10.1177/0002716209351526 · DOI
  29. [45]Augenstein, I. et al. Factuality challenges in the era of large language models. arXiv Prepr. arXiv2310.05189 (2023).
  30. [46]Lermann Henestrosa, A. & Kimmerle, J. Understanding and perception of automated text generation among the public: Two surveys with representative samples in Germany. Behav. Sci. 14 (5), 353 (2024). 38785844 10.3390/bs14050353 PMC11118015 · DOI
  31. [47]Abercrombie, G., Curry, A. C., Dinkar, T., Rieser, V. & Talat, Z. Mirages: On anthropomorphism in dialogue systems. arXiv Prep. arXiv:2305.09800 (2023). · arXiv
  32. [48]Chiesurin, S. et al. The dangers of trusting stochastic parrots: Faithfulness and trust in open-domain conversational question answering. arXiv Prep. arXiv:2305.16519. (2023). · arXiv
  33. [49]Wittenberg, C., Tappin, B. M., Berinsky, A. J. & Rand, D. G. The (minimal) persuasive advantage of political video over text. Proc. Natl. Acad. Sci. 118 (47), e2114388118 (2021). 34782473 10.1073/pnas.2114388118 PMC8617416 · DOI · Paper page
  34. [50]Girotra, K., Meincke, L., Terwiesch, C. & Ulrich, K. T. Ideas are dimes a dozen: Large language models for idea generation in innovation. Available SSRN (2023).
  35. [52]Lee, K. M., Peng, W., Jin, S.-A. & Yan, C. Can robots manifest personality?: An empirical test of personality recognition, social responses, and social presence in human–robot interaction. J. Commun. 56, 754–772 (2006). 10.1111/j.1460-2466.2006.00318.x · DOI
  36. [53]Hayes, A. F. Introduction to mediation, moderation, and conditional process analysis: A Regression-Based Approach. (Guilford Press, 2022).
  37. [54]Perugini, M., Gallucci, M. & Costantini, G. Safeguard power as a protection against imprecise power estimates. Perspect. Psychol. Sci. 9, 319–332 (2014). 26173267 10.1177/1745691614528519 · DOI

Cited by

Sorted by citations

Authors
Anderl, C., Klein, S. H., Sarigül, B., Schneider, F. M., Han, J., Fiedler, P. L., & Utz, S.
Affiliations
Leibniz-Institut für Wissensmedien (DE)
Corresponding author
  • Anderl, Christine
  • Utz, Sonja
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2024-07-25
Volume
14
Issue
1
Pages
17127-17127
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-024-67829-6
PMC
PMC11272919
OpenAlex
W4400985547
Cited by
38
References
13
Citations by year
2026: 21 · 2025: 13 · 2024: 4
FWCI
20.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
Funders
Leibniz-Institut für Wissensmedien
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Strategies and Epistemologies
Keywords
chatbots, large language models, ChatGPT, credibility judgments, misinformation, information search, presentation format, brand influence, misinformation detection, factual accuracy
MeSH terms
Adult, Communication, Female, Humans, Judgment, Language, Male, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review