Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 122 of the 123 references it cites were found in a registry or an index; 54 of the 92 sources its claims cite were checked for support.

"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust

Open access
Kim, Sunnie S. Y., Liao, Q. Vera, Vorvoreanu, Mihaela, Ballard, Stephanie, Vaughan, Jennifer Wortman · 2024 · The 2024 ACM Conference on Fairness Accountability and Transparency

Abstract

Widely deployed large language models (LLMs) can produce convincing yet incorrect outputs, potentially misleading users who may rely on them as if they were correct. To reduce such overreliance, there have been calls for LLMs to communicate their uncertainty to end users. However, there has been little empirical work examining how users perceive and act upon LLMs’ expressions of uncertainty. We explore this question through a large-scale, pre-registered, human-subject experiment (N=404) in which participants answer medical questions with or without access to responses from a fictional LLM-infused search engine. Using both behavioral and self-reported measures, we examine how different natural language expressions of uncertainty impact participants’ reliance, trust, and overall task performance. We find that first-person expressions (e.g., “I’m not sure, but...”) decrease participants’ confidence in the system and tendency to agree with the system’s answers, while increasing participants’ accuracy. An exploratory analysis suggests that this increase can be attributed to reduced (but not fully eliminated) overreliance on incorrect answers. While we observe similar effects for uncertainty expressed from a general perspective (e.g., “It’s not clear, but...”), these effects are weaker and not statistically significant. Our findings suggest that using natural language expressions of uncertainty may be an effective approach for reducing overreliance on LLMs, but that the precise language used matters. This highlights the importance of user testing before deploying LLMs at scale.

Overview

CoChat Summary

TL;DR

In a pre-registered experiment (N=404), first-person LLM uncertainty expressions such as “I’m not sure, but...” reduced participants’ agreement with and confidence in an LLM-infused search system while increasing their accuracy on medical yes/no questions. General-perspective uncertainty expressions showed similar but weaker, non-significant effects.

Key findings

  • First-person uncertainty expressions decreased agreement with the AI system (74.8% vs. 80.9% Control), decreased confidence in the AI’s answer (3.66 vs. 3.95 on a 5-point scale), and increased accuracy (72.8% vs. 63.9% Control). General-perspective expressions showed weaker, non-significant effects (Agree 77.6%, ConfidenceAI 3.80, Correct 67.9%).
  • Within-condition, uncertain vs. not-uncertain AI responses lowered agreement (UncertainGeneral: 70.9% vs. 84.7%; Uncertain1st: 73.4% vs. 79.5%, not significant), lowered confidence in the AI answer (Uncertain1st: 3.44 vs. 3.88; UncertainGeneral: 3.65 vs. 3.95), lowered self-reported use of the AI answer (Uncertain1st: 57.4% vs. 73.2%; UncertainGeneral: 62.9% vs. 79.7%), and increased task time in the general condition (2.23 vs. 1.84 min).
  • Trust intentions were lower in Uncertain1st (2.91) than Control (3.25) and UncertainGeneral (3.36); there were no significant differences in TrustBelief, Anthropomorphism, or Transparency.
  • Exploratory analysis suggested reduced but not eliminated overreliance: uncertainty improved accuracy when the AI was incorrect (Uncertain1st: 43.6% to 52.0%; UncertainGeneral: 32.8% to 48.0%) while slightly reducing accuracy when the AI was correct (Uncertain1st: 92.2% to 89.2%; UncertainGeneral: 94.8% to 83.1%); No-AI participants still had higher accuracy (74.2%) than Control (63.9%).

Methods

  • Pre-registered between-subjects experiment with within-subjects comparisons; final sample N=404 U.S.-based MTurk participants (104 Control, 92 Uncertain1st, 94 UncertainGeneral, 114 No-AI). Participants answered eight challenging yes/no medical questions with or without responses from a fictional LLM-infused search engine, “AI system A.”
  • AI answers were correct for four of eight questions; in uncertainty conditions, half of responses included uncertainty expressed in first person or general perspective. Dependent variables included Agree, Correct, Time, LinkClick, UseAI, UseLink, UseInternet, ConfidenceAI, ConfidenceAnswer, TrustBelief, TrustIntention, Anthropomorphism, and Transparency.
  • Analyses used mixed models with participant and question random effects, ANOVA with Tukey post-hoc tests, within-condition models, exploratory over/underreliance analyses, and thematic analysis of free-form responses. AI responses were based on Copilot in Bing responses from July 2023, with minor modifications and uncertainty phrases inserted.

Limitations

  • The paper states that simple yes/no tasks limit generalizability to complex tasks; time and source-usage measures are less reliable than in a lab study; medical questions were predefined rather than personal symptoms; behavior may differ with repeated system interaction; the AI system had low accuracy and expressed uncertainty often in a poorly calibrated manner; and results may not generalize across cultures and languages because the study was in English with U.S.-based participants.
  • The paper also cautions against overgeneralizing and notes tradeoffs between experimental controllability and real-world generalizability, plus possible tradeoffs between reducing overreliance and decreasing trust.
  • Evident from the text: the Transparency scale had low internal consistency (Cronbach’s alpha = 0.21), and the discrepancy between LinkClick and UseLink suggests measurement or self-report issues; the sample was younger, more educated, and overrepresented white respondents compared with U.S. Census data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3630106.3658941

Cites

122 of 123 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Screenshot of the information-seeking task (shown in the Control, Uncertain1st, and UncertainGeneral conditions). Additional screenshots, including the variant shown in the No-AI condition, can be found in Appendix D.
Figure 1 Screenshot of the information-seeking task (shown in the Control, Uncertain1st, and UncertainGeneral conditions). Additional screenshots, including the variant shown in the No-AI condition, can be found in Appendix D.From the PDF
Image 2
From the PDF
Figure 2 Exploratory analysis of over- and underreliance (Section 5.1). We analyze Correct (%) separately for questions the system answered correctly vs. incorrectly. We show the model-estimated means and standard errors for each condition (Figure 2a) and for not uncertain vs. uncertain responses in the conditions with uncertainty (Figure 2b).
Figure 2 Exploratory analysis of over- and underreliance (Section 5.1). We analyze Correct (%) separately for questions the system answered correctly vs. incorrectly. We show the model-estimated means and standard errors for each condition (Figure 2a) and for not uncertain vs. uncertain responses in the conditions with uncertainty (Figure 2b).From the PDF
Figure D3: Task example page shown for participants in the Control , Uncertain1st , UncertainGeneral conditions.
Figure D3: Task example page shown for participants in the Control , Uncertain1st , UncertainGeneral conditions.From the PDF

Verification

Claims the paper makes

26 Valid · 4 Not valid · 10 Not checked · 107 Own results and interpretations

36 more claims · 3 with their cited sources checked

Create a free account to see all 147 claims and to verify the 8 references still open.

See all claims — free account
  • Probably not a substantial claim. 76
  • It may state two separate claims as one. 58
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 82 references

Research works printed but not linked: 3

  1. Jennifer M. Logg. 2017. Theory of Machine: When Do People Rely on Algorithms? (2017). Harvard Business School NOM Unit Working Paper No. 17-086.
  2. Samir Passi and Mihaela Vorvoreanu. 2022. Overreliance on AI: Literature Review. Technical Report MSR-TR-2022-12. Microsoft. https://www.microsoft.com/en-us/research/publication/overreliance-on-ai-literature-review/
  3. Benjamin Weiser and Nate Schweber. 2023. The ChatGPT Lawyer Explains Himself. New York Times (June 2023).

Cited by

See all 147

Metadata

Authors
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W.
Affiliations
  • Princeton University (US)
  • Microsoft (Canada) (CA)
  • Microsoft (United States) (US)
Venue
The 2024 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2024-06-03
Pages
822-835
OpenAlex
W4396802056
Cited by
147
References
82
Citations by year
2026: 83 · 2025: 58 · 2024: 5
FWCI
97.62
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • Princeton SEAS
  • Microsoft Research
  • NSF
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, uncertainty expression, overreliance, trust in AI, medical question answering, self-reported confidence, task performance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust. https://app.cochat.ai/papers/10.1145/3630106.3658941
The paper
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024). "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust. The 2024 ACM Conference on Fairness Accountability and Transparency. https://doi.org/10.1145/3630106.3658941

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.