Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

What large language models know and what people think they know

Open access
Steyvers, Mark, Tejeda, Heliodoro, Kumar, Aakriti, Belem, Catarina, Karny, Sheer, Hu, Xinyue, Mayer, Lukas W., Smyth, Padhraic · 2025 · Nature Machine Intelligence

Abstract

As artificial intelligence systems, particularly large language models (LLMs), become increasingly integrated into decision-making processes, the ability to trust their outputs is crucial. To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct. Whereas recent work has focused on LLMs’ internal confidence, less is understood about how effectively they convey uncertainty to users. Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models’ actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers. Our experiments with multiple-choice and short-answer questions reveal that users tend to overestimate the accuracy of LLM responses when provided with default explanations. Moreover, longer explanations increased user confidence, even when the extra length did not improve answer accuracy. By adjusting LLM explanations to better reflect the models’ internal confidence, both the calibration gap and the discrimination gap narrowed, significantly improving user perception of LLM accuracy. These findings underscore the importance of accurate uncertainty communication and highlight the effect of explanation length in influencing user trust in artificial-intelligence-assisted decision-making environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-024-00976-7

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Overview of the evaluation methodology for assessing the calibration gap between model confidence and human confidence in the model. The
Fig. 1 Overview of the evaluation methodology for assessing the calibration gap between model confidence and human confidence in the model. TheCC-BY · From the PDF
Fig. 2 Calibration error and discrimination for model confidence and human confidence across the behavioural experiments and LLMs. Calibration error is assessed by ECE (lower is better), while discrimination is assessed by AUC (higher is better). The vertical dashed lines represent the calibration and discrimination gap between model confidence and human confidence for unmodified
Fig. 2 Calibration error and discrimination for model confidence and human confidence across the behavioural experiments and LLMs. Calibration error is assessed by ECE (lower is better), while discrimination is assessed by AUC (higher is better). The vertical dashed lines represent the calibration and discrimination gap between model confidence and human confidence for unmodifiedCC-BY · From the PDF
Fig. 3 Calibration diagrams for model confidence and human confidence across experiments 1 and 2. The top and middle rows show results for multiplechoice questions with the GPT-3.5 and PaLM2 models, respectively. The bottom row shows results for short-answer questions with the GPT-4o model. The
Fig. 3 Calibration diagrams for model confidence and human confidence across experiments 1 and 2. The top and middle rows show results for multiplechoice questions with the GPT-3.5 and PaLM2 models, respectively. The bottom row shows results for short-answer questions with the GPT-4o model. TheCC-BY · From the PDF
Image 4
CC-BY · From the PDF
Image 5
CC-BY · From the PDF
reference, the dashed lines show the average human confidence for the baseline explanations in experiments 1a, 1b, and 1c. The error bars represent the 95% confidence interval of the mean across participants.
reference, the dashed lines show the average human confidence for the baseline explanations in experiments 1a, 1b, and 1c. The error bars represent the 95% confidence interval of the mean across participants.CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Image 8
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 36 references

Research works printed but not linked: 9

  1. Karelitz, T. M., Dhami, M. K., Budescu, D. V. & Wallsten, T. S. Toward a universal translator of verbal probabilities. In Proc. 15th International Florida Artificial Intelligence Research Society Conference (eds Haller, M. S. & Simmons, G.) 498–502 (AAAI Press, 2002).
  2. O’Brien, B. J. Words or numbers? The evaluation of probability expressions in general practice. J. R. Coll. Gen. Pract. 39, 98–100 (1989).
  3. Whalen, J. et al. ChatGPT: challenges, opportunities, and implications for teacher education. Contemp. Iss. Technol. Teach. Educ. 23, 1–23 (2023).
  4. Azaria, A. & Mitchell, T. in Findings of the Association for Computational Linguistics: EMNLP 2023 (eds Bouamor, H. et al.) 967–976 (ACL, 2023).
  5. Xiao, Y. et al. in Findings of the Association for Computational Linguistics: EMNLP 2022 (eds Goldberg, Y. et al.) 7273–7284 (ACL, 2022).
  6. Tanneru, S. H., Agarwal, C. & Lakkaraju, H. Quantifying uncertainty in natural language explanations of large language models. In International Conference on Artificial Intelligence and Statistics 1072–1080 (PMLR, 2024).
  7. Lin, S., Hilton, J. & Evans, O. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res. (2022).
  8. Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. In Proc. 34th International Conference on Machine Learning (eds Precup, D. & Teh, Y. W.) vol. 70 of Proceedings of Machine Learning Research 1321–1330 (PMLR, 2017).
  9. Kumar, A., Liang, P. S. & Ma, T. Verified uncertainty calibration. Adv. Neural Inf. Process. Syst. 32, (2019).

Cited by

See all 211

Metadata

Authors
Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. W., & Smyth, P.
Affiliations
University of California, Irvine (US)
Corresponding author
Steyvers, Mark
Venue
Nature Machine Intelligence
Published
2025-01-21
Volume
7
Issue
2
Pages
221-231
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4406679533
Cited by
209
References
36
Citations by year
2026: 147 · 2025: 60
FWCI
83.03
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1900644)
  • Hasso Plattner Institute (HPI) Research Center in Machine Learning and Data Science at the University of California, Irvine
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Keywords
large language models, uncertainty communication, user confidence, trust in AI, AI-assisted decision-making, confidence calibration, human-AI interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: What large language models know and what people think they know. https://app.cochat.ai/papers/10.1038/s42256-024-00976-7
The paper
Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. W., & Smyth, P. (2025). What large language models know and what people think they know. Nature Machine Intelligence. https://doi.org/10.1038/s42256-024-00976-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.