Cited by 92Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareExplainable Artificial Intelligence (XAI)

Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment

Open access
Savage, Thomas, Wang, John, Gallo, Robert, Boukil, Abdessalem, Patel, Vishwesh, Safavi-Naini, Seyed Amir Ahmad, Soroush, Ali, Chen, Jonathan H·2025·Journal of the American Medical Informatics Association

Abstract

Introduction: The inability of large language models (LLMs) to communicate uncertainty is a significant barrier to their use in medicine. Before LLMs can be integrated into patient care, the field must assess methods to estimate uncertainty in ways that are useful to physician-users. Objective: Evaluate the ability for uncertainty proxies to quantify LLM confidence when performing diagnosis and treatment selection tasks by assessing the properties of discrimination and calibration. Methods: We examined confidence elicitation (CE), token-level probability (TLP), and sample consistency (SC) proxies across GPT3.5, GPT4, Llama2, and Llama3. Uncertainty proxies were evaluated against 3 datasets of open-ended patient scenarios. Results: SC discrimination outperformed TLP and CE methods. SC by sentence embedding achieved the highest discriminative performance (ROC AUC 0.68-0.79), yet with poor calibration. SC by GPT annotation achieved the second-best discrimination (ROC AUC 0.66-0.74) with accurate calibration. Verbalized confidence (CE) was found to consistently overestimate model confidence. Discussion and Conclusions: SC is the most effective method for estimating LLM uncertainty of the proxies evaluated. SC by sentence embedding can effectively estimate uncertainty if the user has a set of reference cases with which to re-calibrate their results, while SC by GPT annotation is the more effective method if the user does not have reference cases and requires accurate raw calibration. Our results confirm LLMs are consistently over-confident when verbalizing their confidence (CE).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocae254

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Safavi-Naini, S. A. A., Soroush, A., & Chen, J. H.
Affiliations
  • Stanford Medicine (US)
  • VA Palo Alto Health Care System (US)
  • Linguamind AI , Sousse 4000,
  • M. P. Shah Medical College (IN)
  • Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai , New York, NY 10029,
  • Icahn School of Medicine at Mount Sinai (US)
Corresponding author
Savage, Thomas
Venue
Journal of the American Medical Informatics Association
Published
2024-10-12
Volume
32
Issue
1
Pages
139-149
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W4403362497
Cited by
92
References
17
Citations by year
2026: 69 · 2025: 21 · 2024: 2
FWCI
8.60
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • PubMed Central (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • NIH
  • National Institute of Allergy and Infectious Diseases (1R01AI17812101)
  • National Institute on Drug Abuse Clinical Trials Network (UG1DA015815—CTN-0136)
  • Gordon and Betty Moore Foundation (#12409)
  • Stanford Artificial Intelligence in Medicine and Imaging
  • Human-Centered Artificial Intelligence
  • Partnership Grant, Google Inc. Research Collaboration, American Heart Association
  • Strategically Focused Research Network
  • Diversity in Clinical Trials (UL1TR003142)
  • Department of Veterans Affairs
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, uncertainty estimation, calibration, discrimination (statistics), sample consistency, overconfidence, diagnosis, treatment selection, GPT-4, Llama 3, sentence embeddings
MeSH terms
Humans, Language, ROC Curve, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review