Large Language Model Uncertainty Measurement and Calibration for Medical Diagnosis and Treatment
Open accessAbstract
Introduction: The inability of Large Language Models (LLMs) to communicate uncertainty is a significant barrier to their use in medicine. Before LLMs can be integrated into patient care, the field must assess methods to measure uncertainty in ways that are useful to physician-users. Objective: Evaluate the ability for uncertainty metrics to quantify LLM confidence when performing diagnosis and treatment selection tasks by assessing the properties of discrimination and calibration. Methods: We examined Confidence Elicitation, Token Level Probability, and Sample Consistency metrics across GPT3.5, GPT4, Llama2 and Llama3. Uncertainty metrics were evaluated against three datasets of open-ended patient scenarios. Results: Sample Consistency methods outperformed Token Level Probability and Confidence Elicitation methods. Sample Consistency by Sentence Embedding achieved the highest discrimination performance (ROC AUC 0.68–0.79) with poor calibration, while Sample Consistency by GPT Annotation achieved the second-best discrimination (ROC AUC 0.66-0.74) with more accurate calibration. Nearly all uncertainty metrics had better discriminative performance with diagnosis rather than treatment selection questions. Furthermore, verbalized confidence (Confidence Elicitation) was found to consistently over-estimate model confidence. Conclusions: Sample Consistency is the most effective method for estimating LLM uncertainty of the metrics evaluated. Sample Consistency by Sentence Embedding can effectively estimate uncertainty if the user has a set of reference cases with which to re-calibrate their results, while Sample Consistency by GPT Annotation is more effective method if the user does not have reference cases and requires accurate raw calibration. Our results confirm LLMs are consistently over-confident when verbalizing their confidence through Confidence Elicitation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our results contradict recent work that suggests that consistency-based methods alleviate some concerns about overconfidence.
“While verbalized confidence is known to be overconfident in clinical question answering [38], our results contradict recent work that suggests that consistency-based methods alleviate some of these concerns [39].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [39]
Cites
Receiver Operating Characteristic Curve in Diagnostic Test Assessment
Mandrekar, J. N. (2010) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181ec173d · Cited by 4,506 · see citing works
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,320 · see citing works
Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · cited 1× · see citing works
“8 A key challenge of modern LLM systems is selectively deploying RAG only when a model is uncertain of an answer, otherwise extra context provided by RAG can errantly lead the model into an incorrect response.”
BackgroundSelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557 · see citing works
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449 · see citing works
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
Towards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311 · see citing works
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · see citing works
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265 · see citing works
Self-citationUse of gpt-4 to diagnose complex clinical cases
Eriksen, A. V., Möller, S., & Ryg, J. (2024) · NEJM AI · doi:10.1056/aip2300031 · Cited by 200 · see citing works
Chatbot vs medical student performance on free-response clinical reasoning examinations.
Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., & Chen, J. H. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2909 · Cited by 169 · see citing works
Self-citationLook Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language ModelsPreprint
Huang, Y., Song, J.-Y., Wang, Z., Chen, H.-M., & Ma, L. (2023) · arXiv.org · doi:10.48550/arxiv.2307.10236 · Cited by 144 · see citing works
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., & Manning, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.330 · Cited by 123 · see citing works
Capabilities of Gemini Models in Medicine
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.18416 · Cited by 117 · see citing works
Tackling prediction uncertainty in machine learning for healthcare
Chua, M., Kim, D., Choi, J., Lee, N. G., Deshpande, V., Schwab, J., Lev, M. H., Gonzalez, R. G., Gee, M. S., & Do, S. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00988-x · Cited by 114 · see citing works
Walsh, C. G., Sharman, K., & Hripcsak, G. M. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.10.008 · Cited by 102 · see citing works
Wu, S., Koo, M., Blum, L., Black, A., Kao, L., Fei, Z., Scalzo, F., & Kurtz, I. (2024) · NEJM AI · doi:10.1056/aidbp2300092 · Cited by 97 · see citing works
Shea, Y. F., Lee, C. M. Y., Ip, W. C., Luk, D. W. A., & Wong, S. S. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.25000 · Cited by 81 · see citing works
Large language models will not replace healthcare professionals: curbing popular fears and hype.
Thirunavukarasu, A. J. (2023) · Journal of the Royal Society of Medicine · doi:10.1177/01410768231173123 · Cited by 63 · see citing works
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
Zhou, K., Jurafsky, D., & Hashimoto, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.335 · Cited by 38 · see citing works
Rag and Rau: A Survey on Retrieval-Augmented Language Model in Natural Language ProcessingPreprint
Hu, Y., & Lu, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5015182 · Cited by 8 · see citing works
Decomposing Uncertainty for Large Language Models through Input Clarification EnsemblingPreprint
Hou, B., Liu, Y., Qian, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.08718 · Cited by 4 · see citing works
MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMsPreprint
Bakman, Y. F., Yaldiz, D. N., Buyukates, B., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.11756 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Socrates, V., Wright, D. S., Huang, T., Fereydooni, S., Dien, C., Chi, L., Albano, J., Patterson, B., Sasidhar Kanaparthy, N., Wright, C. X., et al. (2025) · JMIR Aging · doi:10.2196/69504 · Cited by 17 · see citing works
Trustworthy Medical Question Answering: An Evaluation-Centric Survey
Wang, Y., Wang, B., Mercer, R. E., Rudzicz, F., Roy, S. S., Ren, P., Chen, Z., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2506.03659
- Authors
- Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Ahmad Safavi-Naini, S. A., Soroush, A., & Chen, J. H.
- Affiliations
- Stanford Medicine (US)
- VA Palo Alto Health Care System (US)
- M. P. Shah Medical College (IN)
- Icahn School of Medicine at Mount Sinai (US)
- Corresponding author
- Savage, Thomas
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-06-07
- OpenAlex
- W4399423107
- Semantic Scholar
- 2285c7d0c84eb08b184e2d4ebab1f2f46647c28f
- Cited by
- 11
- References
- 29
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- medRxiv (acceptedVersion, cc-by-nc-nd)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- large language models, uncertainty estimation, calibration, sample consistency, diagnosis, treatment selection, overconfidence, sentence embeddings, GPT-4, Llama 3, Llama 2, GPT-3.5
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
