Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment
Open accessAbstract
Introduction: The inability of large language models (LLMs) to communicate uncertainty is a significant barrier to their use in medicine. Before LLMs can be integrated into patient care, the field must assess methods to estimate uncertainty in ways that are useful to physician-users. Objective: Evaluate the ability for uncertainty proxies to quantify LLM confidence when performing diagnosis and treatment selection tasks by assessing the properties of discrimination and calibration. Methods: We examined confidence elicitation (CE), token-level probability (TLP), and sample consistency (SC) proxies across GPT3.5, GPT4, Llama2, and Llama3. Uncertainty proxies were evaluated against 3 datasets of open-ended patient scenarios. Results: SC discrimination outperformed TLP and CE methods. SC by sentence embedding achieved the highest discriminative performance (ROC AUC 0.68-0.79), yet with poor calibration. SC by GPT annotation achieved the second-best discrimination (ROC AUC 0.66-0.74) with accurate calibration. Verbalized confidence (CE) was found to consistently overestimate model confidence. Discussion and Conclusions: SC is the most effective method for estimating LLM uncertainty of the proxies evaluated. SC by sentence embedding can effectively estimate uncertainty if the user has a set of reference cases with which to re-calibrate their results, while SC by GPT annotation is the more effective method if the user does not have reference cases and requires accurate raw calibration. Our results confirm LLMs are consistently over-confident when verbalizing their confidence (CE).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sample consistency has been shown to outperform token level probability and confidence elicitation in the medical domain.
“Following Lyu et al., we perform uncertainty quantification employing sample consistency40, which has been shown to outperform token level probability and confidence elicitation in the medical domain41.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Receiver Operating Characteristic Curve in Diagnostic Test Assessment
Mandrekar, J. N. (2010) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181ec173d · Cited by 4,506 · see citing works
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,320 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Self-citationAlba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · see citing works
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
Towards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311 · see citing works
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · see citing works
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265 · see citing works
Self-citationChatbot vs medical student performance on free-response clinical reasoning examinations.
Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., & Chen, J. H. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2909 · Cited by 169 · see citing works
Self-citationLook Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language ModelsPreprint
Huang, Y., Song, J.-Y., Wang, Z., Chen, H.-M., & Ma, L. (2023) · arXiv.org · doi:10.48550/arxiv.2307.10236 · Cited by 144 · see citing works
Capabilities of Gemini Models in Medicine
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.18416 · Cited by 117 · see citing works
Tackling prediction uncertainty in machine learning for healthcare
Chua, M., Kim, D., Choi, J., Lee, N. G., Deshpande, V., Schwab, J., Lev, M. H., Gonzalez, R. G., Gee, M. S., & Do, S. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00988-x · Cited by 114 · see citing works
Walsh, C. G., Sharman, K., & Hripcsak, G. M. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.10.008 · Cited by 102 · see citing works
Shea, Y. F., Lee, C. M. Y., Ip, W. C., Luk, D. W. A., & Wong, S. S. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.25000 · Cited by 81 · see citing works
Large language models will not replace healthcare professionals: curbing popular fears and hype.
Thirunavukarasu, A. J. (2023) · Journal of the Royal Society of Medicine · doi:10.1177/01410768231173123 · Cited by 63 · see citing works
Kuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 52 · see citing works
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023) · arXiv · doi:10.48550/arxiv.2306.13063 · Cited by 52 · see citing works
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
Zhou, K., Jurafsky, D., & Hashimoto, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.335 · Cited by 38 · see citing works
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. J. F. (2023) · arXiv · doi:10.48550/arxiv.2303.08896 · Cited by 38 · see citing works
When to Retrieve: Teaching LLMs to Utilize Information Retrieval EffectivelyPreprint
Labruna, T., Campos, J. A., & Azkune, G. (2024) · Recent Advances in Natural Language Processing · doi:10.48550/arxiv.2404.19705 · Cited by 29 · see citing works
Benchmarking LLMs via Uncertainty QuantificationPreprint
Ye, F., Yang, M., Pang, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.12794 · Cited by 9 · see citing works
Decomposing Uncertainty for Large Language Models through Input Clarification EnsemblingPreprint
Hou, B., Liu, Y., Qian, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.08718 · Cited by 4 · see citing works
Rivera, M., Godbout, J., Rabbany, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.08694 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
Uncertainty-aware large language models for explainable disease diagnosis
Zhou, S., Wang, J., Xu, Z., Wang, S., Brauer, D., Welton, L., Cogan, J., Chung, Y.-H., Tian, L., Zhan, Z., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02071-6 · Cited by 27 · see citing works
Self-citationEverett, S., Bunning, B., Jain, P., López, I., Agarwal, A., Desai, M., Gallo, R. J., Goh, E., Kadiyala, V. B., Kanjee, Z., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02545-1 · Cited by 56 · see citing works
Self-citationThe perils of politeness: how large language models may amplify medical misinformation
Rosen, K. L., Sui, M., Heydari, K., Enichen, E. J., & Kvedar, J. C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02135-7 · Cited by 23 · see citing works
Safavi‐Naini, S. A. A., Ali, S., Shahab, O., Shahhoseini, Z., Savage, T., Rafiee, S., Samaan, J. S., Shabeeb, R. Q. A., Ladak, F., Yang, J. O., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02174-0 · Cited by 25 · see citing works
Self-citationSocrates, V., Wright, D. S., Huang, T., Fereydooni, S., Dien, C., Chi, L., Albano, J., Patterson, B., Sasidhar Kanaparthy, N., Wright, C. X., et al. (2025) · JMIR Aging · doi:10.2196/69504 · Cited by 17 · see citing works
Everett, S. S., Bunning, B. J., Jain, P., Lopez, I., Agarwal, A., Desai, M., Gallo, R., Goh, E., Kadiyala, V. B., Kanjee, Z., et al. (2025) · doi:10.1101/2025.06.07.25329176 · Cited by 18 · see citing works
Self-citationBentegeac, R., Guellec, B. L., Kuchcinski, G., Amouyel, P., & Hamroun, A. (2025) · Journal of Medical Internet Research · doi:10.2196/64348 · Cited by 19 · see citing works
A longitudinal analysis of declining medical safety messaging in generative AI models
Sharma, S., Alaa, A. M., & Daneshjou, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01943-1 · Cited by 21 · see citing works
Establishing best practices in large language model research: an application to repeat prompting
Gallo, R. J., Baiocchi, M., Savage, T., & Chen, J. H. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae294 · Cited by 14 · see citing works
Self-citationGao, Y., Myers, S., Chen, S., Dligach, D., Miller, T. A., Bitterman, D. S., Chen, G., Mayampurath, A. M., Churpek, M. M., & Afshar, M. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae154 · Cited by 18 · see citing works
Ghorbian, M., Ghobaei-Arani, M., & Ghorbian, S. (2025) · Methods · doi:10.1016/j.ymeth.2025.04.001 · Cited by 14 · see citing works
Effective prompt design for large language models in clinical practice
Callens, S. F. J. (2026) · Acta Clinica Belgica · doi:10.1080/17843286.2026.2613903 · Cited by 12 · see citing works
Employing large language models safely and effectively as a practicing neurosurgeon
Patil, A., Serrato, P., Cleaver, G., Limbania, D., See, A. P., & Huang, K. T. (2025) · Acta Neurochirurgica · doi:10.1007/s00701-025-06515-6 · Cited by 8 · see citing works
Farrokhi, M., Fallahian, A., Rahmani, E., Aghajan, A., Alipour, M., Khouzani, P. J., Hezarani, H. B., Sabzehie, H., Pirouzan, M., Pirouzan, Z., et al. (2025) · Archives of Academic Emergency Medicine · doi:10.22037/aaemj.v13i1.2712 · Cited by 5 · see citing works
Zayed, A. M., Frans, G., & Delvaux, N. (2025) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-0647 · Cited by 7 · see citing works
Pristoupil, J., Oleaga, L., Junquero, V., Merino, C., Özbek, S. S., Lambert, L., & European Society of Radiology (ESR) (2025) · European Radiology Experimental · doi:10.1186/s41747-025-00591-0 · Cited by 6 · see citing works
Naderi, N., Safavi‐Naini, S. A. A., Savage, T., Khalafi, M. A., Lewis, P. R., Atf, Z., Nadkarni, G. N., & Soroush, A. (2026) · npj gut and liver. · doi:10.1038/s44355-026-00053-3 · Cited by 8 · see citing works
Self-citationCeRTS: certainty retrieval token search in large language model clinical information extraction
Schimmelpfennig, L. E., Bhattarai, K., Oh, I. Y., Lever, J., Griffith, O. L., Griffith, M., Lai, A. M., & Abrams, Z. B. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104900 · Cited by 4 · see citing works
Nguyen, Q., Nguyen, H., Tang, Q.-H., Truong, T., Pham, V., Le, L., & Williams-King, D. (2025) · Eye Tracking Research & Application · doi:10.1145/3715669.3726785 · Cited by 3 · see citing works
Will large language models transform clinical prediction?
Yildiz, Y., Nenadić, G., Jani, M., & Jenkins, D. A. (2025) · Diagnostic and Prognostic Research · doi:10.1186/s41512-025-00211-w · Cited by 4 · see citing works
Uptake of Large Language Models by London Medical Students: Exploratory Qualitative Interview Study
Alazzawi, M., & Lam, K. K.-L. (2026) · JMIR Formative Research · doi:10.2196/82828 · Cited by 7 · see citing works
Schramm, S., Guellec, B. L., Topka, M., Svec, M., Backhaus, P., Eisenkolb, V. M., Riedel, E. O., Beyrle, M., Platzek, P., Ramschütz, C., et al. (2026) · Radiology · doi:10.1148/radiol.253477 · Cited by 4 · see citing works
Güler, I., Kraus, A., Grieb, G., Satır, T., Eberz, P., & Stelling, H. (2026) · Cancers · doi:10.3390/cancers18111750 · Cited by 3 · see citing works
Ahmadov, N., Muradova, A., & Aliyeva, A. (2026) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-026-10245-5 · Cited by 5 · see citing works
Boie, S. D., Reis, F., Frey, N., Grünewald, E., & Balzer, F. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02430-0 · Cited by 4 · see citing works
Lau, G. R., Kasturiratna, K. T. A. S., Goh, A. Y. H., Tong, E. M. W., & Hartanto, A. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101186 · Cited by 7 · see citing works
Uncertainty calibration in PHM model from classification to time-series prediction
Pan, J., Jiao, X., Jing, B., et al. (2026) · Measurement · doi:10.1016/j.measurement.2026.120433 · Cited by 2 · see citing works
When machines explain medicine: Nursing ethics and clinical communication
Yang, X., Liang, Y., Liu, Y., Liu, D., Yang, L., Zhang, B., Xu, C., Wu, Y., & He, J. (2026) · Nursing Ethics · doi:10.1177/09697330261449460 · Cited by 4 · see citing works
On-premise medical AI agents for reliable clinical decision-making
Zhang, L., Wölflein, G., Ferber, D., Liang, J., Carrero, Z. I., Wu, X., Vibert, J., Clusmann, J., Möhrmann, L., Möhrmann, E. E., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04609-x · Cited by 2 · see citing works
Structured reasoning failures compromise LLM interpretation of clinical oncology notes
Kenaston, M. W., Ayub, U., Parmar, M., Kumar, P., Anjum, M. U., Naqvi, S. A. A., Chaudhuri, A. A., Zakharia, Y., Heath, E. I., Bekaii-Saab, T. S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02951-5 · Cited by 2 · see citing works
Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs
Naderi, N., Atf, Z., Lewis, P. R., Mahjoubfar, A., Safavi‐Naini, S. A. A., & Soroush, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01399-6_5 · Cited by 1 · see citing works
Self-citationEun, D.-I., Lee, H.-C., Heo, G., Jeong, J. H., Lee, J., Lee, H., Park, S., Nam, K., Lee, H.-J., & Ju, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02231-8 · Cited by 1 · see citing works
Karaman, A. K., Yılmaz, B., Acar, K. H., Alizada, S., Korkmazer, B., Kartum, T. A., Samancı, C., Dağdelen, M., Çomunoğlu, N., Arslan, S. A., et al. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01793-z · Cited by 1 · see citing works
Gupta, A. K., Adams, L. C., & Rangarajan, K. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01904-4 · Cited by 1 · see citing works
Engineering framework for curiosity-driven and humble AI in clinical decision support
Arslan, J., Benke, K. K., Cajas, S. A., Castro, R., Celi, L. A., Cruz-Suarez, G. A., Reyes, R. D., Engelmann, J., Ercole, A., Hilel, A., et al. (2026) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2025-101877 · Cited by 1 · see citing works
Bayala, Y. L. T., Kaboré, F., Sougué, C., Ouédraogo, A., Zongo, E., Tiendrébeogo, W. J. S. Z., & Ouedraogo, D. (2026) · Health Informatics Journal · doi:10.1177/14604582261448687 · Cited by 2 · see citing works
Ethical Considerations in Personal Health Large Language Models
Liu, J., & Liu, S. (2026) · Journal of Medical Internet Research · doi:10.2196/92240 · Cited by 2 · see citing works
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 2 · see citing works
Naser, M. Z. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09938-4 · Cited by 2 · see citing works
Kang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1 · see citing works
Rule-Based Moral Principles for Explaining Uncertainty in Natural Language Generation
Atf, Z., & Lewis, P. R. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00018 · Cited by 1 · see citing works
The economics of accuracy for medical reasoning with large language models
Bhattacharyya, K., & Kamabattula, S. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001182
“This approach provides calibrated confidence (via agreement rates) while avoiding the overthinking trap and can be used to inform the confidence in the LLM response [29].”
An Information-Theoretic Perspective on Multi-LLM Uncertainty EstimationPreprint
Kruse, M., Afshar, M., Khatwani, S., Mayampurath, A. M., Chen, G., & Gao, Y. (2025) · medRxiv · doi:10.1101/2025.07.09.25331207
Shang, L., Chen, Y., Li, R., Zhang, X., Gao, M., Hou, Y., & Zhang, G. K. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000004062
Tiwari, A., & Gupta, V. (2026) · Research Square · doi:10.21203/rs.3.rs-8589677/v1
AI4Evidence Working Group of the Chinese Medical Doctor Association (2026) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003923
Arslan, J., Benke, K. K., Cajas, S. A., Castro, R., Celi, L. A., Cruz-Suarez, G. A., Reyes, R. D., Engelmann, J., Ercole, A., Hilel, A., et al. (2026) · medRxiv · doi:10.64898/2026.02.06.26345664
Ba, Z., He, M., He, H., Fu, Q., Lai, J. Z., Zhang, R., Diao, X., Liu, M., wang, Z., Wang, X., et al. (2026) · medRxiv · doi:10.64898/2026.02.08.26345860
Preconditions for responsible artificial intelligence adoption in psychiatry: commentary, Armitage
Armitage, R. C. (2026) · The British Journal of Psychiatry · doi:10.1192/bjp.2026.10560
Exploring probability-aware parametric softmax for model calibration in image classification
Zhang, S., Xie, L., & Chen, Y. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113488
Li, L., Zhao, S., Feng, S., Liu, W., & Zhang, Z. (2026) · Journal of Craniofacial Surgery · doi:10.1097/scs.0000000000012589
Markatou, M., Mukhopadhyay, R., Good, J., Tan, Y., Dharia, A., Kennedy, O., Brown, L. S., & Talal, A. H. (2026) · Journal of Medical Internet Research · doi:10.2196/97131
Makrygiannakis, M. A., Giannakopoulos, K., & Kaklamanos, E. G. (2026) · European Journal of Orthodontics · doi:10.1093/ejo/cjag010
Dashti, M., Khosraviani, F., & Golrang, E. (2026) · European Journal of Orthodontics · doi:10.1093/ejo/cjag009
Case Studies of Explainable LLMs in Diagnosis, Treatment Planning, and Patient Interaction
Tabatabaei, A., Gorji, M., & Rahimi, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-15088-2_8
Rao, A., Bali, A., Tran, V. V., Vasavada, S., Chu, J., Kanwal, F., Rusin, C., & El‐Serag, H. B. (2026) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2026.05.017
Benchmarking reliability and calibration of LLMs for multi-cancer early detection test communication
Takabatake, K., Rosito, M. S., Braun, D., Marinac, C. R., O’Donnell, E., & Parmigiani, G. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag093
Calibrated Abstention and Clinical Deferral in Small Language Models via RLVR and GRPOPreprint
Wibisono, N. B. (2026) · Research Square · doi:10.21203/rs.3.rs-10198691/v1
Haupt, M., Bischoff, A., Atoubi, M., Thomas, R. P., & Maurer, M. H. (2026) · Diagnostics · doi:10.3390/diagnostics16132131
Diagnostic performance of artificial intelligence in the syncope unit
van Zanten, S., Boel, T. T., de Jong, J. S. Y., Koomen, E. M., Bais, B., Dara, A., Giele, F., Geertsma, C., Sutton, R., Scheffer, M. G., et al. (2026) · Exploration of Cardiology · doi:10.37349/ec.2026.1012114
Yang, Z., Zhang, Y., Love, Z., et al. (2026) · medRxiv · doi:10.64898/2026.07.16.26358271
Amoei, M., Fraser, K., & Poenaru, D. (2026) · doi:10.2196/preprints.107982
Towards Robust Clinical Reasoning in LLM-aided Clinical Decision Support
Vuruma, S. K. R., Tang, S., & Wu, D. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00148
Evidence-Bound Generation for Medicines Information with Verifiable Claims and Visible Uncertainty
Ferraro, L., Ricci, M., (1560388), G. M., & Greco, P. (2026) · Archives of Pharmacy Practice · doi:10.51847/tzcg1ksqsz
Dan, W., Xi, S., Guo, C., Ran, X., & Qin, Z. S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7305416
高, 同. (2026) · Advances in Clinical Medicine · doi:10.12677/acm.2026.1682938
Hirosawa, T., Harada, Y., Kawamura, R., & Shimizu, T. (2026) · JMIR Formative Research · doi:10.2196/98819
Gu, X., Yang, Y., Lin, X., & Xu, C. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1934968
Language Models for Value-Informed Proxy Decision Support
Nolan, V. J., Brown, M. M., Balch, J. A., Katukota, A. V., Abbas, S., Devaguptapu, P., Baskaran, N., Abbott, K. L., Hong, P. K. W., Shickel, B., et al. (2026) · NEJM AI · doi:10.1056/aics2500562
Tilley, A., Murray, B., Henry, K., et al. (2026) · Journal of Managed Care & Specialty Pharmacy · doi:10.18553/jmcp.2026.32.9.1076
Lei, Q.-K., Wu, T.-X., Zhang, C.-L., & Zheng, Y. (2026) · Liver Research · doi:10.1016/j.livres.2026.08.001
JudgEHR: LLM-Guided Cohort Inference for Clinical Decision Confidence Estimation
Xin, K., Pelat, G., Vitale, J., Takeuchi, S., Berkovsky, S., & Beheshti, A. (2026) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh72779.2026.00046
Transforming large language models into medical specialists via knowledge injection
Kim, K., Song, J., Kim, D. Y., Nam, Y., Park, S., Shin, K., Jo, W., & Kim, N. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.103020
Yu, R., Peng, M., Li, H., Liu, S., Su, T., Guan, H., Shen, Y., Deng, Y., & Zhao, D. (2026) · Medicine · doi:10.1097/md.0000000000050485
Yin, L., Wang, Y., Dai, Z., et al. (2026) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-026-10589-y
Anderson, C. (2026) · medRxiv · doi:10.64898/2026.09.01.26361960
Potenzial und Herausforderungen von Large Language Models in der klinischen Entscheidungsfindung
Jungmann, F., & Hager, P. (2026) · Journal für Gastroenterologische und Hepatologische Erkrankungen · doi:10.1007/s41971-026-00265-7
Wang, G., Lin, X., & Yang, Y. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1937970
Crespo, J., Carballo, F., & Turnés, J. (2026) · doi:10.2196/preprints.112174
Wilson, E., Jong, F. D., & Peters, L. (2026) · Annals of Pharmacy Practice and Pharmacotherapy · doi:10.51847/xt3dqdt8kp
Avoiding common failures in AI for health and medicine
Salaudeen, O., Zhang, H., Kim, E., Singh, K., & Ghassemi, M. (2026) · Cell · doi:10.1016/j.cell.2026.07.025
Azmakan, H., & Azamakan, T. (2026) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2026.103524
The missing links in agentic AI autonomy
Omar, M., Abdulnour, R.-E. E., & Nadkarni, G. N. (2026) · Nature Medicine · doi:10.1038/s41591-026-04658-2
Pierotti, A. P., & Cremaschi, M. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1952196
Barone, M., Selvaggi, F., Pasquale, V., Nusrath, S., Vashist, Y. K., & Marano, L. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15207802
Geduk, G., Çapar, İ., & Hasırcı, U. C. (2026) · Diagnostics · doi:10.3390/diagnostics16203296
- Authors
- Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Safavi-Naini, S. A. A., Soroush, A., & Chen, J. H.
- Affiliations
- Stanford Medicine (US)
- VA Palo Alto Health Care System (US)
- Linguamind AI , Sousse 4000,
- M. P. Shah Medical College (IN)
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai , New York, NY 10029,
- Icahn School of Medicine at Mount Sinai (US)
- Corresponding author
- Savage, Thomas
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-10-12
- Volume
- 32
- Issue
- 1
- Pages
- 139-149
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC11648734
- OpenAlex
- W4403362497
- Semantic Scholar
- e7e79a1719da21ea3a60995e9039bbfaf9ca07b2
- Cited by
- 92
- References
- 17
- Citations by year
- 2026: 69 · 2025: 21 · 2024: 2
- FWCI
- 8.60
- Type
- journal article
- Language
- en
- Open access
- green
- pmc.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed (publishedVersion)
- Funders
- NIH
- National Institute of Allergy and Infectious Diseases (1R01AI17812101)
- National Institute on Drug Abuse Clinical Trials Network (UG1DA015815—CTN-0136)
- Gordon and Betty Moore Foundation (#12409)
- Stanford Artificial Intelligence in Medicine and Imaging
- Human-Centered Artificial Intelligence
- Partnership Grant, Google Inc. Research Collaboration, American Heart Association
- Strategically Focused Research Network
- Diversity in Clinical Trials (UL1TR003142)
- Department of Veterans Affairs
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- large language models, uncertainty estimation, calibration, discrimination (statistics), sample consistency, overconfidence, diagnosis, treatment selection, GPT-4, Llama 3, sentence embeddings
- MeSH terms
- Humans, Language, ROC Curve, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review