CoChat
Sign Up
PreprintCited by 33Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareClinical Reasoning and Diagnostic Skills

The DOI matches its DataCite record; its reference list is not available.

Medical Hallucinations in Foundation Models and Their Impact on Healthcare

Open access
Kim, Yubin, Jeong, Hyewon, Chen, Shan, Li, Shuyue Stella, Park, Chanwoo, Lu, Mingyu, Alhamoud, Kumail, Mun, Jimin, Grau, Cristina, Jung, Minseok, Gameiro, Rodrigo, Fan, Lizhou, Park, Eugene, Lin, Tristan, Yoon, Joonsik, Yoon, Wonjin, Sap, Maarten, Tsvetkov, Yulia, Liang, Paul, Xu, Xuhai, Liu, Xin, Park, Chunjong, Lee, Hyeonhoon, Park, Hae Won, McDuff, Daniel, Tulebaev, Samir, Breazeal, Cynthia·2025·arXiv
DOI 10.48550/arxiv.2503.05777arXiv 2503.05777
Open this paper in CoChat — free accountOpen PDF

Abstract

Hallucinations in foundation models arise from autoregressive training objectives that prioritize token-likelihood optimization over epistemic accuracy, fostering overconfidence and poorly calibrated uncertainty. We define medical hallucination as any model-generated output that is factually incorrect, logically inconsistent, or unsupported by authoritative clinical evidence in ways that could alter clinical decisions. We evaluated 11 foundation models (7 general-purpose, 4 medical-specialized) across seven medical hallucination tasks spanning medical reasoning and biomedical information retrieval. General-purpose models achieved significantly higher proportions of hallucination-free responses than medical-specialized models (median: 76.6% vs 51.3%, difference = 25.2%, 95% CI: 18.7-31.3%, Mann-Whitney U = 27.0, p = 0.012, rank-biserial r = -0.64). Top-performing models such as Gemini-2.5 Pro exceeded 97% accuracy when augmented with chain-of-thought prompting (base: 87.6%), while medical-specialized models like MedGemma ranged from 28.6-61.9% despite explicit training on medical corpora. Chain-of-thought reasoning significantly reduced hallucinations in 86.4% of tested comparisons after FDR correction (q < 0.05), demonstrating that explicit reasoning traces enable self-verification and error detection. Physician audits confirmed that 64-72% of residual hallucinations stemmed from causal or temporal reasoning failures rather than knowledge gaps. A global survey of clinicians (n = 70) validated real-world impact: 91.8% had encountered medical hallucinations, and 84.7% considered them capable of causing patient harm. The underperformance of medical-specialized models despite domain training indicates that safety emerges from sophisticated reasoning capabilities and broad knowledge integration developed during large-scale pre-training, not from narrow optimization.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.05777

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 33

Research works printed but not linked: 213

  1. [1][1] Angelopoulos, A.N., Bates, S.: A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (2022). https://arxiv.org/abs/2107.07511 - · arXiv
  2. [2][2] Addlesee, A.: Grounding llms to in-prompt instructions: reducing hallucinations caused by static pretraining knowledge. In: Joint International Conference on Computational Linguistics, Language Resources and Evaluation 2024, pp. 1-7 (2024). ELRA Language Resources Association -
  3. [3][3] Agarwal, V., Jin, Y., Chandra, M., De Choudhury, M., Kumar, S., Sastry, N.: Medhalu: Hallucinations in responses to healthcare queries by large language models. arXiv preprint arXiv:2409.19492 (2024) - · arXiv
  4. [4][4] Aydin, S., Karabacak, M., Vlachos, V., Margetis, K.: Large language models in patient education: a scoping review of applications in medicine. Frontiers in Medicine 11 , 1477898 (2024) -
  5. [5][5] AMA Journal of Ethics: Are current tort liability doctrines adequate for addressing injury caused by AI? AMA Journal of Ethics 21 (2), 160-166 (2019) https://doi.org/10.1001/amajethics.2019.160 - · DOI · Paper page
  6. [6][6] Alsentzer, E., Murphy, J.R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., McDermott, M.: Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 (2019) - · arXiv
  7. [7][7] Asgari, E., Montana-Brown, N., Dubois, M., Khalil, S., Balloch, J., Pimenta, D.: A framework to assess clinical safety and hallucination rates of llms for medical text summarisation. medRxiv, 2024-09 (2024) -
  8. [8][8] Asgari, E., Monta˜ na-Brown, N., Dubois, M., Khalil, S., Balloch, J., Au Yeung, J., Pimenta, D.: A framework to assess clinical safety and hallucination rates of llms for medical text summarisation. npj Digital Medicine 8 (274) (2025) https://doi.org/10.1038/s41746-025-01670-7 - · DOI · Paper page
  9. [10][10] American Medical Association: Augmented Intelligence in Health Care: AMA's AI Principles. Accessed: 2024-11-04 (2024). https://www.ama-assn.org/system/files/ama-ai-principles.pdf -
  10. [11][11] Asai, A., Min, S., Zhong, Z., Chen, D.: Retrieval-based language models and applications. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pp. 41-46 (2023) -
  11. [12][12] Agarwal, V., Pei, Y., Alamir, S., Liu, X.: CodeMirage: Hallucinations in Code Generated by Large Language Models (2024). https://arxiv.org/abs/2408.08333 - · arXiv
  12. [13][13] Abu-Salih, B., Al-Qurishi, M., Alweshah, M., Al-Smadi, M., Alfayez, R., Saadeh, H.: Healthcare knowledge graph construction: A systematic review of the state-of-the-art, open issues, and opportunities. Journal of Big Data 10 (1), 81 (2023) -
  13. [14][14] Ahmad, M.A., Yaramis, I., Roy, T.D.: Creating trustworthy llms: Dealing with hallucinations in healthcare ai. arXiv preprint arXiv:2311.01463 (2023) - · arXiv
  14. [15][15] Blumenthal-Barby, J.S., Krieger, H.: Cognitive biases and heuristics in medical decision making: A critical review using a systematic search strategy. Medical Decision Making 35 (4), 539-557 (2014) https://doi.org/ 10.1177/0272989X14547740 - · DOI
  15. [16][16] Bari, A., Khan, R.A., Rathore, A.W.: Medical errors; causes, consequences, emotional response and resulting behavioral change. Pakistan Journal of Medical Sciences 32 (3), 523-528 (2016) https://doi.org/10.12669/ pjms.323.9701 -
  16. [17][17] Borden, N., Linklater, D.: Hickam's dictum. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health 14 (2) (2013) -
  17. [18][18] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G.B., Lespiau, J.-B., Damoc, B., Clark, A., et al. : Improving language models by retrieving from trillions of tokens. In: International Conference on Machine Learning, pp. 2206-2240 (2022). PMLR -
  18. [20][20] Blumenthal, D., Patel, B.: The regulation of clinical artificial intelligence. NEJM AI 1 (8), 2400545 (2024) -
  19. [21][21] Brinkmann, R., Rosenberg, E., Louis, D.N., Podolsky, S.H.: Building a Community of Medical Learning-A Century of Case Records of the Massachusetts General Hospital in the Journal. Mass Medical Soc (2024) -
  20. [24][24] Bottomley, D., Thaldar, D.: Liability for harm caused by AI in healthcare: An overview of the core legal concepts. Frontiers in Pharmacology 14 , 1297353 (2023) https://doi.org/10.3389/fphar.2023.1297353 - · DOI · Paper page
  21. [25][25] Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., Shou, M.Z.: Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 (2024) - · arXiv
  22. [26][26] Chen, H., Abdin, Y., et al.: On the diversity of synthetic data and its impact on training large language models. arXiv preprint arXiv:2410.15226 (2024) - · arXiv
  23. [28][28] Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025) - · arXiv
  24. [29][29] Chen, Z., Cano, A.H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., K¨ opf, A., Mohtashami, A., et al.: Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 (2023) - · arXiv
  25. [32][32] Chen, S., Guevara, M., Moningi, S., Hoebers, F., Elhalawani, H., Kann, B.H., Chipidza, F.E., Leeman, J., Aerts, H.J., Miller, T., et al. : The effect of using a large language model to respond to patient messages. The Lancet Digital Health 6 (6), 379-381 (2024) -
  26. [33][33] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S.S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E.H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q.V., Wei, J.: Scaling instruction-finetuned language models. Journal of Machine Learning Research 25 (70), 1-53 (2024) -
  27. [34][34] Chandak, P., Huang, K., Zitnik, M.: Building a knowledge graph to enable precision medicine. Scientific Data 10 (1), 67 (2023) https://doi.org/10.1038/s41597-023-01960-3 - · DOI · Paper page
  28. [35][35] Chen, S., Kann, B.H., Foote, M.B., Aerts, H.J.W.L., Savova, G.K., Mak, R.H., Bitterman, D.S.: Use of artificial intelligence chatbots for cancer treatment information. JAMA Oncology 9 (10), 1459-1462 (2023) https://doi.org/10.1001/jamaoncol.2023.2954 - · DOI · Paper page
  29. [36][36] Chen, S., Kann, B.H., Foote, M.B., Aerts, H.J., Savova, G.K., Mak, R.H., Bitterman, D.S.: The utility of chatgpt for cancer treatment information. medrxiv. Preprint posted March 16 (2023) -
  30. [37][37] Christophe, C., Kanithi, P.K., Munjal, P., Raha, T., Hayat, N., Rajan, R., Al-Mahrooqi, A., Gupta, A., Salman, M.U., Gosal, G., et al.: Med42-evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches. arXiv preprint arXiv:2404.14779 (2024) - · arXiv
  31. [38][38] Chen, J., Kim, G., Sriram, A., Durrett, G., Choi, E.: Complex claim verification with evidence retrieved in the wild. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3569-3587 (2024) -
  32. [41][41] Camburu, O.-M., Rockt¨ aschel, T., Lukasiewicz, T., Blunsom, P.: e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems 31 (2018) -
  33. [42][42] Chen, I.Y., Szolovits, P., Ghassemi, M.: Can ai help reduce disparities in general medical and mental health care? AMA journal of ethics 21 (2), 167-179 (2019) -
  34. [43][43] Cao, S., Wang, Y., Li, J., He, X., Wang, L.: Calibration of pre-trained transformers. arXiv preprint arXiv:2106.07998 (2021) - · arXiv
  35. [44][44] Chen, J., Yang, D., Wu, T., Jiang, Y., Hou, X., Li, M., Wang, S., Xiao, D., Li, K., Zhang, L.: Detecting and evaluating medical hallucinations in large vision language models. arXiv preprint arXiv:2406.10185 (2024) - · arXiv
  36. [46][46] Silva, W., Costa Fonseca, L.C., Labidi, S., Lima Pacheco, J.C.: Mitigation of hallucinations in language models in education: A new approach of comparative and cross-verification. In: 2024 IEEE International Conference on Advanced Learning Technologies (ICALT), pp. 207-209 (2024). https://doi.org/10.1109/ ICALT61570.2024.00066 -
  37. [47][47] De Cao, N., Aziz, W., Titov, I.: Editing factual knowledge in language models. In: EMNLP 2021-2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pp. 6491-6506 (2021) -
  38. [48][48] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1423 . https://aclanthology. org/N19-1423 - · DOI · Paper page
  39. [49][49] Desai, S., Durrett, G.: Calibration of pre-trained transformers. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 295-302 (2020). https://doi.org/10. 18653/v1/2020.emnlp-main.20 . Association for Computational Linguistics -
  40. [50][50] DeepSeek-AI: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025). [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948) - · arXiv
  41. [51][51] Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., Weston, J.: Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023) - · arXiv
  42. [52][52] Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I.: Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325 (2023) - · arXiv
  43. [53][53] Dahl, M., Magesh, V., Suzgun, M., Ho, D.E.: Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis 16 (1), 64-93 (2024) https://doi.org/10.1093/jla/laae003 - · DOI · Paper page
  44. [55][55] Dou, C., Zhang, Y., Chen, Y., Jin, Z., Jiao, W., Zhao, H., Huang, Y.: Detection, diagnosis, and explanation: A benchmark for chinese medial hallucination evaluation. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 4784-4794 (2024) -
  45. [56][56] Eldakak, A., Alremeithi, A., Dahiyat, E., El-Gheriani, M., Mohamed, H., Abdulrahim Abdulla, M.I.: Civil liability for the actions of autonomous ai in healthcare: an invitation to further contemplation. Humanities and Social Sciences Communications 11 (1), 1-8 (2024) -
  46. [58][58] Finch, A., Hwang, Y.-S., Sumita, E.: Using machine translation evaluation techniques to determine sentencelevel semantic equivalence. In: Proceedings of the Third International Workshop on Paraphrasing (IWP2005) (2005) -
  47. [59][59] Farquhar, S., Kossen, J., Kuhn, L., Gal, Y.: Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), 625-630 (2024) -
  48. [60][60] Fanta, G.B., Pretorius, L.: Sociotechnical factors of sustainable digital health systems: A system dynamics model. Sustainable Futures (2023) https://doi.org/10.1016/j.sftr.2023.100127 - · DOI
  49. [61][61] Falke, T., Ribeiro, L.F., Utama, P.A., Dagan, I., Gurevych, I.: Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2214-2220 (2019) -
  50. [62][62] Feng, S., Shi, W., Wang, Y., Ding, W., Balachandran, V., Tsvetkov, Y.: Don't hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00367 (2024) - · arXiv
  51. [63][63] Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., Gurevych, I.: A survey of confidence estimation and calibration in large language models. arXiv preprint arXiv:2311.08298 (2023) - · arXiv
  52. [64][64] Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., Gurevych, I.: A survey of confidence estimation and calibration in large language models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6577-6595 (2024) -
  53. [65][65] Gema, A.P., Grabarczyk, D., De Wulf, W., Borole, P., Alfaro, J.A., Minervini, P., Vergari, A., Rajan, A.: Knowledge graph embeddings in the biomedical domain: Are they useful? a look at link prediction, rule learning, and downstream polypharmacy tasks. Bioinformatics Advances 4 (1), 097 (2024) -
  54. [66][66] Glicksberg, B.S.: Large language models in medicine: A review of current clinical trials and future prospects. PLOS Digital Health 3 (11), 0000662 (2024) -
  55. [69][69] Guerreiro, N.M., Voita, E., Martins, A.F.: Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1059-1075 (2023) -
  56. [71][71] Gu, Z., Yin, C., Liu, F., Zhang, P.: Medvh: Towards systematic evaluation of hallucination for large vision language models in the medical context. arXiv preprint arXiv:2407.02730 (2024) - · arXiv
  57. [72][72] Goodman, K.E., Yi, P.H., Morgan, D.J.: Ai-generated clinical summaries require more than accuracy. JAMA 331 (8), 637-638 (2024) https://doi.org/10.1001/jama.2024.0555 - · DOI
  58. [73][73] Huang, J., Chen, X., Mishra, S., Zheng, H.S., Yu, A.W., Song, X., Zhou, D.: Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798 (2023) - · arXiv
  59. [74][74] Hagendorff, T., Fabi, S., Kosinski, M.: Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science (2023) https://doi.org/10. 1038/s43588-023-00527-x . Brief Communication; Received: 17 February 2023; Accepted: 5 September 2023; Published online: 5 October 2023 -
  60. [75][75] Han, T., Kumar, A., Agarwal, C., Lakkaraju, H.: Towards safe large language models for medicine. In: ICML 2024 Workshop on Models of Human Feedback for AI Alignment (2024) -
  61. [76][76] Hou, B., Liu, Y., Qian, K., Andreas, J., Chang, S., Zhang, Y.: Decomposing uncertainty for large language models through input clarification ensembling. In: Forty-first International Conference on Machine Learning (2024) -
  62. [77][77] Hsieh, C., Moreira, C., Nobre, I.B., Sousa, S.C., Ouyang, C., Brereton, M., Jorge, J., Nascimento, J.C.: DALL-M: Context-Aware Clinical Data Augmentation with LLMs (2024). https://arxiv.org/abs/2407.08227 - · arXiv
  63. [78][78] Hammond, M.E.H., Stehlik, J., Drakos, S.G., Kfoury, A.G.: Bias in medicine: Lessons learned and mitigation strategies. JACC: Basic to Translational Science 6 (1), 78-85 (2021) https://doi.org/10.1016/j.jacbts.2020. 07.012 - · DOI
  64. [80][80] Hegselmann, S., Shen, S.Z., Gierse, F., Agrawal, M., Sontag, D., Jiang, X.: A data-centric approach to generate faithful and high quality patient summaries with large language models. arXiv preprint arXiv:2402.15422 (2024) - · arXiv
  65. [81][81] Hata, T., Shima, H., Nitta, M., Ueda, E., Nishihara, M., Uchiyama, K., Katsumata, T., Neo, M.: The relationship between duration of general anesthesia and postoperative fall risk during hospital stay in orthopedic patients. Journal of Patient Safety 18 (6), 922-927 (2022) https://doi.org/10.1097/PTS.0000000000001021 - · DOI
  66. [82][82] Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems (2023) -
  67. [83][83] Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems (2024) -
  68. [84][84] Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., Grave, E.: Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research 24 (251), 1-43 (2023) -
  69. [86][86] Jiang, Z., Araki, J., Ding, H., Neubig, G.: How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics 9 , 962-977 (2021) -
  70. [87][87] Jiang, Y., Chen, J., Yang, D., Li, M., Wang, S., Wu, T., Li, K., Zhang, L.: Comt: Chain-of-medical-thought reduces hallucination in medical report generation. arXiv preprint arXiv:2406.11451 (2024) - · arXiv
  71. [88][88] Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., Lu, X.: Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019) - · arXiv
  72. [89][89] Jin, Q., Kim, W., Chen, Q., Comeau, D.C., Yeganova, L., Wilbur, W.J., Lu, Z.: Medcpt: Contrastive pretrained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics 39 (11), 651 (2023) -
  73. [90][90] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Computing Surveys 55 (12), 1-38 (2023) https://doi.org/ 10.1145/3571730 - · DOI · Paper page
  74. [91][91] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Computing Surveys 55 (12), 1-38 (2023) -
  75. [93][93] Jalilian, L., McDuff, D., Kadambi, A.: The potential and perils of generative artificial intelligence for quality improvement and patient safety. arXiv preprint arXiv:2407.16902 (2024) - · arXiv
  76. [94][94] Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11 (14) (2021) https://doi.org/10.3390/app11146421 - · DOI · Paper page
  77. [96][96] Jia, F., Sontag, D., Agrawal, M.: Diagnosing our datasets: How does my language model learn clinical information? (2025). https://arxiv.org/abs/2505.15024 - · arXiv
  78. [97][97] Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., Fung, P.: Towards Mitigating Hallucination in Large Language Models via Self-Reflection (2023). https://arxiv.org/abs/2310.06271 - · arXiv
  79. [98][98] Kadavath, S., Conerly, T., Askell, A., Henighan, T.J., Drain, D., Perez, E., Schiefer, N., Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T.B., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., Kaplan, J.: Language models (mostly) know what they know. ArXiv abs/2207.05221 (2022) - · arXiv
  80. [99][99] Kim, H.-K.: The effects of artificial intelligence chatbots on women's health: A systematic review and metaanalysis. Healthcare 12 (5), 534 (2024) https://doi.org/10.3390/healthcare12050534 - · DOI
  81. [100][100] Kamath, A., Jia, R., Liang, P.: Selective question answering under domain shift. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684-5696. Association for Computational Linguistics, Online (2020). https: //doi.org/10.18653/v1/2020.acl-main.503 . https://aclanthology.org/2020.acl-main.503 - · DOI
  82. [101][101] Kim, Y., Jeong, H., Park, C., Park, E., Zhang, H., Liu, X., Lee, H., McDuff, D., Ghassemi, M., Breazeal, C., et al.: Tiered agentic oversight: A hierarchical multi-agent system for ai safety in healthcare. arXiv preprint arXiv:2506.12482 (2025) - · arXiv
  83. [102][102] Kang, H., Liu, X.-Y.: Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination (2023). https://arxiv.org/abs/2311.15548 - · arXiv
  84. [103][103] Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2005.14165 (2020) - · arXiv
  85. [104][104] Karimi, S., Metke-Jimenez, A., Kemp, M., Wang, C.: Cadec: A corpus of adverse drug event annotations. Journal of Biomedical Informatics 55 , 73-81 (2015) https://doi.org/10.1016/j.jbi.2015.03.010 . PMC free article - · DOI
  86. [105][105] Kazi, D.S., Martin, L.M., Litmanovich, D., Pinto, D.S., Clerkin, K.J., Zimetbaum, P.J., Dudzinski, D.M.: Case 18-2020: a 73-year-old man with hypoxemic respiratory failure and cardiac dysfunction. New England Journal of Medicine 382 (24), 2354-2364 (2020) -
  87. [106][106] Kalai, A.T., Nachum, O., Vempala, S.S., Zhang, E.: Why language models hallucinate. arXiv preprint arXiv:2509.04664 (2025) - · arXiv
  88. [107][107] Kim, Y., Park, C., Jeong, H., Chan, Y.S., Xu, X., McDuff, D., Lee, H., Ghassemi, M., Breazeal, C., Park, H.W.: MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (2024). https://arxiv. org/abs/2404.15155 -
  89. [108][108] Ke, Y., Yang, R., Lie, S.A., Lim, T.X.Y., Ning, Y., Li, I., Abdullah, H.R., Ting, D.S.W., Liu, N.: Mitigating cognitive biases in clinical decision-making through multi-agent conversations using large language models: Simulation study. Journal of Medical Internet Research 26 , 59439 (2024) -
  90. [109][109] Koopman, B., Zuccon, G.: Dr chatgpt tell me what i want to hear: How different prompts impact health answer correctness. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15012-15022 (2023) -
  91. [110][110] Lavrinovics, E., Biswas, R., Bjerva, J., Hose, K.: Knowledge graphs, large language models, and hallucinations: An nlp perspective. arXiv preprint arXiv:2411.14258 (2024) - · arXiv
  92. [111][111] Li, S.S., Balachandran, V., Feng, S., Ilgen, J.S., Pierson, E., Koh, P.W., Tsvetkov, Y.: Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024) -
  93. [112][112] Li, J., Chen, J., Ren, R., Cheng, X., Zhao, W.X., Nie, J.-Y., Wen, J.-R.: The dawn after the dark: An empirical study on factuality hallucination in large language models. arXiv preprint arXiv:2401.03205 (2024) - · arXiv
  94. [113][113] Lee, A.N., Hunter, C.J., Ruiz, N.: Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317 (2023) - · arXiv
  95. [114][114] Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al.: Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 (2023) - · arXiv
  96. [115][115] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.t., Rockt¨ aschel, T., et al. : Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 , 9459-9474 (2020) -
  97. [116][116] Lee, J., Park, S., Shin, J., Cho, B.: Analyzing evaluation methods for large language models in the medical field: a scoping review. BMC Medical Informatics and Decision Making 24 , 366 (2024) -
  98. [118][118] Ly, D.P., Shekelle, P.G., Song, Z.: Evidence for anchoring bias during physician decision-making. JAMA internal medicine 183 (8), 818-823 (2023) -
  99. [119][119] Liu, J., Wang, W., Ma, Z., Huang, G., SU, Y., Chang, K.-J., Chen, W., Li, H., Shen, L., Lyu, M.: Medchain: Bridging the gap between llm agents and clinical practice through interactive sequential benchmarking. arXiv preprint arXiv:2412.01605 (2024) - · arXiv
  100. [120][120] Li, Y., Yang, C., Ettinger, A.: When hindsight is not 20/20: Testing limits on reflective thinking in large language models. In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3741-3753. Association for Computational Linguistics, Mexico City, Mexico (2024). https://doi.org/10.18653/v1/2024.findings-naacl.237 . https://aclanthology.org/2024.findings-naacl. 237 - · DOI
  101. [121][121] Li, X., Zhao, R., Chia, Y.K., Ding, B., Joty, S., Poria, S., Bing, L.: Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In: International Conference on Learning Representations (ICLR) (2024). ICLR. https://arxiv.org/pdf/2305.13269 - · arXiv
  102. [122][122] Mishra, A., Asai, A., Balachandran, V., Wang, Y., Neubig, G., Tsvetkov, Y., Hajishirzi, H.: Fine-grained hallucination detection and editing for language models. arXiv preprint arXiv:2401.06855 (2024) - · arXiv
  103. [123][123] Meng, K., Bau, D., Andonian, A., Belinkov, Y.: Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems 35 , 17359-17372 (2022) -
  104. [124][124] Moradi, M., Blagec, K., Samwald, M.: Deep learning models are not robust against noise in clinical text. arXiv preprint arXiv:2108.12242 (2021) - · arXiv
  105. [125][125] Matos, J., Chen, S., Placino, S., Li, Y., Pardo, J.C.C., Idan, D., Tohyama, T., Restrepo, D., Nakayama, L.F., Pascual-Leone, J.M.M., Savova, G., Aerts, H., Celi, L.A., Wong, A.I., Bitterman, D.S., Gallifant, J.: WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation (2024). https://arxiv.org/abs/2410.12722 - · arXiv
  106. [126][126] Mehta, N., Devarakonda, M.V.: Machine learning, natural language programming, and electronic health records: The next step in the artificial intelligence journey? Journal of Allergy and Clinical Immunology 141 (6), 2019-20211 (2018) https://doi.org/10.1016/j.jaci.2018.03.017 - · DOI
  107. [127][127] McDermott, M., Dighe, A., Szolovits, P., Luo, Y., Baron, J.: Using machine learning to develop smart reflex testing protocols. Journal of the American Medical Informatics Association 31 (2), 416-425 (2024) https://doi.org/10.1093/jamia/ocad187 - · DOI
  108. [128][128] Mohammadi, A., Etemad, B., Zhang, X., Li, Y., Bedwell, G.J., Sharaf, R., Kittilson, A., Melberg, M., Crain, C.R., Traunbauer, A.K., Wong, C., Fajnzylber, J., Worrall, D.P., Rosenthal, A., Jordan, H., Jilg, N., Kaseke, C., Giguel, F., Lian, X., Deo, R., Gillespie, E., Chishti, R., Abrha, S., Adams, T., Li, J.Z.: Viral and host mediators of non-suppressible hiv-1 viremia. Nature Medicine 29 , 3212-3223 (2023) https: //doi.org/10.1038/s41591-023-02611-1 - · DOI
  109. [129][129] Mukherjee, S., Gamble, P., Ausin, M.S., Kant, N., Aggarwal, K., Manjunath, N., Datta, D., Liu, Z., Ding, J., Busacca, S., et al.: Polaris: A safety-focused llm constellation architecture for healthcare. arXiv preprint arXiv:2403.13313 (2024) - · arXiv
  110. [130][130] Mohri, C., Hashimoto, T.: Language models with conformal factuality guarantees. arXiv preprint arXiv:2402.10978 (2024) - · arXiv
  111. [131][131] Miles-Jay, A., Snitkin, E.S., Lin, M.Y., Shimasaki, T., Schoeny, M., Fukuda, C., Dangana, T., Moore, N., Sansom, S.E., Yelin, R.D., Bell, P., Rao, K., Keidan, M., Standke, A., Bassis, C., Hayden, M.K., Young, V.B.: Longitudinal genomic surveillance of carriage and transmission of clostridioides difficile in an intensive care unit. Nature Medicine 29 , 2526-2534 (2023) https://doi.org/10.1038/s41591-023-02549-4 - · DOI
  112. [132][132] Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P.W., Iyyer, M., Zettlemoyer, L., Hajishirzi, H.: Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In: The 2023 Conference on Empirical Methods in Natural Language Processing (2023) -
  113. [133][133] Mitchell, E., Lin, C., Bosselut, A., Manning, C.D., Finn, C.: Memory-based model editing at scale. In: International Conference on Machine Learning, pp. 15817-15831 (2022). PMLR -
  114. [134][134] Manes, I., Ronn, N., Cohen, D., Ber, R.I., Horowitz-Kugler, Z., Stanovsky, G.: K-qa: A real-world medical q&a benchmark. arXiv preprint arXiv:2401.14493 (2024) - · arXiv
  115. [135][135] Murphy, J.E., Shampain, K., Riley, L.E., Clark, J.W., Basnet, K.M.: Case 32-2018: A 36-year-old pregnant woman with newly diagnosed adenocarcinoma. New England Journal of Medicine 379 (16), 1562-1570 (2018) -
  116. [136][136] McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al.: Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164 (2023) - · arXiv
  117. [137][137] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al.: Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2024) -
  118. [138][138] Michalopoulos, G., Wang, Y., Kaka, H., Chen, H., Wong, A.: Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. arXiv preprint arXiv:2010.10391 (2020) - · arXiv
  119. [139][139] Mishra, P., Yao, Z., Vashisht, P., Ouyang, F., Wang, B., Mody, V.D., Yu, H.: Synfac-edit: Synthetic imitation edit feedback for factual alignment in clinical summarization. arXiv preprint arXiv:2402.13919 (2024) - · arXiv
  120. [141][141] Nazi, Z.A., Peng, W.: Large language models in healthcare and medical domain: A review. Informatics 11 (3) (2024) https://doi.org/10.3390/informatics11030057 - · DOI · Paper page
  121. [142][142] OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H.W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S.P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S.S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Kaiser, Kamali, A., Kanitscheider, I., Keskar, N.S., Khan, T., Kilpatrick, L., Kim, J.W., Kim, C., Kim, Y., Kirchner, J.H., Kiros, J., Knight, M., Kokotajlo, D., Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C.M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S.M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., M´ ely, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O'Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., Avila Belbute Peres, F., Petrov, M., Oliveira Pinto, H.P., Michael, Pokorny, Pokrass, M., Pong, V.H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F.P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M.B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J.F.C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J.J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., Zoph, B.: GPT-4 Technical Report (2024). https://arxiv.org/abs/2303.08774 - · arXiv
  122. [145][145] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. : Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 , 27730-27744 (2022) -
  123. [146][146] Pressman, S.M., Borna, S., Gomez-Cabello, C.A., Haider, S.A., Haider, C.R., Forte, A.J.: Clinical and surgical applications of large language models: A systematic review. Journal of Clinical Medicine 13 (11), 3041 (2024) https://doi.org/10.3390/jcm13113041 - · DOI · Paper page
  124. [148][148] Pradhan, S., Elhadad, N., Chapman, W., Manandhar, S., Savova, G.: SemEval-2014 task 7: Analysis of clinical text. In: Nakov, P., Zesch, T. (eds.) Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 54-62. Association for Computational Linguistics, Dublin, Ireland (2014). https://doi.org/10.3115/v1/S14-2007 . https://aclanthology.org/S14-2007/ - · DOI
  125. [149][149] Penkov, S.: Mitigating hallucinations in large language models via semantic enrichment of prompts: Insights from biobert and ontological integration. In: Proceedings of the Sixth International Conference on Computational Linguistics in Bulgaria (CLIB 2024), pp. 272-276 (2024) -
  126. [150][150] Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., Wang, W.Y.: Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188 (2023) - · arXiv
  127. [151][151] Pal, A., Umapathi, L.K., Sankarasubbu, M.: Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Conference on Health, Inference, and Learning, pp. 248-260 (2022). PMLR -
  128. [152][152] Pal, A., Umapathi, L.K., Sankarasubbu, M.: Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343 (2023) - · arXiv
  129. [153][153] Peng, C., Yang, X., Chen, A., Smith, K.E., PourNejatian, N., Costa, A.B., Martin, C., Flores, M.G., Zhang, Y., Magoc, T., Lipori, G., Mitchell, D.A., Ospina, N.S., Ahmed, M.M., Hogan, W.R., Shenkman, E.A., Guo, Y., Bian, J., Wu, Y.: A study of generative large language model for medical research and healthcare. npj Digital Medicine 6 (1) (2023) https://doi.org/10.1038/s41746-023-00958-w - · DOI · Paper page
  130. [154][154] Rodriguez, J.A., Alsentzer, E., Bates, D.W.: Leveraging large language models to foster equity in healthcare. Journal of the American Medical Informatics Association, 055 (2024) -
  131. [155][155] Rodman, A., Buckley, T.A., Manrai, A.K., Morgan, D.J.: Artificial intelligence vs. clinician performance in estimating probabilities of diagnoses before and after testing. JAMA Network Open 6 (12), 2347075 (2023) https://doi.org/10.1001/jamanetworkopen.2023.47075 - · DOI
  132. [156][156] Reddy, S.: Generative ai in healthcare: an implementation science informed translational path on application, integration and governance. Implementation Science 19 (1), 27 (2024) -
  133. [158][158] Rawte, V., Sheth, A., Das, A.: A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922 (2023) - · arXiv
  134. [159][159] Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024) -
  135. [160][160] Restrepo, D., Wu, C., V´ asquez-Venegas, C., Matos, J., Gallifant, J., Filipe, L.: Analyzing diversity in healthcare llm research: A scientometric perspective. arXiv preprint arXiv:2406.13152 (2024) - · arXiv
  136. [161][161] Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138 (2022) - · arXiv
  137. [162][162] Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. : Large language models encode clinical knowledge. Nature 620 (7972), 172-180 (2023) -
  138. [163][163] Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., Gallinari, P.: Questeval: Summarization asks for fact-based evaluation. In: 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6594-6604 (2021). Association for Computational Linguistics -
  139. [164][164] Svenstrup, D., Jørgensen, H.L., Winther, O.: Rare disease diagnosis: a review of web search, social media and large-scale data-mining approaches. Rare Diseases 3 (1), 1083145 (2015) -
  140. [165][165] Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al.: Medgemma technical report. arXiv preprint arXiv:2507.05201 (2025) - · arXiv
  141. [166][166] Scialom, T., Lamprier, S., Piwowarski, B., Staiano, J.: Answers unite! unsupervised metrics for reinforced summarization models. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3246-3256 (2019) -
  142. [167][167] Shekelle, P.G., Ortiz, E., Rhodes, S., Morton, S.C., Eccles, M.P., Grimshaw, J.M., Woolf, S.H.: Developing clinical guidelines. Western Journal of Medicine 176 (6), 326 (2002) -
  143. [168][168] Srivastava, A., Rastogi, A., Rao, A., Abid, A., Aslanides, J., Callison-Burch, C., Clark, C., Conerly, T., Das, D., Dey, K., et al.: Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022) - · arXiv
  144. [169][169] Saposnik, G., Redelmeier, D., Ruff, C.C., Tobler, P.N.: Cognitive biases associated with medical decisions: A systematic review. BMC Medical Informatics and Decision Making 16 , 138 (2016) https://doi.org/10.1186/ s12911-016-0377-1 -
  145. [170][170] Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al.: Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525 (2023) - · arXiv
  146. [171][171] Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., Schaekermann, M., Wang, A., Amin, M., Lachgar, S., Mansfield, P., Prakash, S., Green, B., Dominowska, E., Arcas, B.A., Tomasev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S.S., Barral, J., Webster, D., Corrado, G.S., Matias, Y., Azizi, S., Karthikesalingam, A., Natarajan, V.: Towards Expert-Level Medical Question Answering with Large Language Models (2023). https://arxiv.org/abs/2305.09617 - · arXiv
  147. [172][172] Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L., Smyth, P.: The calibration gap between model and human confidence in large language models. arXiv preprint arXiv:2401.13835 (2024) - · arXiv
  148. [173][173] Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., Chaves, J.Z., Hu, S.-Y., Schaekermann, M., Kamath, A., Cheng, Y., Barrett, D.G.T., Cheung, C., Mustafa, B., Palepu, A., McDuff, D., Hou, L., Golany, T., Liu, L., Alayrac, J.-b., Houlsby, N., Tomasev, N., Freyberg, J., Lau, C., Kemp, J., Lai, J., Azizi, S., Kanada, K., Man, S., Kulkarni, K., Sun, R., Shakeri, S., He, L., Caine, B., Webson, A., Latysheva, N., Johnson, M., Mansfield, P., Lu, J., Rivlin, E., Anderson, J., Green, B., Wong, R., Krause, J., Shlens, J., Dominowska, E., Eslami, S.M.A., Chou, K., Cui, C., Vinyals, O., Kavukcuoglu, K., Manyika, J., Dean, J., Hassabis, D., Matias, Y., Webster, D., Barral, J., Corrado, G., Semturs, C., Mahdavi, S.S., Gottweis, J., Karthikesalingam, A., Natarajan, V.: Capabilities of Gemini Models in Medicine (2024). https://arxiv.org/abs/2404.18416 - · arXiv
  149. [174][174] Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al.: Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416 (2024) - · arXiv
  150. [175][175] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017) - · arXiv
  151. [176][176] Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Safavi-Naini, S.A., Soroush, A., Chen, J.H.: Large language model uncertainty proxies: Discrimination and calibration for medical diagnosis and treatment. Journal of the American Medical Informatics Association 31 (10), 254 (2024) https://doi.org/10.1093/jamia/ ocae254 - · DOI
  152. [177][177] Sun, Y., Yin, Z., Guo, Q., Wu, J., Qiu, X., Zhao, H.: Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem (2024). https://arxiv.org/abs/2403.03558 - · arXiv
  153. [178][178] Sambara, S., Zhang, S., Banerjee, O., Acosta, J., Fahrner, J., Rajpurkar, P.: Radflag: A black-box hallucination detection method for medical vision language models. arXiv preprint arXiv:2411.00299 (2024) - · arXiv
  154. [180][180] Shi, X., Zhu, Z., Zhang, Z., Li, C.: Hallucination mitigation in natural language generation from largescale open-domain knowledge graphs. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12506-12521. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.emnlp-main.770 . https: //aclanthology.org/2023.emnlp-main.770 - · DOI
  155. [182][182] Tversky, A., Kahneman, D.: Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. Science 185 (4157), 1124-1131 (1974) https://doi.org/10.1126/ science.185.4157.1124 -
  156. [183][183] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: LLaMA: Open and Efficient Foundation Language Models (2023). https://arxiv.org/abs/2302.13971 - · arXiv
  157. [184][184] Tian, K., Mitchell, E., Yao, H., Manning, C.D., Finn, C.: Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08401 (2023) - · arXiv
  158. [185][185] Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., Manning, C.D.: Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975 (2023) - · arXiv
  159. [186][186] Topol, E.J.: High-performance medicine: the convergence of human and artificial intelligence. Nature medicine 25 (1), 44-56 (2019) -
  160. [187][187] Tjandra, B.A., Razzak, M., Kossen, J., Handa, K., Gal, Y.: Fine-tuning large language models to appropriately abstain with semantic entropy. arXiv preprint arXiv:2410.12345 (2024) - · arXiv
  161. [188][188] Tsiaras, S.V., Safi, L.M., Ghoshhajra, B.B., Lindsay, M.E., Wood, M.J.: Case 39-2017: A 41-year-old woman with recurrent chest pain. New England Journal of Medicine 377 (25), 2475-2484 (2017) -
  162. [189][189] Tonmoy, S., Zaman, S., Jain, V., Rani, A., Rawte, V., Chadha, A., Das, A.: A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313 (2024) - · arXiv
  163. [190][190] Umapathi, L.K., Pal, A., Sankarasubbu, M.: Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343 (2023) - · arXiv
  164. [193][193] U.S. Department of Health and Human Services: Health Information Privacy: Laws & Regulations. https:// www.hhs.gov/hipaa/for-professionals/privacy/laws-regulations/index.html. Accessed on November 4, 2024 (2024) -
  165. [200][200] Vasquez, C.D., Albeck, J.G.: Modeling elucidates context dependence in adipose regulation. Cell Systems (2023) https://doi.org/10.1016/j.cels.2023.11.008 - · DOI
  166. [201][201] Vally, Z.I., Khammissa, R.A.G., Feller, G., Ballyram, R., Beetge, M., Feller, L.: Errors in clinical diagnosis: a narrative review. The Journal of International Medical Research 51 (8), 3000605231162798 (2023) https: //doi.org/10.1177/03000605231162798 - · DOI
  167. [202][202] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017) - · arXiv
  168. [204][204] Willems, R., Annemans, L., Siopis, G., Moschonis, G., Vedanthan, R., Jung, J., Kwasnicka, D., Oldenburg, B., d'Antonio, C., Girolami, S., Agapidaki, E., Manios, Y., Verhaeghe, N., 4You, D.: Cost effectiveness review of text messaging, smartphone application, and website interventions targeting t2dm or hypertension. npj Digital Medicine 6 (2023) https://doi.org/10.1038/s41746-023-00876-x - · DOI
  169. [205][205] Wang, A., Cho, K., Lewis, M.: Asking and answering questions to evaluate the factual consistency of summaries. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5008-5020 (2020) -
  170. [206][206] Wang, Z., Danek, B., Yang, Z., Chen, Z., Sun, J.: Can large language models replace data scientists in clinical research? arXiv preprint arXiv:2410.21591 (2024) - · arXiv
  171. [207][207] Wu, J., Kim, Y., Keller, E.C., Chow, J., Levine, A.P., Pontikos, N., Ibrahim, Z., Taylor, P., Williams, M.C., Wu, H.: Exploring multimodal large language models for radiology report error-checking. arXiv preprint arXiv:2312.13103 (2023) - · arXiv
  172. [208][208] Wu, J., Kim, Y., Wu, H.: Hallucination benchmark in medical visual question answering. arXiv preprint arXiv:2401.05827 (2024) - · arXiv
  173. [209][209] Wang, S., Lin, M., Ghosal, T., Ding, Y., Peng, Y.: Knowledge graph applications in medical imaging analysis: a scoping review. Health data science 2022 , 9841548 (2022) -
  174. [210][210] Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., Liu, T.: Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975 (2023) - · arXiv
  175. [211][211] Wang, D., Liang, J., Ye, J., Li, J., Li, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z., et al. : Enhancement of the performance of large language models in diabetes education through retrieval-augmented generation: Comparative study. Journal of Medical Internet Research 26 , 58041 (2024) -
  176. [212][212] Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., Wang, Y.: Pmc-llama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, 045 (2024) -
  177. [213][213] Wen, B., Norel, R., Liu, J., Stappenbeck, T., Zulkernine, F., Chen, H.: Leveraging large language models for patient engagement: The power of conversational ai in digital health. arXiv preprint arXiv:2406.13659 (2024) - · arXiv
  178. [214][214] Wang, C., Ong, J., Wang, C., Ong, H., Cheng, R., Ong, D.: Potential for gpt technology to optimize future clinical decision-making using retrieval-augmented generation. Annals of Biomedical Engineering 52 (5), 1115-1118 (2024) -
  179. [215][215] Whitehead, S., Petryk, S., Shakib, V., Gonzalez, J., Darrell, T., Rohrbach, A., Rohrbach, M.: Reliable visual question answering: Abstain rather than answer incorrectly. In: European Conference on Computer Vision, pp. 148-166 (2022). Springer -
  180. [216][216] Wu, K., Wu, E., Cassasola, A., Zhang, A., Wei, K., Nguyen, T., Riantawan, S., Riantawan, P.S., Ho, D.E., Zou, J.: How well do llms cite relevant medical references? an evaluation framework and analyses. arXiv preprint arXiv:2402.02008 (2024) - · arXiv
  181. [217][217] Walley, A.Y., Wakeman, S.E., Eng, G.: Case 6-2019: A 29-year-old woman with nausea, vomiting, and diarrhea. New England Journal of Medicine 380 (8), 772-779 (2019) -
  182. [218][218] Wan, A., Wallace, E., Klein, D.: What evidence do language models find convincing? arXiv preprint arXiv:2402.11782 (2024) - · arXiv
  183. [219][219] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al. : Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 , 24824-24837 (2022) -
  184. [220][220] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., Anandkumar, A.: Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 (2023) - · arXiv
  185. [221][221] Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., Wang, L.L.: Know your limits: A survey of abstention in large language models. arXiv preprint arXiv:2407.18418 (2024) - · arXiv
  186. [222][222] Wang, R., Yang, Z., Zhao, Z., Tong, X., Hong, Z., Qian, K.: Llm-based robot task planning with exceptional handling for general purpose service robots. In: 2024 43rd Chinese Control Conference (CCC), pp. 4439-4444 (2024). IEEE -
  187. [223][223] Wang, D., Zhang, S.: Large language models in medical and healthcare fields: applications, advances, and challenges. Artificial Intelligence Review 57 , 299 (2024) -
  188. [224][224] Xiong, G., Jin, Q., Lu, Z., Zhang, A.: Benchmarking retrieval-augmented generation for medicine. In: Ku, L.W., Martins, A., Srikumar, V. (eds.) Findings of the Association for Computational Linguistics: ACL 2024, pp. 6233-6251. Association for Computational Linguistics, Bangkok, Thailand (2024). https://doi.org/10. 18653/v1/2024.findings-acl.372 . https://aclanthology.org/2024.findings-acl.372/ -
  189. [225][225] Xiong, G., Jin, Q., Wang, X., Zhang, M., Lu, Z., Zhang, A.: Improving retrieval-augmented generation in medicine with iterative follow-up questions. In: Biocomputing 2025: Proceedings of the Pacific Symposium, pp. 199-214 (2024). World Scientific -
  190. [226][226] Xia, W., Li, D., He, W., Pickhardt, P.J., Jian, J., Zhang, R., Zhang, J., Song, R., Tong, T., Yang, X., Gao, X., Cui, Y.: Multicenter evaluation of a weakly supervised deep learning model for lymph node diagnosis in rectal cancer at mri. Radiology: Artificial Intelligence (2024) https://doi.org/10.1148/ryai.230152 - · DOI
  191. [227][227] Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., Xu, W.: Knowledge conflicts for llms: A survey. arXiv preprint arXiv:2403.08319 (2024) - · arXiv
  192. [228][228] Xia, F., Yetisgen-Yildiz, M.: Clinical corpus annotation: challenges and strategies. In: Proceedings of the Third Workshop on Building and Evaluating Resources for Biomedical Text Mining (BioTxtM'2012) in Conjunction with the International Conference on Language Resources and Evaluation (LREC), Istanbul, Turkey, pp. 21-27 (2012) -
  193. [229][229] Xie, J., Zhang, K., Chen, J., Lou, R., Su, Y.: Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts. arXiv preprint arXiv:2305.13300 (2023) - · arXiv
  194. [230][230] Xie, S.M., Zhang, M., Huang, M., Shi, R., Guo, L., Peng, C., Yan, P., Zhou, Y., Qiu, X.: Calibrating the confidence of large language models by eliciting fidelity. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2024). https://doi.org/10.18653/v1/2024.emnlp-main. 173 . Association for Computational Linguistics - · DOI
  195. [231][231] Xu, D., Zhang, Z., Zhu, Z., Lin, Z., Liu, Q., Wu, X., Xu, T., Wang, W., Ye, Y., Zhao, X., et al. : Editing factual knowledge and explanatory ability of medical large language models. In: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2660-2670 (2024) -
  196. [232][232] Xu, D., Zhang, Z., Zhu, Z., Lin, Z., Liu, Q., Wu, X., Xu, T., Zhao, X., Zheng, Y., Chen, E.: Mitigating hallucinations of large language models in medical information extraction via contrastive decoding. arXiv preprint arXiv:2410.15702 (2024) - · arXiv
  197. [233][233] Xu, H., Zhu, Z., Zhang, S., Ma, D., Fan, S., Chen, L., Yu, K.: Rejection improves reliability: Training LLMs to refuse unknown questions using RL from knowledge feedback. In: First Conference on Language Modeling (2024). https://openreview.net/forum?id=lJMioZBoR8 -
  198. [234][234] Yu, L., Cao, M., Cheung, J.C.K., Dong, Y.: Mechanistic understanding and mitigation of language model non-factual hallucinations. arXiv preprint arXiv:2403.18167 (2024) - · arXiv
  199. [235][235] Yang, C., Cui, H., Lu, J., Wang, S., Xu, R., Ma, W., Yu, Y., Yu, S., Kan, X., Ling, C., et al.: A review on knowledge graphs for healthcare: Resources, applications, and promises. arXiv preprint arXiv:2306.04802 (2023) - · arXiv
  200. [236][236] Yogatama, D., Masson d'Autume, C., Kong, L.: Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics 9 , 362-373 (2021) -

200 of 213 shown

Metadata

Authors
Kim, Y., Jeong, H., Chen, S., Li, S. S., Park, C., Lu, M., Alhamoud, K., Mun, J., Grau, C., Jung, M., Gameiro, R., Fan, L., Park, E., Lin, T., Yoon, J., Yoon, W., Sap, M., Tsvetkov, Y., Liang, P., Xu, X., Liu, X., Park, C., Lee, H., Park, H. W., McDuff, D., Tulebaev, S., & Breazeal, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2503.05777
arXiv
2503.05777
OpenAlex
W6948557422
Cited by
33
References
0
Citations by year
2026: 17 · 2025: 16
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
foundation models, large language models, chain-of-thought prompting, uncertainty calibration, overconfidence, biomedical information retrieval, medical reasoning, clinical decision support, patient safety

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review