CoChat
Sign Up
PreprintCited by 6Machine Learning in HealthcareArtificial Intelligence in Healthcare and EducationArtificial Intelligence in Healthcare

The DOI matches its DataCite record.

ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

Open access
Chen, Canyu, Yu, Jian, Chen, Shan, Liu, Che, Wan, Zhongwei, Zhou, Shuang, Luo, Yuan, Zhang, Rui, Bitterman, Danielle, Wang, Fei, Shu, Kai·2024·arXiv
DOI 10.48550/arxiv.2411.06469arXiv 2411.06469
Open this paper in CoChat — free account

Abstract

Large Language Models (LLMs) hold great promise to revolutionize current clinical systems for their superior capacities on medical text processing tasks and medical licensing exams. Meanwhile, traditional ML models such as SVM and XGBoost have still been mainly adopted in clinical prediction tasks. An emerging question is: Can LLMs beat traditional ML models in clinical prediction? Thus, we build a new benchmark ClinicalBench to comprehensively study the clinical predictive modeling capacities of both general-purpose and medical LLMs, and compare them with traditional ML models. ClinicalBench embraces three common clinical prediction tasks, two databases, 14 general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through extensive empirical investigation, we discover that both general-purpose and medical LLMs, even with different model scales, diverse prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet, shedding light on their potential deficiency in clinical reasoning and decision-making. We call for caution when practitioners adopt LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap between LLMs' development for healthcare and real-world clinical practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.06469

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • We utilize Llama-3 as the specialist agent for tabular data because it is among the strongest open LLMs in a tabular EHR benchmark study.

    “In our study, we utilize Llama-3 as the specialist agent for tabular data as it is among strongest open LLMs in a tabular EHR benchmark study80”

    From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [80]

    ClinicalBench benchmarks Llama3-8B and Llama3-70B on clinical prediction tasks using MIMIC data (Introduction, p. 2; Reproducibility Statement, p. 14). It does not establish Llama-3 as among the strongest open LLMs in that benchmark. Indeed, the results describe Llama3-8B’s mortality-prediction performance as comparable to a majority- or minority-class baseline (Section 3, p. 5). (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 82

  1. [1][1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv preprint abs/2404.14219 (2024). https://arxiv.org/ abs/2404.14219 - · arXiv
  2. [2][2] Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering. ArXiv preprint abs/2404.05590 (2024). https://arxiv.org/abs/2404.05590 - · arXiv
  3. [3][3] Arkaitz Artetxe, Andoni Beristain, and Manuel Grana. 2018. Predictive models for hospital readmission risk: A systematic review of methods. Computer methods and programs in biomedicine 164 (2018), 49-64. -
  4. [4][4] André Bauer, Simon Trapp, Michael Stenger, Robert Leppich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian Foster. 2024. Comprehensive exploration of synthetic data generation: A survey. ArXiv preprint abs/2401.02524 (2024). https://arxiv.org/abs/2401.02524 - · arXiv
  5. [5][5] Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. Medbench: A large-scale chinese benchmark for evaluating medical large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 17709-17717. -
  6. [6][6] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. ArXiv preprint abs/2403.17297 (2024). https://arxiv.org/abs/2403.17297 - · arXiv
  7. [7][7] Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2024. Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions. ArXiv preprint abs/2402.18060 (2024). https://arxiv.org/abs/2402.18060 - · arXiv
  8. [8][8] Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, et al. 2024. From persona to personalization: A survey on role-playing language agents. ArXiv preprint abs/2404.18231 (2024). https://arxiv.org/abs/2404.18231 - · arXiv
  9. [9][9] Qingyu Chen, Jingcheng Du, Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Rui Zhang, Zhiyong Lu, and Hua Xu. 2023. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. ArXiv preprint abs/2305.16326 (2023). https://arxiv.org/abs/2305. 16326 - · arXiv
  10. [10][10] Xuanzhong Chen, Xiaohao Mao, Qihan Guo, Lun Wang, Shuyang Zhang, and Ting Chen. 2024. RareBench: Can LLMs Serve as Rare Diseases Specialists?. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 4850-4861. -
  11. [11][11] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. 2023. Meditron-70b: Scaling medical pretraining for large language models. ArXiv preprint abs/2311.16079 (2023). https://arxiv. org/abs/2311.16079 - · arXiv
  12. [12][12] Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. Med42-v2: A suite of clinical llms. ArXiv preprint abs/2408.06142 (2024). https://arxiv.org/abs/2408.06142 - · arXiv
  13. [13][13] Suhan Cui, Jiaqi Wang, Yuan Zhong, Han Liu, Ting Wang, and Fenglong Ma. 2024. Automated fusion of multimodal electronic health records for better medical predictions. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM) . SIAM, 361-369. -
  14. [14][14] Trisha Das, Zifeng Wang, and Jimeng Sun. 2023. Twin: Personalized clinical trial digital twin generation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 402-413. -
  15. [15][15] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2023. A survey on in-context learning. ArXiv preprint abs/2301.00234 (2023). https://arxiv.org/abs/2301.00234 - · arXiv
  16. [16][16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. ArXiv preprint abs/2407.21783 (2024). https://arxiv.org/abs/2407.21783 - · arXiv
  17. [17][17] Hui Feng, Francesco Ronzano, Jude LaFleur, Matthew Garber, Rodrigo de Oliveira, Kathryn Rough, Katharine Roth, Jay Nanavati, Khaldoun Zine El Abidine, and Christina Mack. 2024. Evaluation of Large Language Model Performance on the Biomedical Language Understanding and Reasoning Benchmark: Comparative Study. medRxiv (2024), 2024-05. -
  18. [18][18] Jack Gallifant, Shan Chen, Pedro Moreira, Nikolaj Munch, Mingye Gao, Jackson Pond, Leo Anthony Celi, Hugo Aerts, Thomas Hartvigsen, and Danielle Bitterman. 2024. Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks. ArXiv preprint abs/2406.12066 (2024). https://arxiv.org/abs/2406. 12066 - · arXiv
  19. [19][19] Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR medical education 9, 1 (2023), e45312. -
  20. [20][20] Maxime Griot, Coralie Hemptinne, Jean Vanderdonckt, and Demet Yuksel. 2024. Impact of high-quality, mixed-domain data on the performance of medical language models. Journal of the American Medical Informatics Association 31, 9 (2024), 1875-1883. arXiv:https://academic.oup.com/jamia/articlepdf/31/9/1875/58868289/ocae120.pdf doi:10.1093/jamia/ocae120 - · DOI
  21. [21][21] Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine 30, 9 (2024), 2613-2622. -
  22. [22][22] Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. 2024. Towards safe and aligned large language models for medicine. ArXiv preprint abs/2403.03744 (2024). https://arxiv.org/abs/2403.03744 - · arXiv
  23. [23][23] Joshua Harris, Timothy Laurence, Leo Loman, Fan Grayson, Toby Nonnenmacher, Harry Long, Loes WalsGriffith, Amy Douglas, Holly Fountain, Stelios Georgiou, et al. 2024. Evaluating Large Language Models for Public Health Classification and Extraction Tasks. ArXiv preprint abs/2405.14766 (2024). https://arxiv.org/ abs/2405.14766 - · arXiv
  24. [24][24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net. https: //openreview.net/forum?id=d7KBjmI3GmQ -
  25. [25][25] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net. https://openreview.net/forum?id=nZeVKeeFYf9 -
  26. [26][26] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. ArXiv preprint abs/2404.06395 (2024). https://arxiv.org/abs/2404.06395 - · arXiv
  27. [27][27] Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. 2024. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks. Computers in biology and medicine 171 (2024), 108189. -
  28. [28][28] Jacob C Jentzer, Anthony H Kashou, Francisco Lopez-Jimenez, Zachi I Attia, Suraj Kapa, Paul A Friedman, and Peter A Noseworthy. 2021. Mortality risk stratification using artificial intelligence-augmented electrocardiogram in cardiac intensive care unit patients. European Heart Journal Acute Cardiovascular Care 10, 5 (2021), 532-541. -
  29. [29][29] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. ArXiv preprint abs/2310.06825 (2023). https://arxiv.org/abs/2310.06825 - · arXiv
  30. [30][30] Pengcheng Jiang, Cao Xiao, Adam Richard Cross, and Jimeng Sun. 2024. GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs. In The Twelfth International Conference on Learning Representations . https: //openreview.net/forum?id=tVTN7Zs0ml -
  31. [31][31] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421. -
  32. [32][32] Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M Cheung, Robert Chen, Ronald M Summers, Justin F Rousseau, Peiyun Ni, Marc J Landsman, et al. 2024. Hidden flaws behind expert-level accuracy of gpt-4 vision in medicine. ArXiv preprint abs/2401.08396 (2024). https://arxiv.org/abs/2401.08396 - · arXiv
  33. [34][34] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific data 3, 1 (2016), 1-9. -
  34. [35][35] Zahir Kanjee, Byron Crowe, and Adam Rodman. 2023. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. Jama 330, 1 (2023), 78-80. -
  35. [36][36] Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023. Evaluating gpt-4 and chatgpt on japanese medical licensing examinations. ArXiv preprint abs/2303.18027 (2023). https://arxiv.org/abs/2303.18027 - · arXiv
  36. [37][37] Evangelia Katsoulakis, Qi Wang, Huanmei Wu, Leili Shahriyari, Richard Fletcher, Jinwei Liu, Luke Achenie, Hongfang Liu, Pamela Jackson, Ying Xiao, et al. 2024. Digital twins for health: a scoping review. NPJ Digital Medicine 7, 1 (2024), 77. -
  37. [38][38] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199-22213. -
  38. [39][39] Karolina Korgul, Andrew M Bean, Felix Krones, Robert McCraith, and Adam Mahdi. 2023. Exploring the landscape of large language models in medical question answering: Observations and open questions. ArXiv preprint abs/2310.07225 [(2023). https://arxiv.org/abs/2310.07225](https://arxiv.org/abs/2310.07225) - · arXiv
  39. [40][40] Sunjun Kweon, Byungjin Choi, Minkyu Kim, Rae Woong Park, and Edward Choi. 2024. KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations. ArXiv preprint abs/2403.01469 (2024). https://arxiv.org/abs/2403.01469 - · arXiv
  40. [41][41] Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. ArXiv preprint abs/2402.10373 (2024). https://arxiv.org/abs/2402.10373 - · arXiv
  41. [42][42] Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. 2024. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems 36 (2024). -
  42. [43][43] Lei Liu, Xiaoyan Yang, Junchi Lei, Xiaoyang Liu, Yue Shen, Zhiqiang Zhang, Peng Wei, Jinjie Gu, Zhixuan Chu, Zhan Qin, et al. 2024. A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions. ArXiv preprint abs/2406.03712 (2024). https://arxiv.org/abs/2406.03712 - · arXiv
  43. [44][44] Mianxin Liu, Jinru Ding, Jie Xu, Weiguo Hu, Xiaoyang Li, Lifeng Zhu, Zhian Bai, Xiaoming Shi, Benyou Wang, Haitao Song, et al. 2024. MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models. ArXiv preprint abs/2407.10990 (2024). https://arxiv.org/abs/2407.10990 - · arXiv
  44. [45][45] Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. 2024. Best Practices and Lessons Learned on Synthetic Data. In First Conference on Language Modeling . https://openreview.net/forum?id=OJaWBhh61C -
  45. [48][48] Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2023. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. ArXiv preprint abs/2308.09442 (2023). https://arxiv.org/abs/2308. 09442 - · arXiv
  46. [49][49] Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. 2023. Towards accurate differential diagnosis with large language models. ArXiv preprint abs/2312.00164 (2023). https://arxiv.org/abs/2312.00164 - · arXiv
  47. [50][50] Sobhan Moazemi, Sahar Vahdati, Jason Li, Sebastian Kalkhoff, Luis JV Castano, Bastian Dewitz, Roman Bibo, Parisa Sabouniaghdam, Mohammad S Tootooni, Ralph A Bundschuh, et al. 2023. Artificial intelligence for clinical decision support for monitoring patients in cardiovascular ICUs: a systematic review. Frontiers in Medicine 10 (2023), 1109411. -
  48. [51][51] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning . PMLR, 248260. -
  49. [52][52] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. ArXiv preprint abs/2308.03188 (2023). https://arxiv.org/abs/2308.03188 - · arXiv
  50. [53][53] Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. 2018. Benchmarking deep learning models on large healthcare datasets. Journal of biomedical informatics 83 (2018), 112-134. -
  51. [54][54] Zi-Hao Qiu, Siqi Guo, Mao Xu, Tuo Zhao, Lijun Zhang, and Tianbao Yang. 2024. To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO. ArXiv preprint abs/2404.04575 (2024). https://arxiv.org/abs/2404.04575 - · arXiv
  52. [55][55] Alvin Rajkomar, Jeffrey Dean, and Isaac Kohane. 2019. Machine learning in medicine. New England Journal of Medicine 380, 14 (2019), 1347-1358. -
  53. [56][56] Matthew Renze and Erhan Guven. 2024. The effect of sampling temperature on problem solving in large language models. ArXiv preprint abs/2402.05201 (2024). https://arxiv.org/abs/2402.05201 - · arXiv
  54. [57][57] Maciej Rosoł, Jakub S Gąsior, Jonasz Łaba, Kacper Korzeniewski, and Marcel Młyńczak. 2023. Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. Scientific Reports 13, 1 (2023), 20512. -
  55. [58][58] Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. ArXiv preprint abs/2402.07927 (2024). https://arxiv.org/abs/2402.07927 - · arXiv
  56. [59][59] Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, et al. 2024. The Prompt Report: A Systematic Survey of Prompting Techniques. ArXiv preprint abs/2406.06608 (2024). https://arxiv.org/abs/2406.06608 - · arXiv
  57. [60][60] Ofir Ben Shoham and Nadav Rappoport. 2024. MedConceptsQA-Open Source Medical Concepts QA Benchmark. ArXiv preprint abs/2405.07348 (2024). https: //arxiv.org/abs/2405.07348 - · arXiv
  58. [61][61] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature 620, 7972 (2023), 172-180. -
  59. [62][62] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023. Towards expert-level medical question answering with large language models. ArXiv preprint abs/2305.09617 (2023). https://arxiv.org/abs/2305.09617 - · arXiv
  60. [63][63] Kieran Stone, Reyer Zwiggelaar, Phil Jones, and Neil Mac Parthaláin. 2022. A systematic review of the prediction of hospital length of stay: Towards a unified framework. PLOS Digital Health 1, 4 (2022), e0000017. -
  61. [64][64] Tianze Sun, Xiwang He, and Zhonghai Li. 2023. Digital twin in healthcare: Recent updates and challenges. Digital Health 9 (2023), 20552076221149651. -
  62. [65][65] Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large Language Models for Data Annotation: A Survey. arXiv:2402.13446 [cs.CL] - · arXiv
  63. [66][66] Shengpu Tang, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W Sjoding, and Jenna Wiens. 2020. Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data. Journal of the American Medical Informatics Association 27, 12 (2020), 1921-1934. -
  64. [67][67] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. ArXiv preprint abs/2408.00118 (2024). https://arxiv.org/abs/2408. 00118 - · arXiv
  65. [68][68] Robin van de Water, Hendrik Nils Aurel Schmidt, Paul Elbers, Patrick Thoral, Bert Arnrich, and Patrick Rockenschaub. 2024. Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=ox2ATRM90I -
  66. [69][69] Juraj Vladika, Phillip Schneider, and Florian Matthes. 2024. MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering. ArXiv preprint abs/2406.05845 (2024). https://arxiv.org/abs/2406.05845 - · arXiv
  67. [70][70] Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. 2023. Pre-trained language models in biomedical domain: A systematic survey. Comput. Surveys 56, 3 (2023), 1-52. -
  68. [71][71] Chong Wang, Mengyao Li, Junjun He, Zhongruo Wang, Erfan Darzi, Zan Chen, Jin Ye, Tianbin Li, Yanzhou Su, Jing Ke, et al. 2024. A Survey for Large Language Models in Biomedicine. ArXiv preprint abs/2409.00133 (2024). https://arxiv.org/ abs/2409.00133 - · arXiv
  69. [72][72] Guanchu Wang, Junhao Ran, Ruixiang Tang, Chia-Yuan Chang, Yu-Neng Chuang, Zirui Liu, Vladimir Braverman, Zhandong Liu, and Xia Hu. 2024. Assessing and Enhancing Large Language Models in Rare Disease Question-answering. ArXiv preprint abs/2408.08422 (2024). https://arxiv.org/abs/2408.08422 - · arXiv
  70. [73][73] Jiaqi Wang, Junyu Luo, Muchao Ye, Xiaochen Wang, Yuan Zhong, Aofei Chang, Guanjie Huang, Ziyi Yin, Cao Xiao, Jimeng Sun, et al. 2024. Recent Advances in Predictive Modeling with Electronic Health Records. ArXiv preprint abs/2402.01077 (2024). https://arxiv.org/abs/2402.01077 - · arXiv
  71. [74][74] Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. 2020. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. In Proceedings of the ACM conference on health, inference, and learning . 222-235. -
  72. [75][75] Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. 2023. Cmb: A comprehensive medical benchmark in chinese. ArXiv preprint abs/2308.08833 (2023). https://arxiv.org/abs/2308.08833 - · arXiv
  73. [76][76] Zifeng Wang, Hanyin Wang, Benjamin Danek, Ying Li, Christina Mack, Hoifung Poon, Yajun Wang, Pranav Rajpurkar, and Jimeng Sun. 2024. A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challenges. arXiv preprint arXiv: 2411.00024 (2024). - · arXiv
  74. [77][77] Ran Xu, Mohammed K Ali, Joyce C Ho, and Carl Yang. 2023. Hypergraph transformers for ehr-based clinical predictions. AMIA Summits on Translational Science Proceedings 2023 (2023), 582. -
  75. [78][78] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. ArXiv preprint abs/2407.10671 (2024). https://arxiv.org/abs/2407.10671 - · arXiv
  76. [79][79] Chaoqi Yang, Zhenbang Wu, Patrick Jiang, Zhen Lin, Junyi Gao, Benjamin Danek, and Jimeng Sun. 2023. PyHealth: A Deep Learning Toolkit for Healthcare Predictive Modeling. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) 2023 . https: //github.com/sunlabuiuc/PyHealth -
  77. [80][80] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. ArXiv preprint abs/2403.04652 (2024). https://arxiv. org/abs/2403.04652 - · arXiv
  78. [81][81] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. ArXiv preprint abs/2308.10792 (2023). https://arxiv.org/abs/2308.10792 - · arXiv
  79. [82][82] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595-46623. -
  80. [83][83] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. ArXiv preprint abs/2304.06364 (2023). https://arxiv.org/abs/2304.06364 - · arXiv
  81. [84][84] Hongjian Zhou, Boyang Gu, Xinyu Zou, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Xian Wu, et al. 2023. A survey of large language models in medicine: Progress, application, and challenge. ArXiv preprint abs/2311.05112 (2023). https://arxiv.org/abs/2311.05112 - · arXiv
  82. [85][85] Shuang Zhou, Zidu Xu, Mian Zhang, Chunpu Xu, Yawen Guo, Zaifu Zhan, Sirui Ding, Jiashuo Wang, Kaishuai Xu, Yi Fang, et al. 2024. Large Language Models for Disease Diagnosis: A Scoping Review. ArXiv preprint abs/2409.00097 (2024). https://arxiv.org/abs/2409.00097 · arXiv

Cited by

Sorted by citations

Related

Metadata

Authors
Chen, C., Yu, J., Chen, S., Liu, C., Wan, Z., Zhou, S., Luo, Y., Zhang, R., Bitterman, D., Wang, F., & Shu, K.
Affiliations
  • Northwestern University (US)
  • The University of Texas at Austin (US)
  • Boston Children's Hospital (US)
  • Imperial College London (GB)
  • The Ohio State University (US)
  • Harvard University (US)
  • University of Minnesota (US)
  • Cornell University (US)
  • Emory University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
10567-10577
DOI
10.48550/arxiv.2411.06469
arXiv
2411.06469
OpenAlex
W4404401764
Cited by
5
References
0
Citations by year
2026: 2 · 2025: 3
FWCI
17.09
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
large language models, clinical prediction, benchmark, medical LLMs, clinical reasoning, clinical decision-making, prompting strategies, fine-tuning strategies, artificial intelligence in healthcare

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review