CoChat
Sign Up
PreprintCited by 8Machine Learning in HealthcareArtificial Intelligence in Healthcare and EducationArtificial Intelligence in Healthcare

The DOI matches its DataCite record.

Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Data

Open access
Zhu, Yinghao, Wang, Zixiang, Gao, Junyi, Tong, Yuning, An, Jingkun, Liao, Weibin, Harrison, Ewen M., Ma, Liantao, Pan, Chengwei·2024·arXiv
DOI 10.48550/arxiv.2402.01713arXiv 2402.01713
Open this paper in CoChat — free accountOpen PDF

Abstract

The inherent complexity of structured longitudinal Electronic Health Records (EHR) data poses a significant challenge when integrated with Large Language Models (LLMs), which are traditionally tailored for natural language processing. Motivated by the urgent need for swift decision-making during new disease outbreaks, where traditional predictive models often fail due to a lack of historical data, this research investigates the adaptability of LLMs, like GPT-4, to EHR data. We particularly focus on their zero-shot capabilities, which enable them to make predictions in scenarios in which they haven't been explicitly trained. In response to the longitudinal, sparse, and knowledge-infused nature of EHR data, our prompting approach involves taking into account specific EHR characteristics such as units and reference ranges, and employing an in-context learning strategy that aligns with clinical contexts. Our comprehensive experiments on the MIMIC-IV and TJH datasets demonstrate that with our elaborately designed prompting framework, LLMs can improve prediction performance in key tasks such as mortality, length-of-stay, and 30-day readmission by about 35\%, surpassing ML models in few-shot settings. Our research underscores the potential of LLMs in enhancing clinical decision-making, especially in urgent healthcare situations like the outbreak of emerging diseases with no labeled data. The code is publicly available at https://github.com/yhzhu99/llm4healthcare for reproducibility.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.01713

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modern multimodal LLMs can understand modalities beyond plain text, including medical images and structured EHR data, and can convert non-plain text data into text summaries.

    “Modern multimodal LLMs can also understand modalities beyond plain text, including medical images23,34 and structured EHR data35,36, and can convert non-plain text data into text summaries.”

    From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [35]

    The Methodology says structured EHR data are converted into natural-language text before being fed to an LLM, and the Conclusions report successful clinical predictions using that prompting framework. The paper therefore supports LLM use with structured EHR data, but not the implication that the model directly understands raw EHR data or itself converts that data into text summaries. The medical-image portion is attached to other citations, not this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 33

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. Hsu-Hsiang Chang, Te-Cheng Hsu, Yi- Hsien Hsieh, and Che Lin. Meta-ehr: A meta-learning approach for electronic health records with a high imbal- anced ratio and missing rate. In 2023 45th Annual Inter- national Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 1-4. IEEE, 2023.
  3. Gao et al., 2020a] Junyi Gao, Cao Xiao, Lucas M Glass, and Jimeng Sun. Dr. agent: Clinical predictive model via mimicked second opinions. Journal of the American Med- ical Informatics Association, 27(7):1084-1091, 2020.
  4. Gao et al., 2020b] Junyi Gao, Cao Xiao, Yasha Wang, Wen Tang, Lucas M Glass, and Jimeng Sun. Stagenet: Stage- aware neural networks for health risk prediction. In Pro- ceedings of The Web Conference 2020, pages 530-540, 2020.
  5. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023. · arXiv
  6. Lin Lawrence Guo, Ethan Steinberg, Scott Lanyon Fleming, Jose Posada, Joshua Lemmon, Stephen R Pfohl, Nigam Shah, Jason Fries, and Lillian Sung. Ehr foundation models improve robustness in the presence of temporal distribution shift. Scientific Reports, 13(1):3767, 2023.
  7. Zhihua Jin, Xin Jiang, Xingbo Wang, Qun Liu, Yong Wang, Xiaozhe Ren, and Huamin Qu. Numgpt: Improving numeracy ability of generative pre-trained models. arXiv preprint arXiv:2109.03137, 2021. · arXiv
  8. Alistair EW Johnson, Lucas Bulgar- elli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
  9. Stephen A Lauer, Kyra H Grantz, Qifang Bi, Forrest K Jones, Qulu Zheng, Hannah R Meredith, An- drew S Azman, Nicholas G Reich, and Justin Lessler. The incubation period of coronavirus disease 2019 (covid-19) from publicly reported confirmed cases: estimation and application. Annals of internal medicine, 172(9):577-582, 2020.
  10. Liu et al., 2023] Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming- Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525, 2023. · arXiv
  11. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023. · arXiv
  12. Ma et al., 2020a] Liantao Ma, Junyi Gao, Yasha Wang, Chaohe Zhang, Jiangtao Wang, Wenjie Ruan, Wen Tang, Xin Gao, and Xinyu Ma. Adacare: Explainable clini- cal health status representation learning via scale-adaptive feature extraction and recalibration. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):825- 832, Apr. 2020.
  13. Ma et al., 2020b] Liantao Ma, Chaohe Zhang, Yasha Wang, Wenjie Ruan, Jiangtao Wang, Wen Tang, Xinyu Ma, Xin Gao, and Junyi Gao. Concare: Personalized clinical fea- ture embedding via capturing the healthcare context. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 34(01):833-840, Apr. 2020.
  14. Liantao Ma, Xinyu Ma, Junyi Gao, Xian- feng Jiao, Zhihao Yu, Chaohe Zhang, Wenjie Ruan, Yasha Wang, Wen Tang, and Jiangtao Wang. Distilling knowl- edge from publicly available online emr data to emerging epidemic for prognosis. In Proceedings of the Web Con- ference 2021, pages 3558-3568, 2021.
  15. Xinyu Ma, Yasha Wang, Xu Chu, Liantao Ma, Wen Tang, Junfeng Zhao, Ye Yuan, and Guoren Wang. Patient health representation learning via correla- tional sparse prior of medical features. IEEE Transactions on Knowledge and Data Engineering, 2022.
  16. Liantao Ma, Chaohe Zhang, Junyi Gao, Xi- anfeng Jiao, Zhihao Yu, Yinghao Zhu, Tianlong Wang, Xinyu Ma, Yasha Wang, Wen Tang, Xinju Zhao, Wenjie Ruan, and Tao Wang. Mortality prediction with adaptive feature importance recalibration for peritoneal dialysis pa- tients. Patterns, 4(12), 2023.
  17. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023. · arXiv
  18. Christopher R Stephens, Joaquín Giménez Heau, Camila González, Carlos N Ibarra-Cerdeña, Victor Sánchez-Cordero, and Constantino González-Salazar. Using biotic interaction networks for prediction in biodiversity and emerging diseases. PLoS One, 4(5):e5725, 2009.
  19. Zhe Sun, Shaoliang Peng, Yaning Yang, Xiaoqi Wang, and Fei Li. A general fine-tuned transfer learning model for predicting clinical task acrossing di- verse ehrs datasets. In 2019 IEEE International Confer- ence on Bioinformatics and Biomedicine (BIBM), pages 490-495. IEEE, 2019.
  20. Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  21. Wang et al., 2020] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few ex- amples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
  22. Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Process- ing Systems, volume 35, pages 24824-24837.
  23. Brian J Wells, Kevin M Chagin, Amy S Nowacki, and Michael W Kattan. Strategies for han- dling missing data in electronic health record derived data. Egems, 1(3), 2013.
  24. Wornow et al., 2023a] Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason Fries, and Nigam Shah. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. arXiv preprint arXiv:2307.02028, 2023. · arXiv
  25. Wornow et al., 2023b] Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries, and Nigam H Shah. The shaky foundations of large language models and foun- dation models for electronic health records. npj Digital Medicine, 6(1):135, 2023.
  26. Hao Xue and Flora D Salim. Prompt- cast: A new prompt-based learning paradigm for time se- ries forecasting. IEEE Transactions on Knowledge and Data Engineering, 2023.
  27. Li Yan, Hai-Tao Zhang, Jorge Goncalves, Yang Xiao, Maolin Wang, Yuqi Guo, Chuan Sun, Xi- uchuan Tang, Liang Jing, Mingyang Zhang, et al. An inter- pretable mortality prediction model for covid-19 patients. Nature machine intelligence, 2(5):283-288, 2020.
  28. Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, et al. If llm is the wizard, then code is the wand: A survey on how code empowers large lan- guage models to serve as intelligent agents. arXiv preprint arXiv:2401.00812, 2024. · arXiv
  29. Xi Sheryl Zhang, Fengyi Tang, Hi- roko H Dodge, Jiayu Zhou, and Fei Wang. Metapred: Meta-learning for clinical risk prediction with limited pa- tient electronic health records. In Proceedings of the 25th ACM SIGKDD international conference on knowl- edge discovery & data mining, pages 2487-2495, 2019.
  30. Chaohe Zhang, Xu Chu, Liantao Ma, Yinghao Zhu, Yasha Wang, Jiangtao Wang, and Junfeng Zhao. M3care: Learning with missing modalities in mul- timodal healthcare data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 2418-2428, New York, NY, USA, 2022. Association for Computing Machinery.
  31. Zhongji Zhang, Yuhang Wang, Yinghao Zhu, Xinyu Ma, Tianlong Wang, Chaohe Zhang, Yasha Wang, and Liantao Ma. A transfer-learning-based progno- sis prediction paradigm that bridges data distribution shift across emr datasets. arXiv preprint arXiv:2310.07799, 2023. · arXiv
  32. Zhu et al., 2023a] Yinghao Zhu, Jingkun An, Enshen Zhou, Lu An, Junyi Gao, Hao Li, Haoran Feng, Bo Hou, Wen Tang, Chengwei Pan, and Liantao Ma. M3fair: Mitigat- ing bias in healthcare data through multi-level and multi- sensitive-attribute reweighting method. arXiv preprint arXiv:2306.04118, 2023. · arXiv
  33. Zhu et al., 2023c] Yinghao Zhu, Zixiang Wang, Long He, Shiyun Xie, Zixi Chen, Jingkun An, Liantao Ma, and Chengwei Pan. Leveraging prototype patient represen- tations with feature-missing-aware calibration to mitigate ehr data sparsity. arXiv preprint arXiv:2309.04160, 2023. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Zhu, Y., Wang, Z., Gao, J., Tong, Y., An, J., Liao, W., Harrison, E. M., Ma, L., & Pan, C.
Affiliations
  • Peking University (CN)
  • Health Data Research UK (GB)
  • National University of Singapore (SG)
  • Beihang University (CN)
  • University of Edinburgh (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.01713
arXiv
2402.01713
OpenAlex
W4391591648
Cited by
8
References
0
Citations by year
2025: 6 · 2024: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
large language models, zero-shot learning, clinical prediction, electronic health records, prompting, in-context learning, GPT-4, mortality prediction, length of hospital stay, 30-day readmission, MIMIC-IV, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review