Multimodal Clinical Prediction with Unified Prompts and Pretrained Large-Language Models
Abstract
Clinical prediction models (CPMs) increasingly rely on multiple modalities to predict clinical outcomes. The use of free-text data sources (e.g., chief complaint, medical notes) presents new challenges in the heterogeneity of the text across providers and patients and the need to convert the free text to numerical features before combining with other modalities. Prior work has employed multi-head architectures to learn separate heads for different modalities. In this work, we propose a new approach of constructing a unified text prompt that captures information from multiple modalities. We then use existing large-language models (LLMs), optimized with diagnosis-contrastive learning (DCL) to encode the unified prompt and make clinical predictions. We test our approach on the prediction of a variety of outcomes from emergency department visits using a free-text chief complaint and structured numerical data, including demographic information and vital signs. We find that optimized LLMs with unified prompts outperform LLMs that only use the chief complaint by 0.02 weighted F1 score (p < 0.0001) and models that only use the structured data modality as numerical inputs by 0.15 (p < 0.0001) in predicting acuity. We also observe improvements in the prediction of clinical outcomes, including hospital admission, length of stay, and time to revisit.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Machine learning models that integrate task-relevant multimodal EHR yield superior performance compared to those relying on a single modality.
“Machine learning models that integrate task-relevant multimodal EHR have been shown to yield superior performance compared to those relying on a single modality2–6.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [4]
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401 · cited 1× · see citing works
“We evaluate our approach for multimodal clinical prediction with the MIMIC-IV-ED dataset of emergency department (ED) visit records from the Beth Israel Deaconess Medical Center from 2011 to 2019 [20].”
MethodClass-Balanced Loss Based on Effective Number of Samples
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., & Belongie, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00949 · Cited by 3,094 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
Strategies for Reducing the Door-to-Balloon Time in Acute Myocardial Infarction
Bradley, E. H., Herrin, J. S., Wang, Y., Barton, B. A., Webster, T. R., Mattera, J. A., Roumanis, S. A., Curtis, J. P., Nallamothu, B. K., Magid, D. J., et al. (2006) · New England Journal of Medicine · doi:10.1056/nejmsa063117 · Cited by 838 · see citing works
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · see citing works
Wang, Z., Yang, Z., & Dong, T. (2017) · Sensors · doi:10.3390/s17020341 · Cited by 357 · see citing works
Zhou, H.-Y., Yu, Y., Wang, C., Zhang, S., Gao, Y., Pan, J., Shao, J., Lu, G., Zhang, K., & Li, W. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01045-x · Cited by 343 · see citing works
Combining structured and unstructured data for predictive models: a deep learning approach
Zhang, D., Yin, C., Zeng, J., Yuan, X., & Zhang, P. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01297-6 · Cited by 269 · see citing works
Lin, C., Peterson, E. D., Smith, E. E., Saver, J. L., Liang, L., Xian, Y., Olson, D. M., Shah, B. R., Hernandez, A. F., Schwamm, L. H., et al. (2012) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.112.965210 · Cited by 255 · see citing works
Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial
Wang, G., Liu, X., Ying, Z., Yang, G., Chen, Z., Liu, Z., Zhang, M., Yan, H., Lu, Y., Gao, Y., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02552-9 · Cited by 156 · see citing works
Door-to-ECG time in patients with chest pain presenting to the ED
Diercks, D. B., Kirk, J. D., Lindsell, C. J., et al. (2005) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2005.05.016 · Cited by 91 · see citing works
Lu, S.-C., Xu, C., Nguyen, C., Geng, Y., Pfob, A., & Sidey-Gibbons, C. J. (2022) · JMIR Medical Informatics · doi:10.2196/33182 · Cited by 49 · see citing works
Evaluation of preprocessing techniques for chief complaint classification
Dara, J., Dowling, J., Travers, D. A., et al. (2007) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2007.11.004 · Cited by 37 · see citing works
Martin, A., Bauer, V., Datta, A., et al. (2019) · Journal of Asthma · doi:10.1080/02770903.2019.1648505 · Cited by 28 · see citing works
Chen, T., Chen, J. C., Chang, W., Tsai, W., Shih, M.-C., & Nabila, A. W. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104171 · Cited by 19 · see citing works
Development of prehospital assessment findings associated with massive transfusion
Wheeler, A. R., Cuenca, C. M., Fisher, A. D., et al. (2020) · Transfusion · doi:10.1111/trf.15595 · Cited by 14 · see citing works
Coded Chief Complaints—Automated Analysis of Free-text Complaints
Thompson, D. A., Eitel, D. R., Fernandes, C. M. B., et al. (2006) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2006.tb01718.x · Cited by 13 · see citing works
Shi, J., Ye, M., Chen, H., et al. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02367-1 · Cited by 11 · see citing works
ED Triage Process Improvement: Timely Vital Signs for Less Acute Patients
Falconer, S. S., Karuppan, C. M., Kiehne, E., et al. (2018) · Journal of Emergency Nursing · doi:10.1016/j.jen.2018.05.006 · Cited by 10 · see citing works
Evaluating robustness of language models for chief complaint extraction from patient-generated textPreprint
Valmianski, I., Goodwin, C., Finn, I. M., Khan, N., & Zisook, D. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06915 · Cited by 4 · see citing works
CoAI: Cost-Aware Artificial Intelligence for Health CarePreprint
Erion, G., Janizek, J. D., Hudelson, C. E., et al. (2021) · medRxiv · doi:10.1101/2021.01.19.21249356 · Cited by 4 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
Gao, J., Rahman, M., Caskey, J., Oguss, M., O’Rourke, A., Brown, R., Stey, A., Mayampurath, A., Churpek, M. M., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02219-4 · Cited by 2 · see citing works
“Examplepromptsfor chesttraumaseveritystrati fi cation and unhealthy alcohol use screening are provided in Supplementary Note 3.”
Influential
- Authors
- Winston, C., Winston, C., Winston, C., Winston, C., & Winston, C.
- Affiliations
- University of Washington (US)
- Venue
- 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI)
- Publisher
- IEEE
- Published
- 2024-06-03
- Pages
- 679-683
- OpenAlex
- W4401754091
- Cited by
- 6
- References
- 23
- Citations by year
- 2026: 4 · 2025: 2
- FWCI
- 1.73
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Healthcare Informatics (ICHI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, chief complaint, emergency department visits, clinical outcome prediction, hospital admission prediction, length of hospital stay, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
