Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 201 of the 201 references it cites were found in a registry or an index; 44 of the 73 sources its claims cite were checked for support.

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.

Open access

Published version of a preprint

Abstract

In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions. Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg, sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patient interviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics and proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Our investigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research and practical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated clinical AI systems.

Overview

CoChat Summary

## TL;DR

This paper is a narrative review and perspective on multimodal large language models (M-LLMs) in health care, covering their principles, applications, challenges, and future directions. It argues that M-LLMs can extend unimodal LLMs by integrating text, images, audio, video, temporal, and omics data, but real-world use still faces major technical, ethical, and evaluation hurdles.

## Key findings

  • The paper identifies 4 core stages in an M-LLM architecture: modality-specific encoding, embedding alignment and fusion, contextual understanding and cross-modal interactions, and decision-making/output generation.
  • It argues that current unimodal LLMs are insufficient for medicine because clinical decision-making often combines multiple data types over time, including images, EHRs, laboratory results, audio, and omics data.
  • The review highlights wide application areas for M-LLMs, including medical imaging, temporal monitoring, audio analysis, video interpretation, text/EHR analysis, and omics-driven personalized medicine.
  • It reports that a general-domain M-LLM study found that such models can “inherently understand medical images” and, in some contexts, even outperform specialized task-specific pretraining methods; the paper uses CONCH as a use case for zero-shot cross-modal retrieval in histopathology.

## Methods

  • Narrative review/perspective article synthesizing foundational principles, current and potential applications, technical and ethical challenges, and future outlook for medical M-LLMs.
  • The paper organizes the discussion by modality and task: medical images, temporal data, audio, text, video, omics data, and any-to-any M-LLMs.
  • It includes a use case example using the CONCH model, run with pretrained weights from Hugging Face on a Linux machine with an NVIDIA GeForce GTX 1080 Ti, in a Flask-based demo for zero-shot cross-modal retrieval on histopathology slides.

## Limitations

  • The paper states that recent studies on M-LLMs in health care are limited by a narrow range of modalities, largely theoretical discussion, and limited exploration of solutions to technical barriers.
  • The conclusions note that M-LLMs still require further evaluation and benchmarking in real-world medical settings, and that progress toward sophisticated multimodal outputs remains comparatively slow.
  • Evident from the text: this is a conceptual review rather than an empirical study, so it does not report original clinical outcome data or a systematic quantitative comparison across models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/59505

Cites

201 of 201 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Unimodal large language model (LLM) and multimodal LLM (M-LLM) examples. ECG: Electrocardiogram.
Figure 1 Unimodal large language model (LLM) and multimodal LLM (M-LLM) examples. ECG: Electrocardiogram.CC BY · PubMed Central
Figure 2 Main components of multimodal large language model training.
Figure 2 Main components of multimodal large language model training.CC BY · PubMed Central
Figure 3 Applications of multimodal large language models in health care.
Figure 3 Applications of multimodal large language models in health care.CC BY · PubMed Central
Figure 4 Demonstration of the Contrastive Learning From Captions for Histopathology (CONCH) model as a text-vision foundation model for histopathology analysis.
Figure 4 Demonstration of the Contrastive Learning From Captions for Histopathology (CONCH) model as a text-vision foundation model for histopathology analysis.CC BY · PubMed Central
Figure 5 Challenges of multimodal large language models in health care.
Figure 5 Challenges of multimodal large language models in health care.CC BY · PubMed Central
Figure 6 Future directions of multimodal large language models in health care.
Figure 6 Future directions of multimodal large language models in health care.CC BY · PubMed Central

Verification

Claims the paper makes

29 Valid · 13 Not valid · 12 Not checked · 69 Own results and interpretations

34 more claims · 13 with their cited sources checked

Create a free account to see all 123 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 26
  • Probably not a substantial claim. 74

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 175 references

Research works printed but not linked: 25

  1. VaswaniAShazeerNParmarNUszkoreitJJonesLGomezANKaiserLPolosukhinIAttention is all you needProceedings of the 31st International Conference on Neural Information Processing Systems2017NIPS'17December 4-9, 2017Long Beach, CA
  2. Devlin, J, arXiv
  3. Touvron, H, arXiv
  4. Chung, HW, arXiv
  5. Huang, W, arXiv
  6. Zhu, D, arXiv
  7. Su, Y, arXiv
  8. Koh, JY, arXiv
  9. Alayrac, JB, arXiv
  10. Rasheed, H, arXiv
  11. WangZZhangQYuHHuSJinXGongZChenHMulti-level protein structure pre-training with prompt learningProceedings of the Eleventh International Conference on Learning Representations2023ICLR 2023May 1-5, 2023Kigali, Rwanda
  12. Zhou, H, arXiv
  13. Yordanov, Y, arXiv
  14. HewittJManningCDA structural probe for finding syntax in word representationsProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies2019NAACL-HLT 2019June 2-7, 2019Minneapolis, MN
  15. Gerke, S, 2020, Artificial Intelligence in Healthcare, 295
  16. Lee, N, arXiv
  17. Reddy, AG, arXiv
  18. Carlini, N, arXiv
  19. Aghajanyan, A, arXiv
  20. Huang, Y, arXiv
  21. Wang, X, arXiv
  22. Fu, C, arXiv
  23. Yang, X, arXiv
  24. Ge, W, arXiv
  25. Li, S, arXiv

Cited by

Items 201–300 of 330

Metadata

Authors
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J.
Affiliations
  • Weill Cornell Medical College in Qatar (QA)
  • Hamad bin Khalifa University (QA)
  • United Arab Emirates University (AE)
Corresponding author
AlSaad, Rawan
Venue
Journal of medical Internet research
Published
2024-09-25
Volume
26
Pages
e59505
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4402155831
Cited by
349
References
175
Citations by year
2026: 202 · 2025: 139 · 2024: 8
FWCI
32.47
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
multimodal large language models, multimodal data integration, health care applications, medical images, wearable sensor data, electronic health records, clinical notes, audio recordings, surgical video, omics data, clinical decision support, ethical considerations, multimodal fusion
MeSH terms
Delivery of Health Care, Humans, Natural Language Processing, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.