The DOI matches its Crossref record; 201 of the 201 references it cites were found in a registry or an index; 44 of the 73 sources its claims cite were checked for support.
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
Open accessPublished version of a preprint
Abstract
In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions. Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg, sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patient interviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics and proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Our investigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research and practical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated clinical AI systems.
Overview
CoChat Summary
## TL;DR
This paper is a narrative review and perspective on multimodal large language models (M-LLMs) in health care, covering their principles, applications, challenges, and future directions. It argues that M-LLMs can extend unimodal LLMs by integrating text, images, audio, video, temporal, and omics data, but real-world use still faces major technical, ethical, and evaluation hurdles.
## Key findings
- The paper identifies 4 core stages in an M-LLM architecture: modality-specific encoding, embedding alignment and fusion, contextual understanding and cross-modal interactions, and decision-making/output generation.
- It argues that current unimodal LLMs are insufficient for medicine because clinical decision-making often combines multiple data types over time, including images, EHRs, laboratory results, audio, and omics data.
- The review highlights wide application areas for M-LLMs, including medical imaging, temporal monitoring, audio analysis, video interpretation, text/EHR analysis, and omics-driven personalized medicine.
- It reports that a general-domain M-LLM study found that such models can “inherently understand medical images” and, in some contexts, even outperform specialized task-specific pretraining methods; the paper uses CONCH as a use case for zero-shot cross-modal retrieval in histopathology.
## Methods
- Narrative review/perspective article synthesizing foundational principles, current and potential applications, technical and ethical challenges, and future outlook for medical M-LLMs.
- The paper organizes the discussion by modality and task: medical images, temporal data, audio, text, video, omics data, and any-to-any M-LLMs.
- It includes a use case example using the CONCH model, run with pretrained weights from Hugging Face on a Linux machine with an NVIDIA GeForce GTX 1080 Ti, in a Flask-based demo for zero-shot cross-modal retrieval on histopathology slides.
## Limitations
- The paper states that recent studies on M-LLMs in health care are limited by a narrow range of modalities, largely theoretical discussion, and limited exploration of solutions to technical barriers.
- The conclusions note that M-LLMs still require further evaluation and benchmarking in real-world medical settings, and that progress toward sophisticated multimodal outputs remains comparatively slow.
- Evident from the text: this is a conceptual review rather than an empirical study, so it does not report original clinical outcome data or a systematic quantitative comparison across models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
29 Valid · 13 Not valid · 12 Not checked · 69 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Third, although the challenges of integrating diverse data types into M-LLMs are recognized, there is limited exploration of possible solutions or ongoing research aimed at overcoming these technical barriers [38,39].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [38] sourceExperimental — accuracy not yet measured
The ‘Barriers and Limitations’ section identifies significant technical challenges in integrating text, images, video, and sound into M-LLMs. It does not assess whether recent studies have explored solutions or ongoing research to a limited extent; the Conclusions even suggest developing new tokenization strategies as a possible response. No page numbers are available in our copy. (LLM, conf=0.84)
- Disputedcites [39] sourceExperimental — accuracy not yet measured
The “Multi-modal LLM” section (page unavailable) describes ongoing research combining images, audio, and text, while acknowledging limited data availability. The “Cross-modal dental diagnosis with LLMs” section (page unavailable) also explores a way to combine text and images for dental diagnosis. The paper therefore does not support the attributed lack of exploration of solutions or ongoing research. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While this unimodal focus on text-based operation has been transformative in the medical field, it does not fully capture the complex and diverse nature of health care practice [8].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [8] sourceExperimental — accuracy not yet measured
The main text says clinical practice requires more than responding accurately to queries and includes practical tasks that LLMs are not equipped to replace. That supports the claim that text-based operation does not capture the full scope of healthcare. But the paper does not describe LLMs as uniformly unimodal: it says GPT-4 accepts images as well as text. It also discusses potential benefits in medicine rather than establishing that text-based operation has already been transformative there. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, the range of data modalities examined is often restricted to text, images, videos, and audio [38], with some studies focusing narrowly on a limited number of clinical applications [39].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [38] sourceExperimental — accuracy not yet measured
The ‘Multimodal Analysis’ section (page unavailable) lists text, images, audio, and video as data types that future M-LLMs could analyze. The ‘Barriers and Limitations’ section says multimodal training data sets are limited, but the paper does not establish that recent studies often restrict the modalities they examine to those four types. (LLM, conf=0.82)
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
In “Cross-modal dental diagnosis with LLMs” (page unavailable), the paper says most dental AI studies have used image-only approaches, supporting the point that modalities studied have been limited. “Multi-modal LLM” (page unavailable) discusses text, images, and audio in medical learning, but the paper does not establish that health-care M-LLM studies are often restricted to the four listed modalities, including video. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Applications > Audio
What the sources say
- Could not verifycites [103] sourceExperimental — accuracy not yet measured
The Introduction says large models have advanced more gradually in audio than in language, images, and video. The Related Surveys and Differences section mentions healthcare surveys but does not identify a gap in developing large audio models specifically for medical applications. Searches for medical and healthcare applications found no passage making that claim; page numbers were unavailable in our copy. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Moreover, nontextual data types play a crucial role in diagnosis, effective treatment planning, research, and patient care [9-11].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Supportedcites [9] source
The Introduction (page not provided) identifies medical images and omics data as relevant to diagnostics, therapy planning, and patient monitoring. “Towards Future Human-AI Interfaces for Multi-Modal Causability” (page not provided) says that integrating images, omics, and other data is needed to understand diseases and biological processes. Together, these passages support the broad claim about nontextual data in research and patient care. (LLM, conf=0.85)
- Partly supportedcites [10] sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) describes using patients’ genomic data to predict early warning signals of cancer, and the Biologic significance section (page unavailable) identifies potential drug targets for early intervention. This supports a narrower point about nontextual molecular data in cancer prediction and treatment research, but not the broad claim about nontextual data types across diagnosis, effective treatment planning, research, and patient care. The paper’s “multi-modal” refers to alternative biomarker configurations, not different data types. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [11] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
34 more claims · 13 with their cited sources checked
Create a free account to see all 123 claims and every source verdict.
- It may state two separate claims as one. 26
- Probably not a substantial claim. 74
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, such as GPT and BERT, are trained on massive datasets and have the unique capability of understanding both structured and unstructured text (11,81).”
From Generative artificial intelligence in predictive analysis of diabetes and its complications: a narrative review (2025) · cites this paper as [81]
Experimental — accuracy not yet measured
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“Time-series encoders (such as transformer-based models [15,59-61] and LSTM [12]) are intended to detect critical changes over time, signaling the need for urgent medical intervention.”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Knowledge distillation is another approach that involves training a smaller “student” model to replicate the behavior of a larger “teacher” model, requiring less computational power while retaining performance [161].”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Parameter-efficient fine-tuning methods such as adapter layers help reduce the computational load by fine-tuning only a subset of the model’s parameters [157,158].”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Examples of biomedical text LLMs include BiMediX [104], BioBERT [105], PubMedBERT [106], and ClinicalBERT [20].”
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1×
“In addition, quantization approaches can address the scalability and computational demands by shifting toward quantized versions of existing models using curated, domain-specific data rather than pretraining from scratch [159].”
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“Additional optimization techniques, such as model compression [165] and hardware acceleration using graphics processing units (GPUs) or tensor processing units (TPUs) [166], further enhance efficiency.”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Subsequently, a series of medical BERT models were proposed to accelerate medical research [6,17-20].”
Research works printed but not linked: 25
- VaswaniAShazeerNParmarNUszkoreitJJonesLGomezANKaiserLPolosukhinIAttention is all you needProceedings of the 31st International Conference on Neural Information Processing Systems2017NIPS'17December 4-9, 2017Long Beach, CA
- Devlin, J, arXiv
- Touvron, H, arXiv
- Chung, HW, arXiv
- Huang, W, arXiv
- Zhu, D, arXiv
- Su, Y, arXiv
- Koh, JY, arXiv
- Alayrac, JB, arXiv
- Rasheed, H, arXiv
- WangZZhangQYuHHuSJinXGongZChenHMulti-level protein structure pre-training with prompt learningProceedings of the Eleventh International Conference on Learning Representations2023ICLR 2023May 1-5, 2023Kigali, Rwanda
- Zhou, H, arXiv
- Yordanov, Y, arXiv
- HewittJManningCDA structural probe for finding syntax in word representationsProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies2019NAACL-HLT 2019June 2-7, 2019Minneapolis, MN
- Gerke, S, 2020, Artificial Intelligence in Healthcare, 295
- Lee, N, arXiv
- Reddy, AG, arXiv
- Carlini, N, arXiv
- Aghajanyan, A, arXiv
- Huang, Y, arXiv
- Wang, X, arXiv
- Fu, C, arXiv
- Yang, X, arXiv
- Ge, W, arXiv
- Li, S, arXiv
Cited by
Large Language Models in Healthcare and Medical Applications: A Review
Maity, S., & Saikia, M. J. (2025) · Bioengineering · doi:10.3390/bioengineering12060631 · Cited by 249
“The integration of LLMs with medical imaging represents a significant advancement in diagnostic capabilities, particularly through multimodal models that can process both visual and textual information [4,11,56–58].”
Zhou, M., Pan, Y., Zhang, Y., Song, X., & Zhou, Y. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105871 · Cited by 142
Multimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 153
“The defining feature is their ability to integrate and align information across modalities, often mapping them into a shared representational space [8].”
InfluentialAbbas, S. R., Seol, H., Abbas, Z., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13141642 · Cited by 80
“Multimodal AI Models: Research is underway to integrate imaging, EHR data, genomic profiles, and behavioral metrics into unified decision-making tools [103].”
A Current Review of Generative AI in Medicine: Core Concepts, Applications, and Current Limitations
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Shariatnia, M. M., Rouzrokh, P., & Erickson, B. J. (2025) · Current Reviews in Musculoskeletal Medicine · doi:10.1007/s12178-025-09961-y · Cited by 58
“Audio and video generation models in medicine are enabling new ways to enhance clinical training, patient communication, and medical education [ 64 ].”
Metadata
- Authors
- AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J.
- Affiliations
- Weill Cornell Medical College in Qatar (QA)
- Hamad bin Khalifa University (QA)
- United Arab Emirates University (AE)
- Corresponding author
- AlSaad, Rawan
- ORCID
- Venue
- Journal of medical Internet research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-09-25
- Volume
- 26
- Pages
- e59505
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11464944
- OpenAlex
- W4402155831
- Cited by
- 349
- References
- 175
- Citations by year
- 2026: 202 · 2025: 139 · 2024: 8
- FWCI
- 32.47
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- multimodal large language models, multimodal data integration, health care applications, medical images, wearable sensor data, electronic health records, clinical notes, audio recordings, surgical video, omics data, clinical decision support, ethical considerations, multimodal fusion
- MeSH terms
- Delivery of Health Care, Humans, Natural Language Processing, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. https://app.cochat.ai/papers/10.2196/59505
- The paper
- AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024). Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. Journal of medical Internet research. https://doi.org/10.2196/59505
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.