The DOI matches its Crossref record; 201 of the 201 references it cites were found in a registry or an index; 44 of the 73 sources its claims cite were checked for support.
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
Open accessPublished version of a preprint
Abstract
In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions. Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg, sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patient interviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics and proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Our investigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research and practical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated clinical AI systems.
Overview
CoChat Summary
## TL;DR
This paper is a narrative review and perspective on multimodal large language models (M-LLMs) in health care, covering their principles, applications, challenges, and future directions. It argues that M-LLMs can extend unimodal LLMs by integrating text, images, audio, video, temporal, and omics data, but real-world use still faces major technical, ethical, and evaluation hurdles.
## Key findings
- The paper identifies 4 core stages in an M-LLM architecture: modality-specific encoding, embedding alignment and fusion, contextual understanding and cross-modal interactions, and decision-making/output generation.
- It argues that current unimodal LLMs are insufficient for medicine because clinical decision-making often combines multiple data types over time, including images, EHRs, laboratory results, audio, and omics data.
- The review highlights wide application areas for M-LLMs, including medical imaging, temporal monitoring, audio analysis, video interpretation, text/EHR analysis, and omics-driven personalized medicine.
- It reports that a general-domain M-LLM study found that such models can “inherently understand medical images” and, in some contexts, even outperform specialized task-specific pretraining methods; the paper uses CONCH as a use case for zero-shot cross-modal retrieval in histopathology.
## Methods
- Narrative review/perspective article synthesizing foundational principles, current and potential applications, technical and ethical challenges, and future outlook for medical M-LLMs.
- The paper organizes the discussion by modality and task: medical images, temporal data, audio, text, video, omics data, and any-to-any M-LLMs.
- It includes a use case example using the CONCH model, run with pretrained weights from Hugging Face on a Linux machine with an NVIDIA GeForce GTX 1080 Ti, in a Flask-based demo for zero-shot cross-modal retrieval on histopathology slides.
## Limitations
- The paper states that recent studies on M-LLMs in health care are limited by a narrow range of modalities, largely theoretical discussion, and limited exploration of solutions to technical barriers.
- The conclusions note that M-LLMs still require further evaluation and benchmarking in real-world medical settings, and that progress toward sophisticated multimodal outputs remains comparatively slow.
- Evident from the text: this is a conceptual review rather than an empirical study, so it does not report original clinical outcome data or a systematic quantitative comparison across models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
29 Valid · 13 Not valid · 12 Not checked · 69 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Third, although the challenges of integrating diverse data types into M-LLMs are recognized, there is limited exploration of possible solutions or ongoing research aimed at overcoming these technical barriers [38,39].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [38] sourceExperimental — accuracy not yet measured
The ‘Barriers and Limitations’ section identifies significant technical challenges in integrating text, images, video, and sound into M-LLMs. It does not assess whether recent studies have explored solutions or ongoing research to a limited extent; the Conclusions even suggest developing new tokenization strategies as a possible response. No page numbers are available in our copy. (LLM, conf=0.84)
- Disputedcites [39] sourceExperimental — accuracy not yet measured
The “Multi-modal LLM” section (page unavailable) describes ongoing research combining images, audio, and text, while acknowledging limited data availability. The “Cross-modal dental diagnosis with LLMs” section (page unavailable) also explores a way to combine text and images for dental diagnosis. The paper therefore does not support the attributed lack of exploration of solutions or ongoing research. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While this unimodal focus on text-based operation has been transformative in the medical field, it does not fully capture the complex and diverse nature of health care practice [8].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [8] sourceExperimental — accuracy not yet measured
The main text says clinical practice requires more than responding accurately to queries and includes practical tasks that LLMs are not equipped to replace. That supports the claim that text-based operation does not capture the full scope of healthcare. But the paper does not describe LLMs as uniformly unimodal: it says GPT-4 accepts images as well as text. It also discusses potential benefits in medicine rather than establishing that text-based operation has already been transformative there. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, the range of data modalities examined is often restricted to text, images, videos, and audio [38], with some studies focusing narrowly on a limited number of clinical applications [39].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [38] sourceExperimental — accuracy not yet measured
The ‘Multimodal Analysis’ section (page unavailable) lists text, images, audio, and video as data types that future M-LLMs could analyze. The ‘Barriers and Limitations’ section says multimodal training data sets are limited, but the paper does not establish that recent studies often restrict the modalities they examine to those four types. (LLM, conf=0.82)
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
In “Cross-modal dental diagnosis with LLMs” (page unavailable), the paper says most dental AI studies have used image-only approaches, supporting the point that modalities studied have been limited. “Multi-modal LLM” (page unavailable) discusses text, images, and audio in medical learning, but the paper does not establish that health-care M-LLM studies are often restricted to the four listed modalities, including video. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Applications > Audio
What the sources say
- Could not verifycites [103] sourceExperimental — accuracy not yet measured
The Introduction says large models have advanced more gradually in audio than in language, images, and video. The Related Surveys and Differences section mentions healthcare surveys but does not identify a gap in developing large audio models specifically for medical applications. Searches for medical and healthcare applications found no passage making that claim; page numbers were unavailable in our copy. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Moreover, nontextual data types play a crucial role in diagnosis, effective treatment planning, research, and patient care [9-11].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Supportedcites [9] source
The Introduction (page not provided) identifies medical images and omics data as relevant to diagnostics, therapy planning, and patient monitoring. “Towards Future Human-AI Interfaces for Multi-Modal Causability” (page not provided) says that integrating images, omics, and other data is needed to understand diseases and biological processes. Together, these passages support the broad claim about nontextual data in research and patient care. (LLM, conf=0.85)
- Partly supportedcites [10] sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) describes using patients’ genomic data to predict early warning signals of cancer, and the Biologic significance section (page unavailable) identifies potential drug targets for early intervention. This supports a narrower point about nontextual molecular data in cancer prediction and treatment research, but not the broad claim about nontextual data types across diagnosis, effective treatment planning, research, and patient care. The paper’s “multi-modal” refers to alternative biomarker configurations, not different data types. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [11] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
34 more claims · 13 with their cited sources checked
Create a free account to see all 123 claims and every source verdict.
- It may state two separate claims as one. 26
- Probably not a substantial claim. 74
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, such as GPT and BERT, are trained on massive datasets and have the unique capability of understanding both structured and unstructured text (11,81).”
From Generative artificial intelligence in predictive analysis of diabetes and its complications: a narrative review (2025) · cites this paper as [81]
Experimental — accuracy not yet measured
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“Time-series encoders (such as transformer-based models [15,59-61] and LSTM [12]) are intended to detect critical changes over time, signaling the need for urgent medical intervention.”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Knowledge distillation is another approach that involves training a smaller “student” model to replicate the behavior of a larger “teacher” model, requiring less computational power while retaining performance [161].”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Parameter-efficient fine-tuning methods such as adapter layers help reduce the computational load by fine-tuning only a subset of the model’s parameters [157,158].”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Examples of biomedical text LLMs include BiMediX [104], BioBERT [105], PubMedBERT [106], and ClinicalBERT [20].”
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1×
“In addition, quantization approaches can address the scalability and computational demands by shifting toward quantized versions of existing models using curated, domain-specific data rather than pretraining from scratch [159].”
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“Additional optimization techniques, such as model compression [165] and hardware acceleration using graphics processing units (GPUs) or tensor processing units (TPUs) [166], further enhance efficiency.”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Subsequently, a series of medical BERT models were proposed to accelerate medical research [6,17-20].”
Research works printed but not linked: 25
- VaswaniAShazeerNParmarNUszkoreitJJonesLGomezANKaiserLPolosukhinIAttention is all you needProceedings of the 31st International Conference on Neural Information Processing Systems2017NIPS'17December 4-9, 2017Long Beach, CA
- Devlin, J, arXiv
- Touvron, H, arXiv
- Chung, HW, arXiv
- Huang, W, arXiv
- Zhu, D, arXiv
- Su, Y, arXiv
- Koh, JY, arXiv
- Alayrac, JB, arXiv
- Rasheed, H, arXiv
- WangZZhangQYuHHuSJinXGongZChenHMulti-level protein structure pre-training with prompt learningProceedings of the Eleventh International Conference on Learning Representations2023ICLR 2023May 1-5, 2023Kigali, Rwanda
- Zhou, H, arXiv
- Yordanov, Y, arXiv
- HewittJManningCDA structural probe for finding syntax in word representationsProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies2019NAACL-HLT 2019June 2-7, 2019Minneapolis, MN
- Gerke, S, 2020, Artificial Intelligence in Healthcare, 295
- Lee, N, arXiv
- Reddy, AG, arXiv
- Carlini, N, arXiv
- Aghajanyan, A, arXiv
- Huang, Y, arXiv
- Wang, X, arXiv
- Fu, C, arXiv
- Yang, X, arXiv
- Ge, W, arXiv
- Li, S, arXiv
Cited by
Items 201–300 of 330
An, S., Yuan, J., Pan, Y., Ye, M., Chen, Z., Li, M., Yan, P., Yao, J., Guan, Y., Lin, Y., et al. (2026) · Cyborg and Bionic Systems · doi:10.34133/cbsystems.0501
“These models exhibit unprecedented capacities to integrate multiple modalities of data spanning text, images, and time-series signals [ 12 , 13 ].”
Loaiza‐Bonilla, A., Thaker, N. G., Tuysuz, E., Yerdan, S., & Kurnaz, S. (2026) · AI in Precision Oncology · doi:10.1177/2993091x251409792
Yu, H., Cong, R., Jin, Q., et al. (2025) · doi:10.1109/cyberscitech68397.2025.00103
“Existing frameworks like task-specific automation tools for diagnosis or note generation [17]–[19], operate as isolated components.”
Vadisetty, R., Polamarasetti, A., Goyal, M. K., & Gadam, H. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8104-4_4
Applications and future directions in multimodal large language model: opportunities and challenges
Chandra, J., Malviya, M., Sabu, S., Rajendran, R. K., & Joseph, A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-33592-1.00006-1
Alves, P. T. D. A., LIMA, N. M. D., Ramos, B. H. D. S., et al. (2026) · Revista de Geopolítica · doi:10.56238/revgeov17n1-076
Calvera-Rayo, A., Verdú, G., Morales-Ruiz, M., & Bolufer, A. B. F. (2026) · doi:10.2196/preprints.91300
The Impact of Thinking Mode and Multimodal Analysis on Generative AI’s Assessment of Medical Videos
Alesh, Y., & Shoufan, A. (2025) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom60686.2025.11342571
“In addition to improved reasoning, Multimodal Large Language Models (MLLMs) represent a significant advancement, capable of processing and integrating information from various modalities such as text, images, audio, and video [18], [19].”
The Research on Intelligent Medical Triage System Based on Large Language Models
Zhao, X. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356579 · Cited by 1
“Rawan AlSaad’s research team (2024) specifically discussed the application scenarios and potential value of multimodal LLMs from multiple perspectives: medical images, time-series data, audio recordings, text data, video data, and omics data [12].”
Cruz, E. D. L. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-3912-2.ch013
Jabbar, A., Jianjun, H., & Jabbar, M. K. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114046
Auditing M-LLMs for Privacy Risks: A Synthetic Benchmark and Evaluation Framework
Li, J., Chen, J., Feng, Z., & Zhou, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6957-1_3 · Cited by 2
Anindo, I. R., Sajid, M. A. U. I., Mahmood, M. S., Prottush, N., & Morol, M. K. (2025) · IEEE International Conference on Computing, Applications and Systems (COMPAS) · doi:10.1109/compas67506.2025.11381760
Kannan, M. R., Sindhu, V., Umamageshwari, D., & Keerthika, K. (2026) · Elsevier eBooks · doi:10.1016/b978-0-44-333234-0.00020-0
Optimizing medical image analysis with prompt-guided pseudo-labeling
Fan, L., Gong, X., Zheng, C., Li, X., Li, J., & Ou, Y. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109867 · Cited by 1
Kamdar, A., Kamdar, P., & Kamdar, P. A. (2026) · Indian Journal of Ophthalmology · doi:10.4103/ijo.ijo_3098_25
“[8] LLMs likely prioritized processing the textual information over images, suggesting reliance on text‑dominant reasoning.”
Şalbaş, A. (2026) · Medical Records · doi:10.37990/medr.1739579
AI-Powered Web App for Oral Cancer Classification: Integrating CNN Features with LLM APIs
Ormeño-Arriagada, P., Taramasco, C., Herrera, B., & Vargas, S. (2025) · International Conference of the Chilean Computer Science Society (SCCC) · doi:10.1109/sccc67219.2025.11420292
“Despite their versatility, the integration of LLMs as inference components in clinical workflows remains limited, particularly in systems driven by pretrained CNN-derived embeddings [21].”
Bono, M. (2026) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.01599-4
Multilingual NLP, personalization, and global health
Kaur, P., Devi, L., Sandhu, R., & Ghai, D. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-45252-9.00005-0 · Cited by 2
Challenges and future perspectives of AI in gastrointestinal cancers
Kanagala, A., Mareddy, N. S. R., & Nagaraju, G. P. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44121-9.00009-3
Saha, S., Singh, M. K., Obhrai, S., & Astuti, Y. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-45252-9.00020-7
Deepika, N., & Renuka, D. K. (2026) · International Conference on Intelligent Systems for Communication, IoT and Security (ICISCoIS) · doi:10.1109/iciscois62701.2026.11447909
“Due to this limitation, the study of the combination of different sensory modalities was necessitated so as to implement Multimodal Conversational AI (MCAI) which can provide for a richer and more natural interaction [3].”
What will be the next step of LLMs in TCM? A narrative review
Chen, S., Zhong, R., Zhang, W., Li, Z., Su, Y., Gao, L., & Hu, K. (2026) · Science of Traditional Chinese Medicine · doi:10.1097/st9.0000000000000109 · Cited by 1
Alghamdi, A. D. (2026) · Scientific Reports · doi:10.1038/s41598-026-45883-6 · Cited by 3
“AlSaad et al. 21 conducted a comprehensive review of multimodal LLMs in healthcare, identifying privacy and security concerns related to the integration of textual and visual medical data.”
Zhou, Y., Li, X., Li, B., Zhang, Y., & Zhou, F. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134953
Wang, Z., Ma, T., & Wang, H. (2026) · Die Radiologie · doi:10.1007/s00117-026-01608-4 · Cited by 2
The Vision for an Intelligent Discovery Environment for Science
Wang, Z., Sannyasi, A., & Jiang, J. (2026) · Springer eBooks · doi:10.1007/978-3-032-14222-1_9
Lin, S., Li, Y. X., Mao, R., Zou, X., Hu, Y., Ye, H. J., Wu, X., Yang, L., He, J., Lu, S., et al. (2026) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2026.03.010
Stelling, H., Kraus, A., Grieb, G., Breidung, D., & Güler, I. (2026) · Life · doi:10.3390/life16040681 · Cited by 1
“Additionally, parameter-efficient fine-tuning techniques such as LoRA and QLoRA may offer a practical pathway for adapting existing foundation models to ECG-specific tasks without requiring full retraining [34,35].”
Large language models in NLP: evolution, architectural trends, and open challenges
Javed, H., Shah, B., Ali, F., & Kwak, D. (2026) · Journal Of Big Data · doi:10.1186/s40537-026-01429-1 · Cited by 1
Ikeido, T., Togo, R., Ogawa, T., et al. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461482
Kang, D., Yuan, L., Feng, J., Grzybowski, A. E., & Jin, K. (2026) · Acta Ophthalmologica · doi:10.1111/aos.70136
Multimodal foundation models in colorectal cancer: from prediction to trustworthy clinical insight
Gharaie, D., Feriz, A. M., & Safarpour, H. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag179 · Cited by 2
“Vision-language models align histopathology images with unstructured clinical reports, enabling multimodal reasoning across imaging data and EHR [38].”
Tanvir, K., Biswas, R., Gomes, D., Rahman, M. M., Dey, N., & Islam, M. R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7315-7.ch009
Boundary Probing for Input Privacy Protection when Using LMM Services
Hui, X., Qu, H., Hu, P., Rahmani, H., & Liu, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00051
“As a result, people have become increasingly reliant on LMMs in their daily life [3, 33, 38].”
GHOST: Getting to the Bottom of Hallucinations with A Multi-round Consistency Benchmark
Vibashan, V. S., Chang, N., Schmalfuß, J., Patel, V. M., Yu, Z., & Álvarez, J. M. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00596 · Cited by 1
“These advancements hold significant potential for applications in critical fields such as health-care, autonomous driving, and surveillance [7, 16].”
Multimodal Generative AI for Personalized Healthcare and Real-Time Monitoring
Tummuri, S. S. R. (2026) · IEEE International Conference on Emerging Computing and Intelligent Technologies (ICoECIT) · doi:10.1109/icoecit68303.2026.11497204
“AlSaad et al. [4] devoted to the role of multimodal LLMs in the quality assurance of clinical services, diagnostic assistance, and EHR summarization.”
Building Toward a Future for Electronic Health Record Systems
Rosenbloom, S. T., Holmgren, A. J., & Steitz, B. D. (2026) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092724-050949
AI-assisted placenta pathology in clinical use: barriers and opportunities
Walker, E. C., Huang, Y., Glastonbury, C. A., O'Hara, K. P., Fraser, A., Gordijn, S. J., Ernst, L. M., & Nellåker, C. (2026) · Placenta · doi:10.1016/j.placenta.2026.05.006
Gebhard, H., Kartal, A., Manalil, N. F., Chung, L. K., Daulat, S. R., Cheng, C. D., Waheed, A. A., Shahzadi, A., Hussain, I., Härtl, R., et al. (2026) · Global Spine Journal · doi:10.1177/21925682261448823 · Cited by 2
“3,7 Lee et al described a technique for converting DICOM stacks into video-like formats that allow multimodal models to treat them as images.”
Hassanein, F. E. A., Veseli, E., & Abou‐Bakr, A. (2026) · Journal of Stomatology Oral and Maxillofacial Surgery · doi:10.1016/j.jormas.2026.102841
Bongale, A., & Shrivastava, K. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01334-9
Transfer Learning for Explainable AI in Clinical LLMs
Farhadi, A., & Mozayani, N. (2026) · Springer eBooks · doi:10.1007/978-3-032-15088-2_6
Lei, C., Cui, Z., Duan, Y., Wu, Z., Qu, D., Wu, W., Zhang, Z., Ji, J. S., Zhou, B., Wu, J., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.102823 · Cited by 1
“20,21 Architectures that combine speech encoders with LLM-based decoders, along with unified fusion frameworks, enable models to learn jointly from speech and text.”
From interface to outcome: a 4I framework for AI-linked functionality in electronic health records
Zhang, T., & Shin, N. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03562-6 · Cited by 2
Lu, X., Jing, W., Jin, Q., Hao, G., Deng, W. C., Chen, Y., Li, Y., Wei, X., & Wang, D. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_24
An, B. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_15
Mantena, S., Johnson, A., Schütz, N., Tolas, A., Montalvo, S., Delgado-SanMartin, J., Ramírez‐Posada, M., Du, L., Zhang, S., Huynh, A. D., et al. (2026) · medRxiv · doi:10.64898/2026.05.26.26354165
VLMCache: Efficient On-Device Vision-Language Model Inference
Zhang, Y., Xu, D., Chen, Z., Wang, C., Zhang, Y., Xu, M., & Huang, G. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3745756.3809243 · Cited by 3
Use of generative artificial intelligence models in healthcare – a scoping review
Nooruddin, A., Lal, A., Ahmed, S. M. F., Ghori, M. H., Adnan, N., & Umer, F. (2026) · BMC Artificial Intelligence · doi:10.1186/s44398-026-00029-6
“Given the diverse nature of healthcare data, which often includes multiple types of information such as medical images, patient records, and clinical notes, multimodal models are particularly well-suited for handling such complexity [42].”
Responsible AI for safer opioid risk management in older adults
Bakshi, S., Gopal, R., & Yaraghi, N. (2026) · npj Health Systems · doi:10.1038/s44401-026-00092-1
“Their safe integration into health systems, especially for sensitive populations such as older adults, demands architectures that preserve predictive accuracy while ensuring transparent, evidence-based interpretation 44 .”
Shujaat, S., Pillai, A. G., Riaz, M., Alharbi, W., Ganganna, K., Alfadley, A. A., Aboalela, A. A., & Abolemaaty, W. (2026) · Head and Neck Pathology · doi:10.1007/s12105-026-01929-9
Clinical artificial intelligence applications of vision-language foundation models
Thirunavukarasu, A. J., Li, S. Y., Qin, P., Nie, D., Sanghera, R., Lim, E., Yu, J., & Zhang, L. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001453
Zhao, X.-Y., Shen, Y., He, J.-J., Zhou, Q.-Y., Jiang, L.-S., Zhou, Z.-R., An, F.-M., Zhan, Q., Sun, J., & Feng, W. (2026) · World Journal of Gastroenterology · doi:10.3748/wjg.118690
M 3 QuestionIng : Multi-modal Multi-span Medical Question Answering
Saha, A., Rathore, V., Tiwari, A., Ghosh, A. K., Edara, S. R., & Saha, S. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3820162
AI in respiratory care: findings from the GOLD report
Al-waqeerah, A., Bashah, A., Ghaleb, E., He, H., & Gao, L. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-08454-x · Cited by 1
Chen, H., Watanabe, S., Orii, R., Kaneko, M., Yumoto, K., & Kashizaki, F. (2026) · AI and Clinical Practice · doi:10.69854/aicp.2026.0002
Xing, H., Rehman, A., Hussain, M., Mastoi, M. S., Gulzar, N., & Khan, M. A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10699-0 · Cited by 2
Fu, H., Yao, W., Zhang, D., Du, Z., & Pan, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133434
Jiao, X., Liu, X., Yan, F., Yang, S., Wang, Y., Wang, C., Wang, Y., Xie, Y., Guo, Y. Z., Ma, Y., et al. (2026) · Digital Health · doi:10.1177/20552076261450827
Couto, F. D. F. S., & de Almeida, C. P. B. (2026) · Ciência & Saúde Coletiva · doi:10.1590/1413-81232026315.01132026en · Cited by 3
Beyond speed and professionalism: Evidence from a discrete choice experiment on AI coach preferences
Sun, Y., & Hou, Z. (2026) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2026.104960 · Cited by 1
Idakwo, V. O., Strote, C., Goelz, C., Shafei, Q., Stocker, T. J., Hausleiter, J., & Vieluf, S. (2026) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2026.1855775
“Additionally, foundation models, including large language models integrated into large-scale multimodal AI systems are likely to become key enabling technologies as the field shifts towards holistic, personalized risk stratification (219, 231, 232).”
A survey on audio–visual large language models
Yao, W., Yahyaabadi, R., Hassani, H., & Nikan, S. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116569
Evaluating and Understanding Model Editing for Medical Vision Language Models
Zhu, G., Wu, C. E., & Shen, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37362-5_8
Dong, H., Jin, L., Mei, X., Jia, M., Wang, Z., Ren, B., & Cao, L. (2026) · iScience · doi:10.1016/j.isci.2026.116518
“13,16 In the grossing setting, the more clinically relevant question is often not whether a model can assign a single global label, but whether it can provide consistent, auditable guidance about where to sample.”
Chen, F., Wang, L., Liu, H., Khan, Q., Tang, B., & Bao, N. (2026) · Journal of Nanobiotechnology · doi:10.1186/s12951-026-04551-7
“To meet the standards set by regulatory agencies, researchers must work with large-scale, multimodal medical data, which could lead to more dubious approaches, such as clinical gene modification studies [335].”
Maheshwari, P., Chand, K., Rama, P., et al. (2026) · International Conference on Next Generation Electronics (NELEX) · doi:10.1109/nelex69209.2026.11590397
“Unlike prior RAG-based clinical systems limited to text queries [17][18], this framework integrates dermoscopic image analysis with retrieval-augmented generation, enabling visual-conditioned, evidence-grounded medical explanations.”
Zhu, S., Abudupataer, M., Wang, Y., et al. (2026) · Cyborg and Bionic Systems · doi:10.34133/cbsystems.0658
“These observations are supported by earlier multimodal LLM studies showing that compact, interpretable feature vectors can benefit reasoning over raw signals [ 26 ].”
HyMed: An Event-Driven Multi-agent Framework for Smart Hospitals
Wang, H., Pan, J., Chen, K., Li, W., & Zhang, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3495-0_33
Computer Vision Enabled Learning Public Health Systems as a Service (CV-LPHSaaS)
Badr, N. G. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-26214-1_11
Rasheed, H. A., Maideen, I., Lozada, M. E. B., Mohamed, H. J., Hussin, D., Hariprasad, S., Elhag, A. T. M. K., & Adeesh, R. (2026) · Indian Journal of Clinical Cardiology · doi:10.1177/26324636261461988
“…the ECG-LM model that transforms raw ECG signals into a high-dimensional feature space and aligns it with a textual feature space from the LLM. 21,22 Studies by Yang et al. have found that these models outperform existing solutions in cardiovascular disease detection in terms of diagnosis,…”
Buzea, C. G., & Agop, M. (2026) · Artificial intelligence · doi:10.5772/intechopen.1014563
Benchmarking the Evolution of Large Language Models Toward Safe and Collaborative Clinical Reasoning
Asawaponwiput, W., Sri-iesaranusorn, P., Angkurawaranon, C., & Surangsrirat, D. (2026) · International Conference on Human Centric Artificial Intelligence (ICHCAI) · doi:10.1109/ichcai70183.2026.11607497
“Large Language Models (LLMs) have evolved into essential tools for many industries, with increasing application in healthcare [1], [2], [3], [4], [5].”
Kai, C., Tamori, H., Hirono, Y., Ishizuka, S., Kondo, S., Ohtsuka, T., & Kasai, S. (2026) · Japanese Journal of Radiology · doi:10.1007/s11604-026-02032-3
A Language‐Guided Multimodal Foundation Model for Zero‐Shot and Multi‐Task Brain Signal Analysis
Chen, M., Gui, Y., Luo, G., & Yang, Y. (2026) · Advanced Intelligent Systems · doi:10.1002/aisy.70486 · Cited by 1
“General-purpose multimodal foundation models are reshaping AI with remarkable performance in language, vision, and reasoning, driving their adoption in visually grounded medical domains such as pathology and radiology [26, 27].”
Jiang, X., Yang, N., Shi, T., et al. (2026) · Digital Health · doi:10.1177/20552076261473729
“Early studies have primarily focused on unimodal signal analysis, 11 including computer vision–based facial expression recognition and pain classification using physiological signals such as heart rate variability 12 and electroencephalography (EEG).”
Toward Autonomous Clinics: Human–Robot Collaboration in Clinical Care
Wu, X., Zhang, J., Xu, M., Chu, H. K., Chen, X., Zhong, P., Liu, B., He, M., & Shi, D. (2026) · MedComm · doi:10.1002/mco2.70885 · Cited by 2
Li, G., Lin, Y., & Wang, X. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1724332
Tan, M., Tang, S., Kan, S., Wu, B., Ni, Z., Zhang, H., & Ding, J. (2026) · Journal of Medical Internet Research · doi:10.2196/87288 · Cited by 1
“Because data generation can be automated, synthetic data reduce the time, cost, and human annotation burden while avoiding exposure of identifiable patient information [45,46].”
郑, 立., Lin, Z., Liu, X., Fan, Z., He, Z., Xu, J., & Yin, L. (2026) · Journal of Medical Internet Research · doi:10.2196/98026
“…prior studies on LLM diagnostic performance have concentrated on general outpatient clinics, conventional emergency departments, or general ICUs [13-16], with few dedicated comparative investigations in the EICU setting—a context characterized by rapid disease evolution, multiorgan involvement,…”
Wang, W., Lu, Z., Liu, F., Chen, C., Peng, Y., Chen, W., Li, C., & Zhu, H. (2026) · IEEE Access · doi:10.1109/access.2026.3718543
“Whole-exome sequencing (WES) has expanded the spectrum of investigation beyond conventional karyotyping [4] and candidate-gene testing by identifying rare variants in genes related to infertility [5].”
Demographic Bias Evaluation in Omnimodal Language Models
Elobaid, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31452-9_17
Challenges and Transformations in Biomedical Statistics Research Driven by Artificial Intelligence
Liu, Y., Liu, Y., Zhao, W., Fan, G.-L., & Zhang, H. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70121
“…of disease development, such as predicting the risk of disease occurrence (Zhou et al. 2024) and monitoring adverse surgical outcomes (Berg et al. 2024), thus providing support for the early warning of diseases, risk assessment, and optimization of treatment strategies (AlSaad et al. 2024).”
Ji, D., Huang, M., & Wang, L. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1913896
Prospective Deployment of Multimodal AI Grading for Medical Student OSCEsPreprint
Ngo, H.-T., Shakur, A. H., Holcomb, M. J., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10475791/v1
Dığış, M., Kaya, K., Demir, B., Cicek, I. N., Akdağ, D., & Akdağ, Ş. (2026) · Cutaneous and Ocular Toxicology · doi:10.1080/15569527.2026.2711643 · Cited by 1
An Immersive Closed‐Loop Human Machine Interaction System Enabled by Dual‐Modal On‐Skin Sensors
Fu, Y., Wang, Z., Guo, W., Bu, T., & Wu, H. (2026) · Advanced Functional Materials · doi:10.1002/adfm.77569
A Multi-modal Framework for Early Disease Detection Using Clinical and Sensor Data
Dash, R., Goswami, S., Pandey, A., Sethi, S., & Jabar, Z. S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27666-7_21
Maciąg, B. M., Bujak, K., Jegierski, D., & Stolarczyk, A. (2026) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-026-06435-9
Zaker, M., Navimipour, N. J., & Hosseinzadeh, M. (2026) · Journal of Urban Technology · doi:10.1080/10630732.2026.2668307 · Cited by 1
Large language models in spine care and research
Galbusera, F., & Cina, A. (2026) · European Spine Journal · doi:10.1007/s00586-026-10204-y
Jiang, Y., & Liu, Y. (2026) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001426400355
Application of Large Language Models Integrating Medical Knowledge Graphs and Multimodal Data
Zheng, C., Du, Y., 侯明帅, & Zhou, S. (2026) · IEEE International Conference on Electronics, Energy Systems and Power Engineering (EESPE) · doi:10.1109/eespe68405.2026.11648282
“Then, the knowledge is represented by the graph embedding technique [5].”
Buhr, C. R., Müller, L., dos Santos, D. P., Thiem, D. G. E., Ponciano, J.-J., Krüger, M., O’Brien, K., Kaufmann, J., Nolte, H., Gartenschläger, M., et al. (2026) · JMIR AI · doi:10.2196/95081 · Cited by 1
Strechen, I., Krishnan, P., Kılıçkaya, O., Barwise, A. K., Koo, C. B. C., Kor, N. T., Deeduvanu, L., Argue, D. P., Herasevich, V., Pickering, B. W., et al. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106673 · Cited by 1
Research on Multimodal AGI Empowering the Development of LLM-based Multi-Agent System
Bai, Z. (2026) · International Journal of Computer Science and Information Technology · doi:10.62051/ijcsit.v8n5.06
“When faced with professional data such as medical imaging, industrial drawings, and industry-specific speech, interpretive errors and information distortion often occur and information distortion [8].”
From Lakehouses to Vector Twins: A Survey of AI-Native Data Management SystemsPreprint
Harby, A. A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7364190
Metadata
- Authors
- AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J.
- Affiliations
- Weill Cornell Medical College in Qatar (QA)
- Hamad bin Khalifa University (QA)
- United Arab Emirates University (AE)
- Corresponding author
- AlSaad, Rawan
- ORCID
- Venue
- Journal of medical Internet research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-09-25
- Volume
- 26
- Pages
- e59505
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11464944
- OpenAlex
- W4402155831
- Cited by
- 349
- References
- 175
- Citations by year
- 2026: 202 · 2025: 139 · 2024: 8
- FWCI
- 32.47
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- multimodal large language models, multimodal data integration, health care applications, medical images, wearable sensor data, electronic health records, clinical notes, audio recordings, surgical video, omics data, clinical decision support, ethical considerations, multimodal fusion
- MeSH terms
- Delivery of Health Care, Humans, Natural Language Processing, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.