The Impact of Multimodal Large Language Models on Health Care’s Future
Open accessPublished version of a preprint
Abstract
When large language models (LLMs) were introduced to the public at large in late 2022 with ChatGPT (OpenAI), the interest was unprecedented, with more than 1 billion unique users within 90 days. Until the introduction of Generative Pre-trained Transformer 4 (GPT-4) in March 2023, these LLMs only contained a single mode—text. As medicine is a multimodal discipline, the potential future versions of LLMs that can handle multimodality—meaning that they could interpret and generate not only text but also images, videos, sound, and even comprehensive documents—can be conceptualized as a significant evolution in the field of artificial intelligence (AI). This paper zooms in on the new potential of generative AI, a new form of AI that also includes tools such as LLMs, through the achievement of multimodal inputs of text, images, and speech on health care’s future. We present several futuristic scenarios to illustrate the potential path forward as multimodal LLMs (M-LLMs) could represent the gateway between health care professionals and using AI for medical purposes. It is important to point out, though, that despite the unprecedented potential of generative AI in the form of M-LLMs, the human touch in medicine remains irreplaceable. AI should be seen as a tool that can augment health care professionals rather than replace them. It is also important to consider the human aspects of health care—empathy, understanding, and the doctor-patient relationship—when deploying AI.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies on M-LLMs in health care have limited exploration of possible solutions or ongoing research aimed at overcoming technical barriers.
“Third, although the challenges of integrating diverse data types into M-LLMs are recognized, there is limited exploration of possible solutions or ongoing research aimed at overcoming these technical barriers [38,39].”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [38]
The ‘Barriers and Limitations’ section identifies significant technical challenges in integrating text, images, video, and sound into M-LLMs. It does not assess whether recent studies have explored solutions or ongoing research to a limited extent; the Conclusions even suggest developing new tokenization strategies as a possible response. No page numbers are available in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredRecent studies on M-LLMs in health care have a restricted range of data modalities examined, often limited to text, images, videos, and audio.
“First, the range of data modalities examined is often restricted to text, images, videos, and audio [38], with some studies focusing narrowly on a limited number of clinical applications [39].”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [38]
The ‘Multimodal Analysis’ section (page unavailable) lists text, images, audio, and video as data types that future M-LLMs could analyze. The ‘Barriers and Limitations’ section says multimodal training data sets are limited, but the paper does not establish that recent studies often restrict the modalities they examine to those four types. (LLM, conf=0.82)
Experimental — accuracy not yet measuredRecent studies on M-LLMs in health care discuss potential uses largely theoretically, leading to a gap in demonstrating their application in actual health care environments.
“Second, the discussion regarding the potential uses of M-LLMs in health care is largely theoretical [38], leading to a significant gap in demonstrating their application in actual health care environments.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · cited 1× · see citing works
“While this is understandable, knowing how difficult and expensive the process of bringing AI-based medical technologies to the practice of medicine is, this setting represents a major barrier to the wide adoption of AI [5].”
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215 · see citing works
Self-citationThe imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · see citing works
Self-citationPrompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · see citing works
Self-citationFederated learning for healthcare: Systematic review and architecture proposal
Antunes, R. S., André da Costa, C., Küderle, A., Yari, I. A., & Eskofier, B. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3501813 · Cited by 595 · cited 1× · see citing works
“As concerns about data privacy continue to grow, future models could focus on analyzing data while preserving user privacy, implementing techniques like federated learning or differential privacy [7,8] (Figure 1).”
Will artificial intelligence solve the human resource crisis in healthcare?
Meskó, B., Hetényi, G., & Győrffy, Z. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3359-4 · Cited by 360 · see citing works
Self-citationHealthcare in the Age of Interoperability: The Promise of Fast Healthcare Interoperability Resources
Braunstein, M. L. (2018) · IEEE Pulse · doi:10.1109/mpul.2018.2869317 · Cited by 53 · cited 1× · see citing works
“Interoperability has been a major challenge in health care for decades [6].”
Towards Generalist Foundation Model for RadiologyPreprint
Xie, W., Wu, C., Zhang, X., Zhang, Y., & Wang, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3324530/v1 · Cited by 33 · see citing works
Taulli, T. (2023) · Apress eBooks · doi:10.1007/978-1-4842-9367-6 · Cited by 29 · see citing works
Forecasting Artificial Intelligence Trends in Health Care: Systematic International Patent Analysis
Benjamens, S., Dhunnoo, P., Görög, M., & Meskó, B. (2023) · JMIR AI · doi:10.2196/47283 · Cited by 10 · see citing works
Self-citationLee, S.-M., Tan, Y.-H., & Yu, H. (2023) · International Conference on Artificial Intelligence and Law · doi:10.1145/3594536.3595144 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [8]Choudhury O Gkoulalas-Divanis A Salonidis T Sylla I Park Y Hsu G Das A Differential privacy-enabled federated learning for sensitive health data ArXiv Preprint posted online on 27 Feb 2020
- [13]Tu T Azizi S Driess D Schaekermann M Amin M Chang PC Carroll A Lau C Tanno R Ktena I Mustafa B Chowdhery A Liu Y Kornblith S Fleet D Mansfield P Prakash S Wong R Virmani S Semturs C Mahdavi SS Green B Dominowska E Aguera y Arcas B Barral J Webster D Corrado GS Matias Y Singhal K Florence P Karthikesalingam A Natarajan V Towards generalist biomedical AI arXiv. Preprint posted online on 26 Jul 2023
- [14]Xu S Yang L Kelly C Sieniek M Kohlberger T Ma M Weng WH Kiraly A Kazemzadeh S Melamed Z Park J Strachan P Liu Y Lau C Singh P Chen C Etemadi M Kalidindi SR Matias Y Chou K Corrado GS Shetty S Tse D Prabhakara S Golden D Pilgrim R Eswaran K Sellergren A ELIXR: towards a general purpose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders arXiv. Preprint posted online on 7 Sep 2023
Cited by
Sorted by citations
Ethical and regulatory challenges of large language models in medicine
Ong, J. C. L., Chang, S. Y.-H., William, W., Butte, A. J., Shah, N. H., Chew, L. S. T., Liu, N., Doshi-Velez, F., Lu, W., Savulescu, J., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00061-x · Cited by 342 · see citing works
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Second, the discussion regarding the potential uses of M-LLMs in health care is largely theoretical [38], leading to a significant gap in demonstrating their application in actual health care environments.”
InfluentialDevelopment and Evaluation of a Retrieval-Augmented Large Language Model Framework for Ophthalmology
Luo, M.-J., Pang, J., Bi, S., Lai, Y., Zhao, J., Shang, Y., Cui, T., Yang, Y., Lin, Z., Zhao, L., et al. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.2513 · Cited by 70 · see citing works
Large language models in physical therapy: time to adapt and adept
Naqvi, W. M., Shaikh, S. Z., & Mishra, G. V. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1364660 · Cited by 24 · see citing works
- Authors
- Meskó, B.
- Corresponding author
- Meskó, Bertalan
- ORCID
- Meskó, Bertalan
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2023-11-02
- Volume
- 25
- Pages
- e52865
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC10654899
- OpenAlex
- W4387583347
- Cited by
- 182
- References
- 12
- Citations by year
- 2026: 31 · 2025: 76 · 2024: 71 · 2023: 4
- FWCI
- 7.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- multimodal large language models, generative AI, health care, future of medicine, artificial intelligence in healthcare, human-AI collaboration, physician-patient relationship, empathy in medicine, GPT-4
- MeSH terms
- Artificial Intelligence, Empathy, Health Personnel, Humans, Language, Physician-Patient Relations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
