The DOI matches its Crossref record; 202 of the 202 references it cites were found in a registry or an index; 44 of the 73 sources its claims cite were checked for support.
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
Open accessPublished version of a preprint
Abstract
In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions. Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg, sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patient interviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics and proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Our investigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research and practical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated clinical AI systems.
Overview
CoChat Summary
## TL;DR
This paper is a narrative review and perspective on multimodal large language models (M-LLMs) in health care, covering their principles, applications, challenges, and future directions. It argues that M-LLMs can extend unimodal LLMs by integrating text, images, audio, video, temporal, and omics data, but real-world use still faces major technical, ethical, and evaluation hurdles.
## Key findings
- The paper identifies 4 core stages in an M-LLM architecture: modality-specific encoding, embedding alignment and fusion, contextual understanding and cross-modal interactions, and decision-making/output generation.
- It argues that current unimodal LLMs are insufficient for medicine because clinical decision-making often combines multiple data types over time, including images, EHRs, laboratory results, audio, and omics data.
- The review highlights wide application areas for M-LLMs, including medical imaging, temporal monitoring, audio analysis, video interpretation, text/EHR analysis, and omics-driven personalized medicine.
- It reports that a general-domain M-LLM study found that such models can “inherently understand medical images” and, in some contexts, even outperform specialized task-specific pretraining methods; the paper uses CONCH as a use case for zero-shot cross-modal retrieval in histopathology.
## Methods
- Narrative review/perspective article synthesizing foundational principles, current and potential applications, technical and ethical challenges, and future outlook for medical M-LLMs.
- The paper organizes the discussion by modality and task: medical images, temporal data, audio, text, video, omics data, and any-to-any M-LLMs.
- It includes a use case example using the CONCH model, run with pretrained weights from Hugging Face on a Linux machine with an NVIDIA GeForce GTX 1080 Ti, in a Flask-based demo for zero-shot cross-modal retrieval on histopathology slides.
## Limitations
- The paper states that recent studies on M-LLMs in health care are limited by a narrow range of modalities, largely theoretical discussion, and limited exploration of solutions to technical barriers.
- The conclusions note that M-LLMs still require further evaluation and benchmarking in real-world medical settings, and that progress toward sophisticated multimodal outputs remains comparatively slow.
- Evident from the text: this is a conceptual review rather than an empirical study, so it does not report original clinical outcome data or a systematic quantitative comparison across models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
29 Valid · 13 Not valid · 12 Not checked · 69 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Third, although the challenges of integrating diverse data types into M-LLMs are recognized, there is limited exploration of possible solutions or ongoing research aimed at overcoming these technical barriers [38,39].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [38] sourceExperimental — accuracy not yet measured
The ‘Barriers and Limitations’ section identifies significant technical challenges in integrating text, images, video, and sound into M-LLMs. It does not assess whether recent studies have explored solutions or ongoing research to a limited extent; the Conclusions even suggest developing new tokenization strategies as a possible response. No page numbers are available in our copy. (LLM, conf=0.84)
- Disputedcites [39] sourceExperimental — accuracy not yet measured
The “Multi-modal LLM” section (page unavailable) describes ongoing research combining images, audio, and text, while acknowledging limited data availability. The “Cross-modal dental diagnosis with LLMs” section (page unavailable) also explores a way to combine text and images for dental diagnosis. The paper therefore does not support the attributed lack of exploration of solutions or ongoing research. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While this unimodal focus on text-based operation has been transformative in the medical field, it does not fully capture the complex and diverse nature of health care practice [8].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [8] sourceExperimental — accuracy not yet measured
The main text says clinical practice requires more than responding accurately to queries and includes practical tasks that LLMs are not equipped to replace. That supports the claim that text-based operation does not capture the full scope of healthcare. But the paper does not describe LLMs as uniformly unimodal: it says GPT-4 accepts images as well as text. It also discusses potential benefits in medicine rather than establishing that text-based operation has already been transformative there. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, the range of data modalities examined is often restricted to text, images, videos, and audio [38], with some studies focusing narrowly on a limited number of clinical applications [39].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Partly supportedcites [38] sourceExperimental — accuracy not yet measured
The ‘Multimodal Analysis’ section (page unavailable) lists text, images, audio, and video as data types that future M-LLMs could analyze. The ‘Barriers and Limitations’ section says multimodal training data sets are limited, but the paper does not establish that recent studies often restrict the modalities they examine to those four types. (LLM, conf=0.82)
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
In “Cross-modal dental diagnosis with LLMs” (page unavailable), the paper says most dental AI studies have used image-only approaches, supporting the point that modalities studied have been limited. “Multi-modal LLM” (page unavailable) discusses text, images, and audio in medical learning, but the paper does not establish that health-care M-LLM studies are often restricted to the four listed modalities, including video. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Applications > Audio
What the sources say
- Could not verifycites [103] sourceExperimental — accuracy not yet measured
The Introduction says large models have advanced more gradually in audio than in language, images, and video. The Related Surveys and Differences section mentions healthcare surveys but does not identify a gap in developing large audio models specifically for medical applications. Searches for medical and healthcare applications found no passage making that claim; page numbers were unavailable in our copy. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Moreover, nontextual data types play a crucial role in diagnosis, effective treatment planning, research, and patient care [9-11].”
§ Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook > Introduction
What the sources say
- Supportedcites [9] source
The Introduction (page not provided) identifies medical images and omics data as relevant to diagnostics, therapy planning, and patient monitoring. “Towards Future Human-AI Interfaces for Multi-Modal Causability” (page not provided) says that integrating images, omics, and other data is needed to understand diseases and biological processes. Together, these passages support the broad claim about nontextual data in research and patient care. (LLM, conf=0.85)
- Partly supportedcites [10] sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) describes using patients’ genomic data to predict early warning signals of cancer, and the Biologic significance section (page unavailable) identifies potential drug targets for early intervention. This supports a narrower point about nontextual molecular data in cancer prediction and treatment research, but not the broad claim about nontextual data types across diagnosis, effective treatment planning, research, and patient care. The paper’s “multi-modal” refers to alternative biomarker configurations, not different data types. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [11] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
34 more claims · 13 with their cited sources checked
Create a free account to see all 123 claims and every source verdict.
- It may state two separate claims as one. 26
- Probably not a substantial claim. 74
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, such as GPT and BERT, are trained on massive datasets and have the unique capability of understanding both structured and unstructured text (11,81).”
From Generative artificial intelligence in predictive analysis of diabetes and its complications: a narrative review (2025) · cites this paper as [81]
Experimental — accuracy not yet measured
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“Time-series encoders (such as transformer-based models [15,59-61] and LSTM [12]) are intended to detect critical changes over time, signaling the need for urgent medical intervention.”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 1×
“For example, image encoders (eg, transformers [54] and convolutional neural networks (CNNs) [55,56]) are designed to capture fine-grained patterns or anomalies crucial for diagnosis, whereas text encoders (BERT [15]) aim to comprehend complex medical terminology and patient histories.”
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Knowledge distillation is another approach that involves training a smaller “student” model to replicate the behavior of a larger “teacher” model, requiring less computational power while retaining performance [161].”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Parameter-efficient fine-tuning methods such as adapter layers help reduce the computational load by fine-tuning only a subset of the model’s parameters [157,158].”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Examples of biomedical text LLMs include BiMediX [104], BioBERT [105], PubMedBERT [106], and ClinicalBERT [20].”
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1×
“In addition, quantization approaches can address the scalability and computational demands by shifting toward quantized versions of existing models using curated, domain-specific data rather than pretraining from scratch [159].”
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“Additional optimization techniques, such as model compression [165] and hardware acceleration using graphics processing units (GPUs) or tensor processing units (TPUs) [166], further enhance efficiency.”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Subsequently, a series of medical BERT models were proposed to accelerate medical research [6,17-20].”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“Parameter-efficient fine-tuning methods such as adapter layers help reduce the computational load by fine-tuning only a subset of the model’s parameters [157,158].”
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401 · cited 2×
“Existing multimodal medical data sets available for public use [84,85,148] are often relatively small in scale and demand the consolidation of numerous resources.”
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · cited 1×
“In proteomics, protein language models are used for secondary structure and contact prediction (ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property…”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 1×
“Examples of biomedical text LLMs include BiMediX [104], BioBERT [105], PubMedBERT [106], and ClinicalBERT [20].”
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 1,989
A universal SNP and small-indel variant caller using deep neural networks
Poplin, R., Chang, P.-C., Alexander, D., Schwartz, S., Colthurst, T., Ku, A., Newburger, D., Dijamco, J., Nguyen, N., Afshar, P. T., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4235 · Cited by 1,840 · cited 1×
“Finally, omics encoders (eg, DeepVariant [62], Basenji [63], and DeepCpG [64]) focus on identifying genetic markers or patterns associated with specific diseases, aiding in the development of targeted therapies.”
Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab083 · Cited by 1,303
ShareGPT4V: Improving Large Multi-Modal Models with Better CaptionsPreprint
Chen, L., Li, J., Dong, X.-W., Zhang, P., He, C.-H., Wang, J., Zhao, F., & Lin, D.-H. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.12793 · Cited by 1,271 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Identifying Personal Genomes by Surname Inference
Gymrek, M., McGuire, A. L., Golan, D., Halperin, E., & Erlich, Y. (2013) · Science · doi:10.1126/science.1229566 · Cited by 1,228 · cited 1×
“Other studies have shown that, even when sensitive patient data are anonymized, certain algorithms can still identify individual patients [189-191].”
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video UnderstandingPreprint
Zhang, H., Li, X. S., & Bing, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02858 · Cited by 1,165 · cited 2×
“Similarly, models such as Video-Chat [47] and Video-LLaMA [48] are designed for video comprehension, whereas SpeechGPT [49] is tailored for audio processing.”
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078 · cited 1×
“…proteomics, protein language models are used for secondary structure and contact prediction (ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property…”
Kelley, D. R., Snoek, J., & Rinn, J. L. (2016) · Genome Research · doi:10.1101/gr.200535.115 · Cited by 1,048 · cited 1×
“Finally, omics encoders (eg, DeepVariant [62], Basenji [63], and DeepCpG [64]) focus on identifying genetic markers or patterns associated with specific diseases, aiding in the development of targeted therapies.”
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · cited 1×
“In health care, LLMs have shown remarkable potential, primarily through their ability to process and analyze textual content [2,3].”
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009 · cited 1×
“…(ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property prediction (PromptProtein [131]), and advancing the state of the art in proteomics [132,133].”
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1240 · Cited by 986
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966 · cited 2×
“In this section, we present a use case that demonstrates the practical application of M-LLMs in health care using the Contrastive Learning From Captions for Histopathology (CONCH) model [141].”
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863
A Survey of Model Compression and Acceleration for Deep Neural Networks
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2017) · arXiv · doi:10.48550/arxiv.1710.09282 · Cited by 857 · cited 1×
“Additional optimization techniques, such as model compression [165] and hardware acceleration using graphics processing units (GPUs) or tensor processing units (TPUs) [166], further enhance efficiency.”
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · cited 2×
“M-LLMs are instrumental in extracting meaningful insights from EHRs, which encompass diverse and nonregularly timed medical events [89].”
MotionGPT: Human Motion as a Foreign Language
Jiang, B., Chen, X., Wen, L. I. U., Jingyi Yu, A., Yu, G., & Chen, T. (2023) · neural information processing systems · doi:10.52202/075280-0880 · Cited by 817
NExT-GPT: Any-to-Any Multimodal LLMPreprint
Wu, S., Fei, H., Qu, L., Ji, W., & Chua, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05519 · Cited by 791 · cited 1×
“For instance, NExT-GPT [136] enhances an LLM with multimodal adapters and a range of diffusion decoders, enabling the model to process and generate outputs in any combination of text, images, videos, and audio.”
Med-Flamingo: a Multimodal Medical Few-shot LearnerPreprint
Moor, M., Huang, Q., Wu, S., Yasunaga, M., Zakka, C., Dalmia, Y., Reis, E. P., Rajpurkar, P., & Leskovec, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.15189 · Cited by 752 · cited 1×
“Such M-LLMs with vision capabilities can be adapted for medical imaging applications (eg, LLaVA-Med [84], PMC-VQA [85], Med-Flamingo [86], and PeFoMed [87]).”
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750 · cited 1×
“It is a careful balance of debiasing and enhancing the models, requiring ongoing monitoring and adjustment to align with ethical standards, particularly in the sensitive domain of health care [185].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
DeepTrust^RT: Confidential Deep Neural Inference Meets Real-Time!
Babar, M. F., & Hasan, M. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.ecrts.2024.13 · Cited by 708
LLaMA-VID: An Image is Worth 2 Tokens in Large Language ModelsPreprint
Li, Y., Wang, C.-Y., & Jia, J. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.17043 · Cited by 684 · cited 1×
“The progress in M-LLMs for video data analysis, demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development…”
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · cited 3×
“For instance, Flamingo [45] uses a cross-attention layer to link a frozen image encoder with LLMs. LLaVA [46] uses a basic projection method to incorporate image features into the word embedding space.”
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681 · cited 4×
“This challenge necessitates the development of advanced methods for explaining transformer-based language models [167,168], particularly methods for local explanations, such as feature attribution explanation, attention-based explanation, example-based explanation, and natural language explanation…”
InfluentialAudioPaLM: A Large Language Model That Can Speak and ListenPreprint
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., Quitry, F. D. C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.12925 · Cited by 671 · cited 1×
“Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
Artificial Intelligence in Health Care: A Report From the National Academy of Medicine
Matheny, M. E., Whicher, D., & Thadaney Israni, S. (2020) · JAMA · doi:10.1001/jama.2019.21579 · Cited by 628
(2019) · doi:10.18653/v1/n19-1419 · Cited by 601
BEHRT: Transformer for Electronic Health Records
Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., & Salimi-Khorshidi, G. (2020) · Scientific Reports · doi:10.1038/s41598-020-62922-y · Cited by 595
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 1×
“Current M-LLMs are primarily limited to multimodal comprehension on the input side, possessing limited capabilities to generate content across various modalities [134,135].”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · cited 3×
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
DeepCpG: accurate prediction of single-cell DNA methylation states using deep learning
Angermueller, C., Lee, H. J., Reik, W., & Stegle, O. (2017) · Genome Biology · doi:10.1186/s13059-017-1189-z · Cited by 536 · cited 1×
“Finally, omics encoders (eg, DeepVariant [62], Basenji [63], and DeepCpG [64]) focus on identifying genetic markers or patterns associated with specific diseases, aiding in the development of targeted therapies.”
Multimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528 · cited 1×
“Finally, multimodality characterizes systems designed to generate outputs in >1 modality, such as systems capable of producing both textual and image-based content [40].”
PandaGPT: One Model To Instruction-Follow Them AllPreprint
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., & Cai, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16355 · Cited by 488 · cited 1×
“A notable example is PandaGPT [50], which uniquely manages to interpret 6 different modalities at the same time, achieved through the integration of a multimodal encoder known as ImageBind [51].”
Algorithms to estimate Shapley value feature attributions
Chen, H., Covert, I. C., Lundberg, S. M., & Lee, S.-I. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00657-x · Cited by 472
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30570 · Cited by 454
LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
Chang, C., Wang, W., Peng, W.-C., & Chen, T.-F. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3719207 · Cited by 452
VTimeLLM: Empower LLM to Grasp Video Moments
Huang, B., Wang, X. Y., Chen, H., Song, Z., & Zhu, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01353 · Cited by 441
Audiolm: A language modeling approach to audio generation
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3288409 · Cited by 432 · cited 1×
“Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
Maaz, M., Rasheed, H., Khan, S., & Khan, F. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.679 · Cited by 424
Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Han, X., Qiu, H., Lin, C., Shao, W., Chen, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07575 · Cited by 378 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Identity inference of genomic data using long-range familial searches
Erlich, Y., Shor, T., Pe’er, I., & Carmi, S. (2018) · Science · doi:10.1126/science.aau4832 · Cited by 367 · cited 1×
“Other studies have shown that, even when sensitive patient data are anonymized, certain algorithms can still identify individual patients [189-191].”
Biases in Large Language Models: Origins, Inventory, and Discussion
Navigli, R., Conia, S., & Ross, B. (2023) · Journal of Data and Information Quality · doi:10.1145/3597307 · Cited by 364 · cited 1×
“Mitigating bias and improving fairness within medical M-LLMs necessitates a multifaceted approach centered on 3 pillars: data integrity, model refinement, and comprehensive evaluation [181,182].”
Ethical and regulatory challenges of large language models in medicine
Ong, J. C. L., Chang, S. Y.-H., William, W., Butte, A. J., Shah, N. H., Chew, L. S. T., Liu, N., Doshi-Velez, F., Lu, W., Savulescu, J., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00061-x · Cited by 342 · cited 3×
“If M-LLMs are trained on patient data that contain biases related to gender, ethnicity, socioeconomic status, or geographic location, they may inadvertently cause biases in their predictions or recommendations [175,178,179].”
ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model
Huang, H., Zheng, O., Wang, D., Yin, J., Wang, Z., Ding, S., Yin, H., Xu, C., Yang, R., Zheng, Q., et al. (2023) · International Journal of Oral Science · doi:10.1038/s41368-023-00239-y · Cited by 324 · cited 3×
“Recent studies [38,39] have explored the capabilities of M-LLMs within the health care sector.”
OneLLM: One Framework to Align All Modalities with Language
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., & Yue, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02510 · Cited by 309
PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting
Xue, H., & Salim, F. D. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3342137 · Cited by 302 · cited 1×
“Significant advancements have been made in M-LLMs with temporal analysis capabilities, including models such as Time-LLM [90], LLM4TS [91], TEMPO [92], and PromptCast [93], among others [94,95].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01370 · Cited by 281
Towards artificial general intelligence via a multimodal foundation model
Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30761-2 · Cited by 280 · cited 1×
“This underscores the importance of developing any-to-any M-LLMs, which are crucial for realizing real artificial general intelligence (AGI) [52,53].”
Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01728 · Cited by 276
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., & Tu, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27999 · Cited by 272
Automated Concatenation of Embeddings for Structured Prediction
Wang, X., Jiang, Y., Bách, N., Wang, T., Huang, Z., Huang, F., & Tu, K. (2021) · doi:10.18653/v1/2021.acl-long.206 · Cited by 270 · cited 1×
“This might involve techniques such as concatenation [65] and attention mechanisms [13] or more sophisticated methods such as cross-modal attention [66,67] and tensor fusion [68].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., & Xia, F. (2024) · arXiv · doi:10.48550/arxiv.2401.12168 · Cited by 269 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Yu, P., Xu, H., Hu, X., & Deng, C. (2023) · Healthcare · doi:10.3390/healthcare11202776 · Cited by 257 · cited 1×
“These models play a crucial role in assisting with diagnoses as they can efficiently process extensive textual patient histories and vast medical literature, providing clinicians with valuable insights [4-7].”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
Yang, A., Nagrani, A., Seo, P. H., Miech, A., Pont-Tuset, J., Laptev, I., Sivic, J., & Schmid, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01032 · Cited by 254 · cited 1×
“…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook
Jin, M., Kong, Y., Liang, Y., Zhang, C., Xue, S., Wang, X., Zhang, J., Wang, Y., Chen, H., Li, X., et al. (2026) · ACM Computing Surveys · doi:10.1145/3821637 · Cited by 246 · cited 1×
“Significant advancements have been made in M-LLMs with temporal analysis capabilities, including models such as Time-LLM [90], LLM4TS [91], TEMPO [92], and PromptCast [93], among others [94,95].”
Osprey: Pixel Understanding with Visual Instruction Tuning
Yuan, Y., Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L., & Zhu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02664 · Cited by 241
ShareGPT4V: Improving Large Multi-modal Models with Better Captions
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., & Lin, D. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72643-9_22 · Cited by 237
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235 · cited 1×
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
Large Language Models are Few-Shot Health LearnersPreprint
Liu, X., McDuff, D., Kovacs, G., Galatzer‐Levy, I. R., Sunshine, J. E., Zhan, J., Poh, M., Liao, S., Achille, P. D., & Patel, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15525 · Cited by 219 · cited 1×
“Some of the existing M-LLMs with temporal capabilities could be adapted for medical applications [89,96], or new models specifically designed and pretrained on medical temporal data can be developed.”
Privacy-Preserving Federated Learning for Internet of Medical Things Under Edge Computing
Wang, R., Lai, J., Zhang, Z., Li, X., Vijayakumar, P., & Karuppiah, M. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3157725 · Cited by 203
TimeLLM: Time Series Forecasting by Reprogramming Large Language Models
Jin, M., Pan, S., Li, Y.-F., Ma, L., Zhang, J. Y., Chu, Z., Wang, S., Shi, X., Chen, P., Wen, Q., et al. (2026) · CRC Press eBooks · doi:10.1201/9781003616719-7 · Cited by 197
Multi-Modal Graph Learning for Disease Prediction
Zheng, S., Zhu, Z., Liu, Z., Guo, Z., Liu, Y., Yang, Y., & Zhao, Y. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3159264 · Cited by 196 · cited 1×
“Moreover, nontextual data types play a crucial role in diagnosis, effective treatment planning, research, and patient care [9-11].”
SVIT: Scaling up Visual Instruction TuningPreprint
Zhao, B., Wu, B., He, M., & Huang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04087 · Cited by 193 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2308.12966 · Cited by 193 · cited 1×
“Significant progress has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46],…”
Large Language Models Are Zero-Shot Time Series Forecasters
Gruver, N., Finzi, M., Qiu, S., & Wilson, A. (2023) · neural information processing systems · doi:10.52202/075280-0861 · Cited by 192
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · neural information processing systems · doi:10.52202/075280-3155 · Cited by 191
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2305.11000 · Cited by 189
Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology
Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.35237 · Cited by 187
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.1055 · Cited by 187
VideoLLM: Modeling Video Sequence with Large Language ModelsPreprint
Guo, C., Zheng, Y.-D., Wang, J., Xu, J., Huang, Y., Pan, J., Wang, Y., Wang, Y., Qiao, Y., Lü, T., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13292 · Cited by 185 · cited 1×
“…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
VideoChat: chat-centric video understanding
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., & Qiao, Y. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4321-9 · Cited by 185
The Impact of Multimodal Large Language Models on Health Care’s Future
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/52865 · Cited by 182 · cited 4×
“Second, the discussion regarding the potential uses of M-LLMs in health care is largely theoretical [38], leading to a significant gap in demonstrating their application in actual health care environments.”
InfluentialDNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R., & Liu, H. (2023) · arXiv · doi:10.48550/arxiv.2306.15006 · Cited by 179 · cited 1×
“These tasks include predicting genome-wide variant effects (GPN [116]; DNABERT [117]; and its subsequent evolution, DNABERT-2 [118]), predicting DNA cis-regulatory regions (DNAGPT [119], DNABERT, and DNABERT-2), predicting DNA-protein interactions (TFBert [120] and MoDNA [121]), and determining RNA…”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.16449 · Cited by 176 · cited 1×
“…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01725 · Cited by 176
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., & Glass, J. R. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33016309 · Cited by 174 · cited 1×
“…feature attribution explanation, attention-based explanation, example-based explanation, and natural language explanation [169-172], and global explanations, such as probing-based explanation, neuron activation explanation, concept-based explanation, and mechanistic interpretability [168,173,174].”
ProtST: Multi-Modality Learning of Protein Sequences and Biomedical TextsPreprint
Xu, M., Yuan, X., Miret, S., & Tang, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12040 · Cited by 172 · cited 1×
“…for secondary structure and contact prediction (ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property prediction (PromptProtein [131]), and advancing the…”
Informed Consent and Medical Artificial Intelligence: What to Tell the Patient?
Cohen, I. G. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3529576 · Cited by 165 · cited 1×
“In particular, it remains uncertain whether patient consent is necessary for training M-LLMs using their data if consent was previously obtained for research purposes in general or for AI development specifically [178,186].”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models
Li, Y., Wang, C., & Jia, J. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72952-2_19 · Cited by 162
Na, L., Yang, C., Lo, C.-C., Zhao, F., Fukuoka, Y., & Aswani, A. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.6040 · Cited by 161
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.523 · Cited by 161 · cited 1×
“…explanations, such as feature attribution explanation, attention-based explanation, example-based explanation, and natural language explanation [169-172], and global explanations, such as probing-based explanation, neuron activation explanation, concept-based explanation, and mechanistic…”
Supervised Multimodal Bitransformers for Classifying Images and Text
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., & Testuggine, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02950 · Cited by 159 · cited 2×
“This might involve techniques such as concatenation [65] and attention mechanisms [13] or more sophisticated methods such as cross-modal attention [66,67] and tensor fusion [68].”
Muse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157 · cited 1×
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
DNA language models are powerful predictors of genome-wide variant effects
Benegas, G., Batra, S. S., & Song, Y. S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311219120 · Cited by 157
Li, Y., Mamouei, M., Salimi-Khorshidi, G., Rao, S., Hassaine, A., Canoy, D., Lukasiewicz, T., & Rahimi, K. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3224727 · Cited by 156 · cited 1×
“Subsequently, a series of medical BERT models were proposed to accelerate medical research [6,17-20].”
LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
Chen, G., Shen, L., Shao, R., Deng, X., & Nie, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02506 · Cited by 152
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models
Hoover, B., Strobelt, H., & Gehrmann, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.22 · Cited by 151
MotionGPT: Human Motion as a Foreign Language
Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., & Chen, T. (2023) · arXiv · doi:10.48550/arxiv.2306.14795 · Cited by 149 · cited 1×
“…M-LLMs for video data analysis, demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models…”
VioLA: Conditional Language Models for Speech Recognition, Synthesis, and Translation
Wang, T., Zhou, L., Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J., & Wei, F. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434425 · Cited by 148 · cited 1×
“Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
Chang, C., Wang, W.-Y., Peng, W.-C., & Chen, T.-F. (2023) · arXiv · doi:10.48550/arxiv.2308.08469 · Cited by 146
Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models
Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021) · arXiv · doi:10.48550/arxiv.2102.02503 · Cited by 135 · cited 1×
“Large language models (LLMs) are sophisticated machine learning algorithms designed to process, understand, and generate humanlike language, enabling key developments in applications such as automated conversation, text analysis, creative writing, and complex problem-solving [1].”
Huang, Y., Gomaa, A., Semrau, S., Haderlein, M., Lettmaier, S., Weißmann, T., Grigo, J., Tkhayat, H. B., Frey, B., Gaipl, U. S., et al. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1265024 · Cited by 131
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12995 · Cited by 131
Horiuchi, D., Tatekawa, H., Shimono, T., Walston, S. L., Takita, H., Matsushita, S., Oura, T., Mitsuyama, Y., Miki, Y., & Ueda, D. (2023) · Neuroradiology · doi:10.1007/s00234-023-03252-4 · Cited by 127
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Li, L., Gan, Z., Lin, K., Lin, C.-C., Liu, Z., Liu, C., & Wang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02214 · Cited by 125 · cited 1×
“…data analysis, demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical…”
DNAGPT: A Generalized Pre-trained Tool for Multiple DNA Sequence Analysis TasksPreprint
Zhang, D., Zhang, W., Zhao, Y., Zhang, J., He, B., Qin, C., & Yao, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.11.548628 · Cited by 124 · cited 1×
“…variant effects (GPN [116]; DNABERT [117]; and its subsequent evolution, DNABERT-2 [118]), predicting DNA cis-regulatory regions (DNAGPT [119], DNABERT, and DNABERT-2), predicting DNA-protein interactions (TFBert [120] and MoDNA [121]), and determining RNA splice sites from DNA…”
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · cited 1×
“Pretraining and fine-tuning are fundamental processes in the development and optimization of LLMs, including multimodal ones [69].”
Mihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2249588 · Cited by 121
NetGO 3.0: Protein Language Model Improves Large-Scale Functional Annotations
Wang, S., You, R., Liu, Y., Xiong, Y., & Zhu, S. (2023) · Genomics Proteomics & Bioinformatics · doi:10.1016/j.gpb.2023.04.001 · Cited by 106 · cited 1×
“…(ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property prediction (PromptProtein [131]), and advancing the state of the art in proteomics [132,133].”
Wavelet-Improved Score-Based Generative Model for Medical Imaging
Wu, W., Wang, Y., Liu, Q., Wang, G., & Zhang, J. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3325824 · Cited by 105 · cited 1×
“Synthetic data generation using generative models can create realistic, diverse data sets that mimic real-world multimodal medical scenarios, thus expanding the training data set without compromising privacy or ethical standards [149-151].”
Vary: Scaling up the Vision Vocabulary for Large Vision-Language ModelsPreprint
Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang, J., Sun, J., Han, C., & Zhang, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.06109 · Cited by 104 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Local Interpretations for Explainable Natural Language Processing: A Survey
Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024) · ACM Computing Surveys · doi:10.1145/3649450 · Cited by 103 · cited 1×
“…explanations, such as feature attribution explanation, attention-based explanation, example-based explanation, and natural language explanation [169-172], and global explanations, such as probing-based explanation, neuron activation explanation, concept-based explanation, and mechanistic…”
OtterHD: A High-Resolution Multi-modality ModelPreprint
Li, B., Zhang, P., Yang, J., Zhang, Y., Pu, F., & Liu, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04219 · Cited by 100 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Pengi: An Audio Language Model for Audio TasksPreprint
Deshmukh, S., Elizalde, B., Singh, R., & Wang, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11834 · Cited by 99 · cited 1×
“Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
Large Language Models Are Zero-Shot Time Series ForecastersPreprint
Gruver, N., Finzi, M., Qiu, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07820 · Cited by 96 · cited 1×
“Significant advancements have been made in M-LLMs with temporal analysis capabilities, including models such as Time-LLM [90], LLM4TS [91], TEMPO [92], and PromptCast [93], among others [94,95].”
Large language models improve annotation of prokaryotic viral proteins
Flamholz, Z. N., Biller, S. J., & Kelly, L. (2024) · Nature Microbiology · doi:10.1038/s41564-023-01584-8 · Cited by 92 · cited 1×
“…(ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property prediction (PromptProtein [131]), and advancing the state of the art in proteomics [132,133].”
VideoChat: Chat-Centric Video UnderstandingPreprint
Li, K., He, Y., Wang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06355 · Cited by 89 · cited 2×
“Similarly, models such as Video-Chat [47] and Video-LLaMA [48] are designed for video comprehension, whereas SpeechGPT [49] is tailored for audio processing.”
Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae163 · Cited by 89
Explainable AI and Multi-Modal Causability in Medicine
Holzinger, A. (2021) · i-com · doi:10.1515/icom-2020-0024 · Cited by 87 · cited 1×
“Moreover, nontextual data types play a crucial role in diagnosis, effective treatment planning, research, and patient care [9-11].”
Mallio, C. A., Sertorio, A. C., Bernetti, C., & Zobel, B. B. (2023) · La radiologia medica · doi:10.1007/s11547-023-01651-4 · Cited by 86
Multimodal sentiment analysis based on multi-head attention mechanism
Chen, X., Lu, G., & Yan, J. (2020) · International Conference on Machine Learning and Soft Computing (ICMLSC) · doi:10.1145/3380688.3380693 · Cited by 80 · cited 2×
“This might involve techniques such as concatenation [65] and attention mechanisms [13] or more sophisticated methods such as cross-modal attention [66,67] and tensor fusion [68].”
Prediction of RNA–protein interactions using a nucleotide language model
Yamada, K., & Hamada, M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac023 · Cited by 80 · cited 1×
“…prediction (SpliceBERT [122]), assessment of long noncoding RNAs’ coding potential (LncCat [123]), RNA-binding protein interactions (BERT-RBP [124]), RNA modification identification (BERT-m7G [125]), and predictions related to protein expression and messenger RNA degradation (CodonBERT…”
Koga, S., Martin, N. B., & Dickson, D. W. (2024) · Brain Pathology · doi:10.1111/bpa.13207 · Cited by 78
Rise of artificial general intelligence: risks and opportunities
Buttazzo, G. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1226990 · Cited by 76 · cited 1×
“This underscores the importance of developing any-to-any M-LLMs, which are crucial for realizing real artificial general intelligence (AGI) [52,53].”
Mantey, E. A., Zhou, C., Anajemba, J. H., Arthur, J. K., Hamid, Y., Chowhan, A., & Otuu, O. O. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3350232 · Cited by 75 · cited 2×
“One of these measures is the implementation of federated learning techniques [153,155,156] to enable M-LLMs to be trained on decentralized data sources without the need to transfer sensitive or private information to a central location, thereby preserving data privacy and security.”
Advancing bioinformatics with language models: components, applications, and perspectives
Liu, J., Yang, M., Yu, Y., Xu, H., Wang, T., Li, K., & Zhou, X. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag367 · Cited by 72
MR-Transformer: Multiresolution Transformer for Multivariate Time Series Prediction
Zhu, S., Zheng, J., & Ma, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3327416 · Cited by 69 · cited 1×
“Time-series encoders (such as transformer-based models [15,59-61] and LSTM [12]) are intended to detect critical changes over time, signaling the need for urgent medical intervention.”
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
BiMediX: Bilingual Medical Mixture of Experts LLM
Pieri, S., Mullappilly, S. S., Khan, F. S., Anwer, R. M., Khan, S., Baldwin, T., & Cholakkal, H. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.989 · Cited by 69 · cited 1×
“Examples of biomedical text LLMs include BiMediX [104], BioBERT [105], PubMedBERT [106], and ClinicalBERT [20].”
MT-clinical BERT: scaling clinical information extraction with multitask learning
Mulyar, A., Uzuner, Ö., & McInnes, B. T. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab126 · Cited by 67 · cited 2×
“Examples of biomedical text LLMs include BiMediX [104], BioBERT [105], PubMedBERT [106], and ClinicalBERT [20].”
Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment
Lai, U. H., Wu, K. S., Hsu, T.-Y., & Kan, J. K. C. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1240915 · Cited by 67
Improving radiology workflow using ChatGPT and artificial intelligence
Meşe, İ., Taşlıçay, C. A., & Sivrioğlu, A. K. (2023) · Clinical Imaging · doi:10.1016/j.clinimag.2023.109993 · Cited by 67
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67 · cited 1×
“Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66 · cited 3×
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
Ho, Q., Nguyen, T., Le, N. Q. K., & Ou, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104258 · Cited by 65
Large language models will not replace healthcare professionals: curbing popular fears and hype.
Thirunavukarasu, A. J. (2023) · Journal of the Royal Society of Medicine · doi:10.1177/01410768231173123 · Cited by 63 · cited 1×
“While this unimodal focus on text-based operation has been transformative in the medical field, it does not fully capture the complex and diverse nature of health care practice [8].”
Efficient Multimodal Fusion via Interactive Prompting
Li, Y., Quan, R., Zhu, L., & Yang, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00256 · Cited by 60 · cited 1×
“Prompt-based multimodal fusion [143] is one such framework that enables bidirectional interaction among different modalities through a 2-stream structure, typically involving parallel construction of the multimodal model through pretrained language and image models.”
TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting
Cao, D., Jia, F., Arik, S. O., Pfister, T., Zheng, Y., Ye, W., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.04948 · Cited by 59 · cited 1×
“Significant advancements have been made in M-LLMs with temporal analysis capabilities, including models such as Time-LLM [90], LLM4TS [91], TEMPO [92], and PromptCast [93], among others [94,95].”
An Empirical Study of Scaling Instruct-Tuned Large Multimodal ModelsPreprint
Lu, Y., Li, C., Liu, H., Yang, J., Gao, J., & Shen, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.09958 · Cited by 55 · cited 1×
“This method capitalizes on the foundational strengths of established models, significantly reducing the computational resources needed for initial training [160].”
Model-Heterogeneous Semi-Supervised Federated Learning for Medical Image Segmentation
Ma, Y., Wang, J., Yang, J., & Wang, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3348982 · Cited by 55 · cited 1×
“One of these measures is the implementation of federated learning techniques [153,155,156] to enable M-LLMs to be trained on decentralized data sources without the need to transfer sensitive or private information to a central location, thereby preserving data privacy and security.”
Governing AI‐Driven Health Research: Are IRBs Up to the Task?
Friesen, P., Douglas‐Jones, R., Marks, M., Pierce, R., Fletcher, K., Mishra, A., Lorimer, J., Véliz, C., Hallowell, N., Graham, M., et al. (2021) · Ethics & Human Research · doi:10.1002/eahr.500085 · Cited by 50 · cited 1×
“In addition, it can be argued that traditional institutional review boards (IRBs) and ethical oversight committees may be ill-equipped to deal with AI and M-LLM applications due to the lack of understanding of such novel technologies in the medical arena [187].”
Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.31.526427 · Cited by 43 · cited 1×
“In transcriptomics, RNA sequence language models are used for RNA splicing prediction (SpliceBERT [122]), assessment of long noncoding RNAs’ coding potential (LncCat [123]), RNA-binding protein interactions (BERT-RBP [124]), RNA modification identification (BERT-m7G [125]), and predictions related…”
Gobira, M., Nakayama, L. F., Moreira, R., de Andrade, E. P., Regatieri, C. V., & Belfort, R. (2023) · Revista da Associação Médica Brasileira · doi:10.1590/1806-9282.20230848 · Cited by 43 · cited 1×
“Following the release of GPT-4, several medically adapted versions of this model have been developed [33-37].”
InfluentialPFB-Diff: Progressive Feature Blending diffusion for text-driven image editing
Huang, W., Tu, S., & Xu, L. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106777 · Cited by 41
Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2306.09093 · Cited by 38 · cited 1×
“Macaw-LLM [137] integrates images, audio, and textual data using 3 primary components: a modality module for encoding multimodal data, a cognitive module for leveraging pretrained LLMs, and an alignment module for synchronizing diverse representations.”
Zhang, L., Qin, X., Liu, M., Liu, G., & Ren, Y. (2021) · Computational and Mathematical Methods in Medicine · doi:10.1155/2021/7764764 · Cited by 35 · cited 1×
“…prediction (SpliceBERT [122]), assessment of long noncoding RNAs’ coding potential (LncCat [123]), RNA-binding protein interactions (BERT-RBP [124]), RNA modification identification (BERT-m7G [125]), and predictions related to protein expression and messenger RNA degradation (CodonBERT [126]).”
Automatic Detection of Abnormal EEG Signals Using WaveNet and LSTM
Albaqami, H., Hassan, G. M., & Datta, A. (2023) · Sensors · doi:10.3390/s23135960 · Cited by 32 · cited 1×
“Similarly, audio encoders (such as WaveNet [57] and DeepSpeech [58]) are optimized to distinguish subtle variations in sounds, such as differentiating between normal and abnormal heart or lung sounds.”
Luo, H., Shan, W., Chen, C., Ding, P., & Luo, L. (2022) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-022-00537-9 · Cited by 31 · cited 1×
“…(GPN [116]; DNABERT [117]; and its subsequent evolution, DNABERT-2 [118]), predicting DNA cis-regulatory regions (DNAGPT [119], DNABERT, and DNABERT-2), predicting DNA-protein interactions (TFBert [120] and MoDNA [121]), and determining RNA splice sites from DNA sequences (DNABERT and DNABERT-2).”
PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00426 · Cited by 30 · cited 1×
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality CollaborationPreprint
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., Huang, F., & Zhou, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04257 · Cited by 28 · cited 1×
“These models, among others [139,140], can be tailored and fine-tuned to specifically address the unique demands of tasks related to health care.”
An, W., Guo, Y., Bian, Y., Ma, H., Yang, J., Li, C., & Huang, J. (2022) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3535508.3545512 · Cited by 27 · cited 1×
“…(GPN [116]; DNABERT [117]; and its subsequent evolution, DNABERT-2 [118]), predicting DNA cis-regulatory regions (DNAGPT [119], DNABERT, and DNABERT-2), predicting DNA-protein interactions (TFBert [120] and MoDNA [121]), and determining RNA splice sites from DNA sequences (DNABERT and DNABERT-2).”
Anomaly Detection in Time Series Data Using Reversible Instance Normalized Anomaly Transformer
Baidya, R., & Jeong, H. (2023) · Sensors · doi:10.3390/s23229272 · Cited by 26
Quan, Z., Sun, T., Su, M., & Wei, J. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/4767437 · Cited by 25 · cited 1×
“Gated fusion [145,146] uses mechanisms such as neural network gates or attention mechanisms to dynamically emphasize or de-emphasize different aspects or modalities of the data based on the context.”
PAFormer: Anomaly Detection of Time Series With Parallel-Attention Transformer
Bai, N., Wang, X., Han, R., Wang, Q., & Liu, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3337876 · Cited by 25 · cited 1×
“Time-series encoders (such as transformer-based models [15,59-61] and LSTM [12]) are intended to detect critical changes over time, signaling the need for urgent medical intervention.”
Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models
Ye, J., Hai, J., Song, J., & Wang, Z. (2023) · doi:10.1101/2023.08.24.23294597 · Cited by 24 · cited 1×
“Hybrid fusion [144] integrates unstructured and structured data along with other multimodal sources via a pretrained language model, capturing a more comprehensive patient representation.”
Yue, G., Wei, P., Zhou, T., Song, Y., Zhao, C., Wang, T., & Lei, B. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3319516 · Cited by 24 · cited 1×
“One of these measures is the implementation of federated learning techniques [153,155,156] to enable M-LLMs to be trained on decentralized data sources without the need to transfer sensitive or private information to a central location, thereby preserving data privacy and security.”
Liang, J., Li, Z.-W., Yue, C.-T., Hu, Z., Cheng, H., Liu, Z.-X., & Guo, W.-F. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac254 · Cited by 23
A distribution information sharing federated learning approach for medical image data
Zhao, L., & Huang, J. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01035-1 · Cited by 23 · cited 1×
“In addition, federated learning presents a viable solution for leveraging multimodal data from multiple health care institutions without the need to share the actual data, thus maintaining patient privacy [152-156].”
AlSaad, R., Malluhi, Q. M., Janahi, I. A., & Boughorbel, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0951-4 · Cited by 22 · cited 1×
“While the computation in architectures such as RNNs and LSTM networks is sequential and slow for long sequences [14], self-attention can be parallelized and made highly scalable.”
Self-citationLi, J., Chen, D., Hong, Y., Chen, Z., Chen, P., Shen, Y., & Gan, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.03354 · Cited by 22 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Empowering radiology: the transformative role of ChatGPT
Arachchige, A. S. P. M. (2023) · Clinical Radiology · doi:10.1016/j.crad.2023.08.006 · Cited by 19 · cited 1×
“It provided opportunities for advancing diagnostic accuracy, personalized treatment planning, and medical research, as well as optimizing health care administration and enhancing communication in patient care [22-28].”
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic TasksPreprint
Chen, Z. S., Wu, J., Wang, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.14238 · Cited by 19 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Multimodal Large Language Models are Generalist Medical Image Interpreters
Han, T., Adams, L. C., Nebelung, S., Kather, J. N., Bressem, K. K., & Truhn, D. (2023) · doi:10.1101/2023.12.21.23300146 · Cited by 18 · cited 1×
“Previous work [88] evaluated the performance of a general-domain M-LLM in biomedical image classification tasks.”
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup
Yordanov, Y., Kocijan, V., Lukasiewicz, T., & Camburu, O.-M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.255 · Cited by 18
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · cited 1×
“However, there is a gap in the development of large audio models specifically tailored for medical applications [103].”
Feng, H., Wang, S., Wang, Y., Ni, X., Yang, Z., Hu, X., & Yang, S. (2022) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.02.012 · Cited by 16 · cited 1×
“…RNA sequence language models are used for RNA splicing prediction (SpliceBERT [122]), assessment of long noncoding RNAs’ coding potential (LncCat [123]), RNA-binding protein interactions (BERT-RBP [124]), RNA modification identification (BERT-m7G [125]), and predictions related to protein…”
ImageBind: One Embedding Space To Bind Them AllPreprint
Girdhar, R., El-Nouby, A., Liu, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05665 · Cited by 14 · cited 1×
“A notable example is PandaGPT [50], which uniquely manages to interpret 6 different modalities at the same time, achieved through the integration of a multimodal encoder known as ImageBind [51].”
Botelho, F. M. N., Tshimula, J. M., & Poenaru, D. (2023) · World Journal of Surgery · doi:10.1007/s00268-023-07167-2 · Cited by 12 · cited 1×
“These models play a crucial role in assisting with diagnoses as they can efficiently process extensive textual patient histories and vast medical literature, providing clinicians with valuable insights [4-7].”
PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical ImagingPreprint
He, J., Li, P., Liu, G., He, G., Chen, Z., & Zhong, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.02797 · Cited by 12
CodonBERT: Using BERT for Sentiment Analysis to Better Predict Genes with Low Expression
Babjac, A., Lu, Z., & Emrich, S. (2023) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3584371.3613013 · Cited by 11 · cited 1×
“…prediction (SpliceBERT [122]), assessment of long noncoding RNAs’ coding potential (LncCat [123]), RNA-binding protein interactions (BERT-RBP [124]), RNA modification identification (BERT-m7G [125]), and predictions related to protein expression and messenger RNA degradation (CodonBERT [126]).”
Deep multi-modal fusion network with gated unit for breast cancer survival prediction
Yuan, H., & Xu, H. (2024) · Computer Methods in Biomechanics and Biomedical Engineering · doi:10.1080/10255842.2023.2211188 · Cited by 10 · cited 1×
“Gated fusion [145,146] uses mechanisms such as neural network gates or attention mechanisms to dynamically emphasize or de-emphasize different aspects or modalities of the data based on the context.”
Barreto, A. G., de Oliveira, J. M., Gois, F. N. B., Cortez, P. C., & de Albuquerque, V. H. C. (2023) · Bioengineering · doi:10.3390/bioengineering10091098 · Cited by 8 · cited 1×
“Synthetic data generation using generative models can create realistic, diverse data sets that mimic real-world multimodal medical scenarios, thus expanding the training data set without compromising privacy or ethical standards [149-151].”
Deciphering Stereotypes in Pre-Trained Language Models
Ma, W., Scheible, H., Wang, B., Veeramachaneni, G., Chowdhary, P., Sun, A., Koulogeorge, A., Wang, L., Yang, D., & Vosoughi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.697 · Cited by 8 · cited 1×
“If M-LLMs are trained on patient data that contain biases related to gender, ethnicity, socioeconomic status, or geographic location, they may inadvertently cause biases in their predictions or recommendations [175,178,179].”
AlSaad, R., Malluhi, Q. M., Abd-Alrazaq, A. A., & Boughorbel, S. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102802 · Cited by 8 · cited 1×
“The self-attention mechanism allows for the computation of the probability of predicting the next word in a document by attending to the most relevant parts of the input sequence [13,16].”
Self-citationProblematic Interactions Between AI and Health Privacy
Price, N. (2021) · Utah Law Review Society · doi:10.26054/0d-th4e-sgvq · Cited by 8
Praveen, S. V., & Deepika, R. (2023) · Journal of Infection · doi:10.1016/j.jinf.2023.07.011 · Cited by 7
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Moon, S., Madotto, A., Lin, Z., Nagarajan, T., Smith, M., Jain, S., Yeh, C.-F., Murugesan, P., Heidari, P., Liu, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16058 · Cited by 7 · cited 1×
“Current M-LLMs are primarily limited to multimodal comprehension on the input side, possessing limited capabilities to generate content across various modalities [134,135].”
Akrout, M., Cirone, K. D., & Vender, R. B. (2024) · Journal of Cutaneous Medicine and Surgery · doi:10.1177/12034754231220934 · Cited by 7 · cited 1×
“In dermatology, M-LLMs apply their image analysis processes to assess photos of skin lesions, aiding in the early detection of skin cancers such as melanoma [70].”
Kamath, U., Liu, J., & Whitaker, J. (2019) · Springer eBooks · doi:10.1007/978-3-030-14596-5_12 · Cited by 6
[Prospects for the Application of Healthcare Big Data Combined With Large Language Models].
Chen, R. (2023) · PubMed · doi:10.12182/20230960301 · Cited by 4
OneLLM: One Framework to Align All Modalities with LanguagePreprint
Han, J., Gong, K., Zhang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.03700 · Cited by 4 · cited 1×
“OneLLM [138] incorporates 8 unique modalities within a single framework using a multimodal alignment pipeline, which can be further expanded to include additional data modalities.”
Handbook of Bioethical Decisions. Volume I
Valdés, E., & Lecarós, J. A. (2023) · Collaborative bioethics · doi:10.1007/978-3-031-29451-8 · Cited by 3
Zhang, Q., & Liang, Y. (2023) · Global Spine Journal · doi:10.1177/21925682231222268 · Cited by 3
Li, W., Yang, J., & Min, X. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/6334435 · Cited by 2
ChatGPT-4, Medical Education, and Clinical Exposure Challenges
Kleebayoon, A., & Wiwanitkit, V. (2023) · Indian Journal of Orthopaedics · doi:10.1007/s43465-023-00997-1 · Cited by 2 · cited 1×
“Following the release of GPT-4, several medically adapted versions of this model have been developed [33-37].”
InfluentialOsprey: Pixel Understanding with Visual Instruction TuningPreprint
Yuan, Y., Li, W., Liu, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.10032 · Cited by 2 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1
SwiftQueue: Optimizing Low-Latency Applications with Swift Packet QueuingPreprint
Ray, S., Jiang, X., Luo, J., Feamster, N., & Jiang, J. (2024) · arXiv (Cornell University) · doi:10.4230/oasics.nines.2026.24 · Cited by 1
LION : Empowering Multimodal Large Language Model with Dual-Level Visual KnowledgePreprint
Chen, G., Shen, L., Shao, R., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.11860 · Cited by 1 · cited 1×
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
VTimeLLM: Empower LLM to Grasp Video MomentsPreprint
Huang, B., Wang, X., Chen, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.18445 · Cited by 1 · cited 1×
“…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
Maaz, M., Rasheed, H., Khan, S., & Khan, F. S. (2023) · arXiv · doi:10.48550/arxiv.2306.05424 · cited 1×
“The progress in M-LLMs for video data analysis, demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development…”
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · arXiv · doi:10.48550/arxiv.1707.07250 · cited 3×
“This might involve techniques such as concatenation [65] and attention mechanisms [13] or more sophisticated methods such as cross-modal attention [66,67] and tensor fusion [68].”
Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook
Jin, M., Kong, Y., Liang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.10196
Research works printed but not linked: 25
- VaswaniAShazeerNParmarNUszkoreitJJonesLGomezANKaiserLPolosukhinIAttention is all you needProceedings of the 31st International Conference on Neural Information Processing Systems2017NIPS'17December 4-9, 2017Long Beach, CA
- Devlin, J, arXiv
- Touvron, H, arXiv
- Chung, HW, arXiv
- Huang, W, arXiv
- Zhu, D, arXiv
- Su, Y, arXiv
- Koh, JY, arXiv
- Alayrac, JB, arXiv
- Rasheed, H, arXiv
- WangZZhangQYuHHuSJinXGongZChenHMulti-level protein structure pre-training with prompt learningProceedings of the Eleventh International Conference on Learning Representations2023ICLR 2023May 1-5, 2023Kigali, Rwanda
- Zhou, H, arXiv
- Yordanov, Y, arXiv
- HewittJManningCDA structural probe for finding syntax in word representationsProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies2019NAACL-HLT 2019June 2-7, 2019Minneapolis, MN
- Gerke, S, 2020, Artificial Intelligence in Healthcare, 295
- Lee, N, arXiv
- Reddy, AG, arXiv
- Carlini, N, arXiv
- Aghajanyan, A, arXiv
- Huang, Y, arXiv
- Wang, X, arXiv
- Fu, C, arXiv
- Yang, X, arXiv
- Ge, W, arXiv
- Li, S, arXiv
Cited by
Large Language Models in Healthcare and Medical Applications: A Review
Maity, S., & Saikia, M. J. (2025) · Bioengineering · doi:10.3390/bioengineering12060631 · Cited by 249
“The integration of LLMs with medical imaging represents a significant advancement in diagnostic capabilities, particularly through multimodal models that can process both visual and textual information [4,11,56–58].”
Zhou, M., Pan, Y., Zhang, Y., Song, X., & Zhou, Y. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105871 · Cited by 142
Multimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 153
“The defining feature is their ability to integrate and align information across modalities, often mapping them into a shared representational space [8].”
InfluentialAbbas, S. R., Seol, H., Abbas, Z., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13141642 · Cited by 80
“Multimodal AI Models: Research is underway to integrate imaging, EHR data, genomic profiles, and behavioral metrics into unified decision-making tools [103].”
A Current Review of Generative AI in Medicine: Core Concepts, Applications, and Current Limitations
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Shariatnia, M. M., Rouzrokh, P., & Erickson, B. J. (2025) · Current Reviews in Musculoskeletal Medicine · doi:10.1007/s12178-025-09961-y · Cited by 66
“Audio and video generation models in medicine are enabling new ways to enhance clinical training, patient communication, and medical education [ 64 ].”
Metadata
- Authors
- AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J.
- Affiliations
- Weill Cornell Medical College in Qatar (QA)
- Hamad bin Khalifa University (QA)
- United Arab Emirates University (AE)
- Corresponding author
- AlSaad, Rawan
- ORCID
- Venue
- Journal of medical Internet research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-09-25
- Volume
- 26
- Pages
- e59505
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11464944
- OpenAlex
- W4402155831
- Cited by
- 349
- References
- 175
- Citations by year
- 2026: 202 · 2025: 139 · 2024: 8
- FWCI
- 32.47
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- multimodal large language models, multimodal data integration, health care applications, medical images, wearable sensor data, electronic health records, clinical notes, audio recordings, surgical video, omics data, clinical decision support, ethical considerations, multimodal fusion
- MeSH terms
- Delivery of Health Care, Humans, Natural Language Processing, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.