LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Open accessAbstract
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing multimodal medical data sets available for public use are often relatively small in scale and demand the consolidation of numerous resources.
“Existing multimodal medical data sets available for public use [84,85,148] are often relatively small in scale and demand the consolidation of numerous resources.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
The Introduction describes PMC-15M as two orders of magnitude larger than the next-largest public biomedical image-text dataset, and Section 5.2 gives examples of smaller medical visual-question-answering datasets. These passages support the point that many existing datasets were relatively small, but the paper does not say that using them demands consolidating numerous resources. Page numbers are not available in our copy. (LLM, conf=0.83)
Experimental — accuracy not yet measuredPMC-VQA and LLaVA-Med include text and image modalities for medical visual question answering.
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2× · see citing works
“LLaVA-Med Model Variants VQA-RAD SLAKE PathVQA Average Instruct Stage 1 Stage 2 FT Open Closed Open Closed Open Closed CLIP Vision Encoder [36], 7B Language Model 0 1 0 0 15.”
MethodBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1× · see citing works
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1× · see citing works
“By contrast, LLaVA-Med uses a linear projection and a 7B LM [8, 43].”
MethodRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1× · see citing works
“Note that the image is not used to prompt GPT-4; we only show it here as a reference. original PubMed paper as additional context to the caption, inspired by the observations that external knowledge helps generalization [21, 28].”
BackgroundLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,754 · cited 1× · see citing works
“To clarify, throughout the entire experiments, we only utilize the language-only GPT-4.”
MethodDomain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 1× · see citing works
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundSelf-citationDon’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · cited 1× · see citing works
“Similar to the don’t stop pre-training argument in [12], we consider a scalable pipeline to create domain-specific instruct data from large unlabelled data, and advocate don’t stop instruction-tuning to build customized LMM. ( iii ) Low serving cost.”
BackgroundJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1× · see citing works
“Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations.”
BackgroundMIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · cited 2× · see citing works
“This dataset is two orders of magnitude larger than the next largest public dataset, MIMIC-CXR [16], and covers a diverse image types.”
BackgroundMethodBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 1× · see citing works
“Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.”
BackgroundBioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · cited 2× · see citing works
“In [44], a three-layer MLP network is used to map the visual features into a visual prefix, and the pre-trained LM are GPT2-XL [40], BioMedLM [45] and BioGPT [31], with size varying from 1.5B to 2.7B.”
BackgroundMethodSelf-citationG-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · cited 1× · see citing works
“Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations.”
BackgroundLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · cited 5× · see citing works
“Inspired by recent work in instruction-tuning [37, 27], LLaVA-Med uses GPT-4 to generate diverse biomedical multimodal instruction-following data using image-text pairs from PMC-15M, and fine-tune a large biomedical-domain VL model [27] using a novel curriculum learning method.”
BackgroundMethodInfluentialSelf-citationClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · cited 1× · see citing works
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundLi, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512 · cited 1× · see citing works
“…by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo [46].”
BackgroundCapabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · cited 1× · see citing works
“Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.”
BackgroundSlake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering
Liu, B., Zhan, L.-M., Xu, L., Ma, L., Yang, Y., & Wu, X.-M. (2021) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi48211.2021.9434010 · Cited by 351 · see citing works
Shih, G., Wu, C. C., Halabi, S. S., Kohli, M. D., Prevedello, L. M. S., Cook, T. S., Sharma, A., Amorosa, J. K., Arteaga, V. A., Galperin-Aizenberg, M., et al. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019180041 · Cited by 341 · cited 1× · see citing works
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundMaking the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317 · cited 1× · see citing works
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundSelf-citationVision-Language Pre-training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C.-Y., Wang, L.-J., Liu, Z.-C., & Gao, J.-F. (2022) · Foundations and Trends in Computer Graphics and Vision · doi:10.48550/arxiv.2210.09263 · Cited by 237 · cited 1×
“…inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].”
BackgroundSelf-citationELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual ModelsPreprint
Li, C.-Y., Liu, H., Li, L., Zhang, P.-C., Aneja, J., Yang, J.-W., Jin, P., Lee, Y. J., Hu, H., Liu, Z.-C., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2204.08790 · Cited by 215 · cited 1× · see citing works
“…inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].”
BackgroundSelf-citationPubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain?
Eslami, S., Meinel, C., & de Melo, G. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.88 · Cited by 193 · cited 2× · see citing works
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundPeng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183 · cited 1× · see citing works
“Inspired by recent work in instruction-tuning [37, 27], LLaVA-Med uses GPT-4 to generate diverse biomedical multimodal instruction-following data using image-text pairs from PMC-15M, and fine-tune a large biomedical-domain VL model [27] using a novel curriculum learning method.”
MethodSelf-citationPRD: Peer Rank and Discussion Improve Large Language Model based EvaluationsPreprint
Li, R., Patel, T., & Du, X. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2307.02762 · Cited by 150 · cited 1× · see citing works
“Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations.”
BackgroundA General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146 · cited 1× · see citing works
“…inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].”
BackgroundMedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training DataPreprint
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 117 · cited 1× · see citing works
“…by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo [46].”
BackgroundHuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., & Liu, T. (2023) · arXiv · doi:10.48550/arxiv.2304.06975 · Cited by 115 · cited 1× · see citing works
“…by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo [46].”
BackgroundSelf-Supervised Vision-Language Pretraining for Medial Visual Question Answering
Li, P., Liu, G., Tan, L., Liao, J., & Zhong, S. (2023) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi53787.2023.10230743 · Cited by 91 · cited 1× · see citing works
“While discriminative methods yield good performance, they deal with closed-set predictions [14], and require mitigation when a customized answer set is provided in at inference [24, 51, 9].”
BackgroundVision–Language Model for Visual Question Answering in Medical Imagery
Bazi, Y., Rahhal, M. M. A., Bashmal, L., & Zuair, M. (2023) · Bioengineering · doi:10.3390/bioengineering10030380 · Cited by 90 · cited 1× · see citing works
“To this end, generative methods have been developed to predict answers as a free-form text sequence [5, 30, 44].”
MethodThe AI revolution in medicine: GPT-4 and beyond
Craig, K. (2024) · Journal of IT Cases and Applications · doi:10.1080/15228053.2024.2417522 · Cited by 85 · cited 1× · see citing works
“Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [19, 30, 18], current investigations are often limited to unimodal text.”
BackgroundDoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
Xiong, H., Wang, S., Zhu, Y., Zhao, Z., Liu, Y., Huang, L., Wang, Q., & Shen, D. (2023) · arXiv · doi:10.48550/arxiv.2304.01097 · Cited by 81 · cited 1× · see citing works
“…by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo [46].”
BackgroundPathVQA: 30000+ Questions for Medical Visual Question Answering
He, X., Zhang, Y., Mou, L., Xing, E., & Xie, P. (2020) · arXiv · doi:10.48550/arxiv.2003.10286 · Cited by 79 · cited 1× · see citing works
“While discriminative methods yield good performance, they deal with closed-set predictions [14], and require mitigation when a customized answer set is provided in at inference [24, 51, 9].”
BackgroundQ2ATransformer: Improving Medical VQA via an Answer Querying DecoderPreprint
Liu, Y.-Y., Wang, Z., Xu, D., et al. (2023) · Information Processing in Medical Imaging · doi:10.48550/arxiv.2304.01611 · Cited by 65 · cited 1× · see citing works
“To this end, generative methods have been developed to predict answers as a free-form text sequence [5, 30, 44].”
MethodPMC-LLaMA: Towards Building Open-source Language Models for MedicinePreprint
Wu, C., Lin, W., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14454 · Cited by 48 · cited 1× · see citing works
“…by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo [46].”
BackgroundLearning Customized Visual Models with Retrieval-Augmented Knowledge
Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., & Li, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01454 · Cited by 41 · cited 1× · see citing works
“Note that the image is not used to prompt GPT-4; we only show it here as a reference. original PubMed paper as additional context to the caption, inspired by the observations that external knowledge helps generalization [21, 28].”
BackgroundSelf-citationMultimodal Foundation Models: From Specialists to General-Purpose AssistantsPreprint
Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., & Gao, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.10020 · Cited by 24 · cited 1× · see citing works
“…inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].”
BackgroundSelf-citationCOVID or Not COVID? A Great Mimicker Behind the Smoke Screen
Ayoub, M., Quamme, M., Abdel-Reheem, A.-R. K., & Lwin, P. (2021) · Cureus · doi:10.7759/cureus.19480 · Cited by 16 · see citing works
Conventional osteosarcoma of the mandible: Report of a rare case
Sadeghi, H. M., Karimi, A., Derakhshan, S., et al. (2021) · Clinical Case Reports · doi:10.1002/ccr3.4843 · Cited by 12 · cited 1×
“The domain example images are from [3, 33, 4, 29, 48].”
Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language ModelsPreprint
van Sonsbeek, T., Derakhshani, M. M., Najdenkoska, I., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05977 · Cited by 11 · cited 4× · see citing works
“[44] focuses efforts on exploring various modeling choices.”
BackgroundMethodInfluentialSelf-supervised vision-language pretraining for Medical visual question answeringPreprint
Li, P., Liu, G., Tan, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.13594 · Cited by 9 · see citing works
Chondromyxoid Fibroma of the Rib: A Rare Benign Tumor With Potential for Local Recurrence
Basak, B., Haragan, A., Shackcloth, M., et al. (2021) · Cureus · doi:10.7759/cureus.19172 · Cited by 6 · cited 1×
“The domain example images are from [3, 33, 4, 29, 48].”
Quadratus Femoris Partial Tear Secondary to Occult Ischiofemoral Impingement
Papavasiliou, K., Stamiris, D., Stamiris, S., et al. (2021) · Journal of Orthopaedic Case Reports · doi:10.13107/jocr.2021.v11.i09.2392 · Cited by 4 · cited 1×
“The domain example images are from [3, 33, 4, 29, 48].”
Pringle, R., & Wyatt, L. H. (2006) · Chiropractic and Osteopathy · doi:10.1186/1746-1340-14-8 · Cited by 4 · cited 1×
“[38], along with the corresponding citances (mentions of the given figure in the article).”
BackgroundDelayed Spontaneous Regression of Metastatic Gastric Cancer: A Case Report of a Rare Finding
Zafar, M., Paracha, A., Ashraf, M., et al. (2021) · Cureus · doi:10.7759/cureus.20224 · Cited by 2
Vaping-Induced Lung Injury: An Uncharted Territory
Bharat, A., Jain, N., Sheikh, B., Jeelani, H. M., & Shayuk, M. (2020) · Cureus · doi:10.7759/cureus.8970 · Cited by 1 · see citing works
Learning Customized Visual Models with Retrieval-Augmented KnowledgePreprint
Liu, H., Son, K., Yang, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.07094
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 2×
“Inspired by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo…”
MethodSelf-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2023. 2, 3, 6, 9 · arXiv
Cited by
Sorted by citations
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · see citing works
“The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
BackgroundSelf-citationA whole-slide foundation model for digital pathology from real-world data
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07441-w · Cited by 921 · see citing works
Self-citationA multimodal generative AI copilot for human pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Zhao, M., Chow, A. K., Ikemura, K., Kim, A., Pouli, D., Patel, A., et al. (2024) · Nature · doi:10.1038/s41586-024-07618-3 · Cited by 461 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
“Further studies in the medical domain are essential to elucidate the potential of LLMs in advancing health equity.”
BackgroundA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
A generalist vision-language foundation model for diverse biomedical tasks
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333 · see citing works
“Figure 4c presents the prediction-accuracy results for five models, demonstrating that all three versions of BiomedGPT outperformed BioGPT and LLaVA-Med.”
InfluentialSelf-citationUllah, E., Parwani, A., Baig, M. M., & Singh, R. (2024) · Diagnostic Pathology · doi:10.1186/s13000-024-01464-7 · Cited by 281 · see citing works
On the challenges and perspectives of foundation models for medical image analysis
Zhang, S., & Metaxas, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102996 · Cited by 288 · see citing works
Self-citationMM-LLMs: Recent Advances in MultiModal Large Language Models
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.738 · Cited by 260 · see citing works
Self-citationTowards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · see citing works
“Furthermore, the integration of multimodal inputs by LLMs is an area of novel research [22, 23], with a large potential number of data modalities to consider, and little precedent for how information from multiple modalities should be integrated over time for a single case by AI systems.”
BackgroundVision-language models for medical report generation and visual question answering: a review
Hartsock, I., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1430984 · Cited by 257 · see citing works
A guide to artificial intelligence for cancer researchers
Pérez-López, R., Laleh, N. G., Mahmood, F., & Kather, J. N. (2024) · Nature reviews. Cancer · doi:10.1038/s41568-024-00694-7 · Cited by 220 · see citing works
“Open-source LLMs, such as Llava 133 and its medical specialist model Llava-Med 134 , demonstrate promising capabilities in biomedical image analysis.”
Collaboration between clinicians and vision–language models in radiology report generation
Tanno, R., Barrett, D. G. T., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Lau, C., Tu, T., Azizi, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03302-1 · Cited by 186 · see citing works
“Although we have witnessed initial signs of such possibilities in the recent work on interactive, multimodal medical AI 26,33,38 , there remains a considerable amount of progress to be made toward building a clinically useful writing assistant for radiology.”
Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50043-3 · Cited by 189 · see citing works
“Advancements in technology have led to the development of various tools and techniques to aid dermatologists in their diagnosis 13 – 15 .”
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167 · see citing works
“Additionally, the scarcity of training data for this specific task further complicates the recognition process for LMMs.”
Self-citationWang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B., & Yin, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/22769 · Cited by 226 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
Nam, Y., Kim, J., Jung, S., Woerner, J., Suh, E. H., Lee, D. A., Shivakumar, M., Lee, M. E., & Kim, D. (2024) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-102523-103801 · Cited by 150 · see citing works
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 133 · see citing works
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01274 · Cited by 171 · see citing works
Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data
Wu, C., Zhang, X., Zhang, Y., Hui, H., Wang, Y., & Xie, W. (2025) · Nature Communications · doi:10.1038/s41467-025-62385-7 · Cited by 269 · see citing works
“Their ability to address diverse and challenging tasks has also attracted tremendous attention among researchers in the fi eldofAIforMedicine(AI4Medicine) 4 – 8 .”
CLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 175 · see citing works
Zhao, T., Gu, Y., Yang, J., Usuyama, N., Lee, H. H., Kiblawi, S., Naumann, T., Gao, J., Crabtree, A., Abel, J., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02499-w · Cited by 134 · see citing works
“Despite the proliferation of advances in the general domain, research on adapting them for large-scale biomedical image analysis across a wide range of organ or tissue classes remains relatively sparse 50 .”
Self-citationIshmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 100 · see citing works
“If these particular values are used while testing, we can say that the testing samples are di ff erent as they were not present in the training set.”
BackgroundA comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 130 · see citing works
“Additionally, emerging medical LLMs and MLLMs, such as ChatDoctor [16], ChatCAD [17], and LLaVA-Med [18], represent novel opportunities enabled by artificial intelligence in the medical field.”
InfluentialMultimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 106 · see citing works
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98 · see citing works
“Similarly, in (Xia et al., 2023a) the LLM is trained to output detailed language-based generation prompts which are employed for generation or editing tasks.”
Exploring scalable medical image encoders beyond text supervision
Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00965-w · Cited by 115 · see citing works
“Moreover, evaluating CLIP@336 and CLIP@224 within the same framework highlights the current limitations of medical multimodal learning literature [8, 9], which largely depends on static CLIP-based image encoders.”
InfluentialAn overview of large AI models and their applications
Tu, X., He, Z., Huang, Y., Zhang, Z., Yang, M., & Zhao, J. C. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00065-8 · Cited by 95 · see citing works
“In addition, there are re-lated works based on LLaVA, such as LLaVA-Med [135], which adapts LLaVA for the biomedical field, continuing the two-stage training approach of LLaVA while incorporating an additional fine-tuning phase on downstream datasets.”
Large language models and multimodal foundation models for precision oncology
Truhn, D., Eckardt, J.-N., Ferber, D., & Kather, J. N. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00573-2 · Cited by 79 · see citing works
Sniffer: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection
Qi, P., Yan, Z., Hsu, W., & Lee, M. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01240 · Cited by 98 · see citing works
“The success of LLMs in the general domain has inspired interest in applications such as biomedical [20], law [15] and education [51].”
BackgroundLVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507000 · Cited by 150 · see citing works
“This methodsimplifiesthealignmentofimageandtextfeatures withinthesamesemanticspace,anditseffectivenesshasbeen demonstratedbyLLaVA[7].”
Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation
Li, C.-Y., Chang, K.-J., Yang, C.-F., Wu, H.-Y., Chen, W., Bansal, H., Chen, L., Yang, Y.-P., Chen, Y.-C., Chen, S.-P., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57426-0 · Cited by 75 · see citing works
“15,16 However, during testing, the brain CT data is presented in a 3D format, which poses challenges for the MLLM to identify the specific slice containing the lesion of interest precisely in situ .”
Self-citationHas multimodal learning delivered universal intelligence in healthcare? A comprehensive survey
Lin, Q., Zhu, Y., Mei, X., Huang, L., Ma, J., He, K., Peng, Z., Cambria, E., & Feng, M. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102795 · Cited by 76 · see citing works
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66 · see citing works
Seyfioğlu, M. S., Ikezogwo, W. O., Ghezloo, F., Krishna, R., & Shapiro, L. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01252 · Cited by 57 · see citing works
“Meanwhile in natural images, the success of the Large Language and Vision Assistant (LLaVA) [14] demonstrates the possibility of multi-modal models as chat partners.”
BackgroundMethodInfluentialMMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.510 · Cited by 54 · see citing works
Holste, G., Zhou, Y., Wang, S., Jaiswal, A. K., Lin, M., Zhuge, S., Yang, Y., Kim, D., Nguyen-Mau, T.-H., Tran, M., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103224 · Cited by 56 · see citing works
VLAAD: Vision and Language Assistant for Autonomous Driving
Park, S., Lee, M., Kang, J., Choi, H., Park, Y., Cho, J., Lee, A., & Kim, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00107 · Cited by 49 · see citing works
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58344-x · Cited by 45 · see citing works
“We quantify these totals in reports from publicly accessible models, enabling us to compare LLaVA-Rad with LLaVA-Med, LLaVA, CheXagent, and GPT-4V.”
InfluentialSelf-citationA Systematic Review of ChatGPT and Other Conversational Large Language Models in HealthcarePreprint
Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B., & Yin, Z. (2024) · medRxiv · doi:10.1101/2024.04.26.24306390 · Cited by 41 · see citing works
“…the multi-modal LLMs that can process medical images, such as GPT-4, Large Language and Vision Assistant (LLaVA) [85] based on LLaMA, and LLaVA-Med [73] based on LLaVA, were just released before September 2023 and are still being examined by researchers regarding their capabilities in healthcare…”
BackgroundMMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning
Peng, L., Cai, S., Wu, Z., Shang, H., Zhu, X., & Li, X. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103225 · Cited by 41 · see citing works
Holistic Autonomous Driving Understanding by Bird'View Injected Multi-Modal Large Models
Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., & Li, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01297 · Cited by 44 · see citing works
“Leveraging the capabilities of pre-trained LLMs like LLaMA [40] and Vi-cuna [4], Multimodal LLMs (MLLMs) are expanding their application spectrum, handling inputs from images [1, 3, 18, 19, 46], videos [20, 45], and 3D data [13] to medical data [17].”
Effectiveness assessment of recent large vision-language models
Jiang, Y., Yan, X., Ji, G.-P., Fu, K., Sun, M., Xiong, H., Fan, D.-P., & Khan, F. S. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00050-1 · Cited by 40 · see citing works
Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance
Gao, Z., Chen, Z., Cui, E., Ren, Y., Wang, W., Zhu, J., Tian, H., He, J., He, J., Zhu, X., et al. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00067-6 · Cited by 50 · see citing works
“Section 3.1 provides a comprehensive overview of Mini-InternVL.”
InfluentialClinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37 · see citing works
“The LLaVA-Med model [90] can answer open-ended questions about biomedical images, while the CheXagent model [91], designed specifically for chest X-ray interpretation, exemplifies the successful transition from general medical models to specialist models capable of tackling problems in narrow…”
MedCLIP-SAMv2: Towards universal text-driven medical image segmentation
Koleilat, T., Asgariandehkordi, H., Rivaz, H. J., & Xiao, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103749 · Cited by 47 · see citing works
Vision-language foundation model for 3D medical imaging
Wu, J., Wang, Y., Zhong, Z., Liao, W., Trayanova, N., Jiao, Z., & Bai, H. X. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00015-9 · Cited by 41 · see citing works
“Tosigni fi cantlyimprovetheperformanceofVLFMsin generating accurate radiology reports from 3D medical images, it is crucial to focus on both the development and re fi nement of model architectures tailored to the inherent complexities of 3D medical scans 9 .”
FaultExplainer: Leveraging large language models for interpretable fault detection and diagnosis
Khan, A., Nahar, R., Chen, H., Constante‐Flores, G. E., & Li, C. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109152 · Cited by 42 · see citing works
Self-citationA foundational architecture for AI agents in healthcare
Liu, F., Niu, Y., Zhang, Q., Wang, K., Dong, Z., Wong, I. N., Cheng, L., Li, T., Duan, L., Li, K., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102374 · Cited by 41 · see citing works
“44 Medical AI agents are uniquely positioned to address these multifaceted challenges through their advanced capabilities in data processing, autonomous decision-making, and adaptive learning.”
Large models in medical imaging: Advances and prospects
Fang, M., Wang, Z., Pan, S., Feng, X., Zhao, Y., Hou, D., Wu, L., Xie, X., Zhang, X.-Y., Tian, J., et al. (2025) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003699 · Cited by 39 · see citing works
“For instance, LLaVA-Med [20] uses GPT-4 to assess the accuracy, relevance, and detail richness of medical dialogues.”
InfluentialGuo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 50 · see citing works
Self-citationLiu, M., Hu, W., Ding, J., Xu, J., Li, X., Zhu, L., Bai, Z., Shi, X., Wang, B., Song, H., et al. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020044 · Cited by 41 · see citing works
Self-citationYi, Z., Xiao, T., & Albert, M. V. (2025) · Information · doi:10.3390/info16020136 · Cited by 36 · see citing works
Generative Models in Medical Visual Question Answering: A Survey
Dong, W., Shen, S., Han, Y., Tao, T., Wu, J., & Xu, H. (2025) · Applied Sciences · doi:10.3390/app15062983 · Cited by 35 · see citing works
A survey of deep-learning-based radiology report generation using multimodal inputs
Wang, X., Figueredo, G. P., Li, R., Zhang, W. E., Chen, W., & Chen, X. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103627 · Cited by 50 · see citing works
Danish, S., Sadeghi‐Niaraki, A., Khan, S. U., Dang, L. M., Tightiz, L., & Moon, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103623 · Cited by 38 · see citing works
Towards Generalist Foundation Model for RadiologyPreprint
Xie, W., Wu, C., Zhang, X., Zhang, Y., & Wang, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3324530/v1 · Cited by 33 · see citing works
PneumoLLM: Harnessing the power of large language model for pneumoconiosis diagnosis
Song, M., Wang, J., Yu, Z., Wang, J., Yang, L., Lu, Y., Li, B., Wang, X., Wang, X., Huang, Q., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103248 · Cited by 31 · see citing works
Self-citationOphGLM: An ophthalmology large language-and-vision assistant
Deng, Z., Gao, W., Chen, C. H., Niu, Z., Gong, Z., Zhang, R., Cao, Z., Li, F., Ma, Z., Wei, W., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103001 · Cited by 30 · see citing works
Multi-modal large language models in radiology: principles, applications, and potential
Shen, Y., Xu, Y., Ma, J., Rui, W., Zhao, C., Heacock, L. C., & Huang, C. (2024) · Abdominal Radiology · doi:10.1007/s00261-024-04708-8 · Cited by 31 · see citing works
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 28 · see citing works
“…models, there has been increased interest in their application in healthcare (Clusmann et al., 2023; Thirunavukarasu et al., 2023; Lee et al., 2023; Liu et al., 2023; Moor et al., 2023; Singhal et al., 2023; Omiye et al., 2024; Zakka et al., 2024; Nori et al., 2023; Cosentino et al., 2024; Shah et…”
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28 · see citing works
Lameesa, A., Silpasuwanchai, C., & Alam, M. S. B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128730 · Cited by 28 · see citing works
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483 · see citing works
K-PathVQA: Knowledge-Aware Multimodal Representation for Pathology Visual Question Answering
Naseem, U., Khushi, M., Dunn, A. G., & Kim, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3294249 · Cited by 27 · see citing works
“Apart from reliance on non-domain specific language models, these existing PathVQA approaches [3], [4], [5], [6], [7], [8], [10], [11], [12] also relied exclusively on the information within the image, and this caused wide failures when the visual content of a medical image was insufficient to answer…”
Deep learning applications for human embryo assessment using time-lapse imaging: scoping review
AlSaad, R., Abusarhan, L., Odeh, N., et al. (2025) · Frontiers in Reproductive Health · doi:10.3389/frph.2025.1549642 · Cited by 25 · see citing works
A systematic evaluation of GPT-4V's multimodal capability for chest X-ray image analysis
Liu, Y., Li, Y., Wang, Z., Liang, X., Liu, L., Wang, L., Cui, L., Tu, Z., Wang, L., & Zhou, L. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100099 · Cited by 25 · see citing works
Moglia, A., Georgiou, K. E., Cerveri, P., Mainardi, L. T., Satava, R. M., & Cuschieri, A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10849-5 · Cited by 24 · see citing works
Biomedical Natural Language Processing in the Era of Large Language Models
Usuyama, N., Wong, C., Zhang, S., Naumann, T., & Poon, H. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095406 · Cited by 21 · see citing works
Self-citationA Comprehensive Review of Multimodal Large Language Models for Medical Imaging and Omics Data
Vavekanand, R. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10504-y · Cited by 29 · see citing works
Alzubaidi, L., Fadhel, M. A., Hollman, F., Salhi, A., Santamaría, J., Duan, Y., Gupta, A., Cutbush, K., Abbosh, A. M., & Gu, Y. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10878-0 · Cited by 23 · see citing works
“Using the same approach, several published articles have shown the effectiveness of these solutions for medical images and other domains (Silva et al. 2021; Pham et al. 2020; Fan and Bu 2022; Al-kubaisi and Khamiss 2021; Cheng et al. 2021; Li et al. 2024).”
AI-driven multimodal colorimetric analytics for biomedical and behavioral health diagnostics
Hagos, D. H., Aryal, S. K., Ymele‐Leki, P., & Burge, L. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.05.015 · Cited by 23 · see citing works
Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework
Chang, Y., Yin, J., Li, J., Liu, C., Cao, L., & Lin, S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02132-5 · Cited by 19 · see citing works
“LLaVA-Med [162] constructs a multimodal conversation generation LLM for the medical field.”
Application of deep learning-based multimodal fusion technology in cancer diagnosis: A survey
Li, Y., Pan, L., Peng, Y., Li, X., Wang, X., Qu, L., Song, Q., Liang, Q., & Peng, S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109972 · Cited by 21 · see citing works
Task-Oriented Semantic Communication in Large Multimodal Models-Based Vehicle Networks
Du, B., Du, H., Niyato, D. T., & Li, R. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3564543 · Cited by 24 · see citing works
Opportunities and challenges in the application of large artificial intelligence models in radiology
Pan, L., Zhao, Z., Lu, Y., Tang, K., Fu, L., Liang, Q., & Peng, S. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100080 · Cited by 22 · see citing works
A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot
Du, F., Ma, X.-J., Yang, J., Liu, Y., Luo, C.-R., Wang, X., Jiang, H.-O., & Jing, X. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-3767-3 · Cited by 22 · see citing works
Self-citationPre-trained Multimodal Large Language Model Enhances Dermatological Diagnosis using SkinGPT-4Preprint
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., & Gao, X. (2023) · medRxiv · doi:10.1101/2023.06.10.23291127 · Cited by 20 · see citing works
Advances in artificial intelligence for meibomian gland evaluation: A comprehensive review
Li, L., Xiao, K., Shang, X., et al. (2024) · Survey of Ophthalmology · doi:10.1016/j.survophthal.2024.07.005 · Cited by 19 · see citing works
Zhang, K., Li, P., & Wang, J. (2024) · Remote Sensing · doi:10.3390/rs16214113 · Cited by 23 · see citing works
“The meta caption [143] mentioned above can also achieve the effect of few-shot learning, as the model acquired meta-features from natural image classification and remote sensing image classification, so that it requires minimal data to train the RSIC model for generating accurate captions.”
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 34 · see citing works
Ryan, L., & Agaian, S. С. (2025) · Bioengineering · doi:10.3390/bioengineering12060639 · Cited by 26 · see citing works
Xu, X., & Sankar, R. (2025) · Information · doi:10.3390/info16100894 · Cited by 20 · see citing works
“Representative systems including LLaVA-Med [52] and GeneVerse [53] enable agents to interpret imaging and genetic data through joint embedding and instruction-following capabilities.”
PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology
Sun, Y., Wu, H., Zhu, C., Zheng, S., Chen, Q., Zhang, K., Zhang, Y., Wan, D., Lan, X., Zheng, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73033-7_4 · Cited by 24 · see citing works
Integrating language into medical visual recognition and reasoning: A survey
Lu, Y., & Wang, A. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103514 · Cited by 26 · see citing works
Wu, Y., Lu, Y., Zhou, Y., Ding, Y., Liu, J., & Ruan, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129999 · Cited by 18 · see citing works
Liu, F., Zhou, H., Wang, K., Yu, Y., Gao, Y., Sun, Z., Liu, S., Sun, S., Zou, Z., Li, Z., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102056 · Cited by 22 · see citing works
Large Language Models in Medical Image Analysis: A Systematic Survey and Future Directions
Urooj, B., Fayaz, M., Ali, S., Dang, L. M., & Kim, K. W. (2025) · Bioengineering · doi:10.3390/bioengineering12080818 · Cited by 20 · see citing works
Multimodal integration strategies for clinical application in oncology
Zhang, B., Wan, Z., Luo, Y., Zhao, X., Samayoa, J. A., Zhao, W., & Wu, S. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1609079 · Cited by 23 · see citing works
EndoChat: Grounded multimodal large language model for endoscopic surgery
Wang, G., Bai, L., Wang, J., Yuan, K., Li, Z., Jiang, T., He, X., Wu, J., Chen, Z., Lei, Z., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103789 · Cited by 31 · see citing works
Self-citationGuo, Y., & Wan, Z. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00080 · Cited by 19 · see citing works
DALL-M: Context-aware clinical data augmentation with large language models
Hsieh, C., Moreira, C., Nobre, I. B., Sousa, S. C., Ouyang, C., Brereton, M. F., Jorge, J. A., & Nascimento, J. C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110022 · Cited by 22 · see citing works
Trust it or not: Confidence-guided automatic radiology report generation
Wang, Y., Lin, Z., Xu, Z., Dong, H., Luo, J., Tian, J., Shi, Z., Huang, L., Zhang, Y., Jianping Fan, B., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127374 · Cited by 17 · see citing works
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., & Zhu, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02508 · Cited by 17 · see citing works
“Though effective in various fields, such as science [37] and biomedicine [33], full-parameter fine-tuning (FFT) is computationally demanding and storage-intensive, particularly for models with billions of parameters.”
BackgroundSelf-citationJack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
Pramanick, S., Han, G., Hou, R., Nag, S., Lim, S.-N., Ballas, N., Wang, Q., Chellappa, R., & Almahairi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01335 · Cited by 18 · see citing works
“To name a few, LLaVA [57], MiniGPT-4 [126], MM-REACT [113], BLIP2 [46], mPLUS-OWL [114], LLaMA-Adapter v2 [22], Otter [42], Instruct-BLIP [15], LLaVA-Med [43] have been proven to be effective.”
MammoVLM: A generative large vision–language model for mammography-related diagnostic assistance
Cao, Z., Deng, Z., Ma, J., Hu, J., & Ma, L. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102998 · Cited by 20 · see citing works
Generating Medical Reports With a Novel Deep Learning Architecture
Uçan, M., Kaya, B., & Kaya, M. (2025) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.70062 · Cited by 17 · see citing works
UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling
Yu, K., Zhou, Y., Bai, Y., Soh, Z. D., Xu, X., Goh, R. S. M., Cheng, C.-Y., & Liu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_70 · Cited by 17 · see citing works
- Authors
- Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 28541-28564
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2306.00890
- OpenAlex
- W7133197087
- Cited by
- 590
- References
- 45
- Citations by year
- 2026: 288 · 2025: 179 · 2024: 115 · 2023: 8
- FWCI
- 63.10
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- multimodal language models, medical question answering, biomedicine, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
