The DOI matches its DataCite record; 47 of the 47 references it cites were found in a registry or an index.
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Open accessAbstract
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing multimodal medical data sets available for public use [84,85,148] are often relatively small in scale and demand the consolidation of numerous resources.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“LLaVA-Med Model Variants VQA-RAD SLAKE PathVQA Average Instruct Stage 1 Stage 2 FT Open Closed Open Closed Open Closed CLIP Vision Encoder [36], 7B Language Model 0 1 0 0 15.”
MethodBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“By contrast, LLaVA-Med uses a linear projection and a 7B LM [8, 43].”
MethodRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“Note that the image is not used to prompt GPT-4; we only show it here as a reference. original PubMed paper as additional context to the caption, inspired by the observations that external knowledge helps generalization [21, 28].”
BackgroundLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 1×
“To clarify, throughout the entire experiments, we only utilize the language-only GPT-4.”
MethodDomain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 1×
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundSelf-citationDon’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · cited 1×
“Similar to the don’t stop pre-training argument in [12], we consider a scalable pipeline to create domain-specific instruct data from large unlabelled data, and advocate don’t stop instruction-tuning to build customized LMM. ( iii ) Low serving cost.”
BackgroundJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1×
“Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations.”
BackgroundMIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · cited 2×
“This dataset is two orders of magnitude larger than the next largest public dataset, MIMIC-CXR [16], and covers a diverse image types.”
BackgroundMethodBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 1×
“Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.”
Background
Research works printed but not linked: 1
- Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2023. 2, 3, 6, 9 · arXiv
Cited by
Sorted by citations
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748
“The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
BackgroundSelf-citationA whole-slide foundation model for digital pathology from real-world data
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07441-w · Cited by 921
Self-citationA multimodal generative AI copilot for human pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Zhao, M., Chow, A. K., Ikemura, K., Kim, A., Pouli, D., Patel, A., et al. (2024) · Nature · doi:10.1038/s41586-024-07618-3 · Cited by 461
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 430
“Further studies in the medical domain are essential to elucidate the potential of LLMs in advancing health equity.”
BackgroundA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
A generalist vision-language foundation model for diverse biomedical tasks
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333
“Figure 4c presents the prediction-accuracy results for five models, demonstrating that all three versions of BiomedGPT outperformed BioGPT and LLaVA-Med.”
InfluentialSelf-citationUllah, E., Parwani, A., Baig, M. M., & Singh, R. (2024) · Diagnostic Pathology · doi:10.1186/s13000-024-01464-7 · Cited by 281
On the challenges and perspectives of foundation models for medical image analysis
Zhang, S., & Metaxas, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102996 · Cited by 340
Self-citationMM-LLMs: Recent Advances in MultiModal Large Language Models
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.738 · Cited by 260
Self-citationTowards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311
“Furthermore, the integration of multimodal inputs by LLMs is an area of novel research [22, 23], with a large potential number of data modalities to consider, and little precedent for how information from multiple modalities should be integrated over time for a single case by AI systems.”
BackgroundVision-language models for medical report generation and visual question answering: a review
Hartsock, I., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1430984 · Cited by 298
A guide to artificial intelligence for cancer researchers
Pérez-López, R., Laleh, N. G., Mahmood, F., & Kather, J. N. (2024) · Nature reviews. Cancer · doi:10.1038/s41568-024-00694-7 · Cited by 227
“Open-source LLMs, such as Llava 133 and its medical specialist model Llava-Med 134 , demonstrate promising capabilities in biomedical image analysis.”
Collaboration between clinicians and vision–language models in radiology report generation
Tanno, R., Barrett, D. G. T., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Lau, C., Tu, T., Azizi, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03302-1 · Cited by 186
“Although we have witnessed initial signs of such possibilities in the recent work on interactive, multimodal medical AI 26,33,38 , there remains a considerable amount of progress to be made toward building a clinically useful writing assistant for radiology.”
Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50043-3 · Cited by 189
“Advancements in technology have led to the development of various tools and techniques to aid dermatologists in their diagnosis 13 – 15 .”
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167
“Additionally, the scarcity of training data for this specific task further complicates the recognition process for LMMs.”
Self-citationWang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B., & Yin, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/22769 · Cited by 239
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Nam, Y., Kim, J., Jung, S., Woerner, J., Suh, E. H., Lee, D. A., Shivakumar, M., Lee, M. E., & Kim, D. (2024) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-102523-103801 · Cited by 156
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 376
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01274 · Cited by 270
Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data
Wu, C., Zhang, X., Zhang, Y., Hui, H., Wang, Y., & Xie, W. (2025) · Nature Communications · doi:10.1038/s41467-025-62385-7 · Cited by 429
“Their ability to address diverse and challenging tasks has also attracted tremendous attention among researchers in the fi eldofAIforMedicine(AI4Medicine) 4 – 8 .”
CLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 248
Zhao, T., Gu, Y., Yang, J., Usuyama, N., Lee, H. H., Kiblawi, S., Naumann, T., Gao, J., Crabtree, A., Abel, J., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02499-w · Cited by 209
“Despite the proliferation of advances in the general domain, research on adapting them for large-scale biomedical image analysis across a wide range of organ or tissue classes remains relatively sparse 50 .”
Self-citationIshmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 107
“If these particular values are used while testing, we can say that the testing samples are di ff erent as they were not present in the training set.”
BackgroundA comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 154
“Additionally, emerging medical LLMs and MLLMs, such as ChatDoctor [16], ChatCAD [17], and LLaVA-Med [18], represent novel opportunities enabled by artificial intelligence in the medical field.”
InfluentialMultimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 106
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“Similarly, in (Xia et al., 2023a) the LLM is trained to output detailed language-based generation prompts which are employed for generation or editing tasks.”
Exploring scalable medical image encoders beyond text supervision
Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00965-w · Cited by 115
“Moreover, evaluating CLIP@336 and CLIP@224 within the same framework highlights the current limitations of medical multimodal learning literature [8, 9], which largely depends on static CLIP-based image encoders.”
InfluentialAn overview of large AI models and their applications
Tu, X., He, Z., Huang, Y., Zhang, Z., Yang, M., & Zhao, J. C. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00065-8 · Cited by 106
“In addition, there are re-lated works based on LLaVA, such as LLaVA-Med [135], which adapts LLaVA for the biomedical field, continuing the two-stage training approach of LLaVA while incorporating an additional fine-tuning phase on downstream datasets.”
Large language models and multimodal foundation models for precision oncology
Truhn, D., Eckardt, J.-N., Ferber, D., & Kather, J. N. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00573-2 · Cited by 79
Sniffer: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection
Qi, P., Yan, Z., Hsu, W., & Lee, M. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01240 · Cited by 120
“The success of LLMs in the general domain has inspired interest in applications such as biomedical [20], law [15] and education [51].”
BackgroundLVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507000 · Cited by 273
“This methodsimplifiesthealignmentofimageandtextfeatures withinthesamesemanticspace,anditseffectivenesshasbeen demonstratedbyLLaVA[7].”
Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation
Li, C.-Y., Chang, K.-J., Yang, C.-F., Wu, H.-Y., Chen, W., Bansal, H., Chen, L., Yang, Y.-P., Chen, Y.-C., Chen, S.-P., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57426-0 · Cited by 75
“15,16 However, during testing, the brain CT data is presented in a 3D format, which poses challenges for the MLLM to identify the specific slice containing the lesion of interest precisely in situ .”
Self-citationHas multimodal learning delivered universal intelligence in healthcare? A comprehensive survey
Lin, Q., Zhu, Y., Mei, X., Huang, L., Ma, J., He, K., Peng, Z., Cambria, E., & Feng, M. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102795 · Cited by 86
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66
Seyfioğlu, M. S., Ikezogwo, W. O., Ghezloo, F., Krishna, R., & Shapiro, L. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01252 · Cited by 81
“Meanwhile in natural images, the success of the Large Language and Vision Assistant (LLaVA) [14] demonstrates the possibility of multi-modal models as chat partners.”
BackgroundMethodInfluentialMMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.510 · Cited by 54
Holste, G., Zhou, Y., Wang, S., Jaiswal, A. K., Lin, M., Zhuge, S., Yang, Y., Kim, D., Nguyen-Mau, T.-H., Tran, M., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103224 · Cited by 65
VLAAD: Vision and Language Assistant for Autonomous Driving
Park, S., Lee, M., Kang, J., Choi, H., Park, Y., Cho, J., Lee, A., & Kim, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00107 · Cited by 49
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58344-x · Cited by 45
“We quantify these totals in reports from publicly accessible models, enabling us to compare LLaVA-Rad with LLaVA-Med, LLaVA, CheXagent, and GPT-4V.”
InfluentialSelf-citationA Systematic Review of ChatGPT and Other Conversational Large Language Models in HealthcarePreprint
Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B., & Yin, Z. (2024) · medRxiv · doi:10.1101/2024.04.26.24306390 · Cited by 41
“…the multi-modal LLMs that can process medical images, such as GPT-4, Large Language and Vision Assistant (LLaVA) [85] based on LLaMA, and LLaVA-Med [73] based on LLaVA, were just released before September 2023 and are still being examined by researchers regarding their capabilities in healthcare…”
BackgroundMMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning
Peng, L., Cai, S., Wu, Z., Shang, H., Zhu, X., & Li, X. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103225 · Cited by 41
Holistic Autonomous Driving Understanding by Bird'View Injected Multi-Modal Large Models
Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., & Li, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01297 · Cited by 66
“Leveraging the capabilities of pre-trained LLMs like LLaMA [40] and Vi-cuna [4], Multimodal LLMs (MLLMs) are expanding their application spectrum, handling inputs from images [1, 3, 18, 19, 46], videos [20, 45], and 3D data [13] to medical data [17].”
Effectiveness assessment of recent large vision-language models
Jiang, Y., Yan, X., Ji, G.-P., Fu, K., Sun, M., Xiong, H., Fan, D.-P., & Khan, F. S. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00050-1 · Cited by 54
Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance
Gao, Z., Chen, Z., Cui, E., Ren, Y., Wang, W., Zhu, J., Tian, H., He, J., He, J., Zhu, X., et al. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00067-6 · Cited by 84
“Section 3.1 provides a comprehensive overview of Mini-InternVL.”
InfluentialClinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
“The LLaVA-Med model [90] can answer open-ended questions about biomedical images, while the CheXagent model [91], designed specifically for chest X-ray interpretation, exemplifies the successful transition from general medical models to specialist models capable of tackling problems in narrow…”
MedCLIP-SAMv2: Towards universal text-driven medical image segmentation
Koleilat, T., Asgariandehkordi, H., Rivaz, H. J., & Xiao, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103749 · Cited by 47
Vision-language foundation model for 3D medical imaging
Wu, J., Wang, Y., Zhong, Z., Liao, W., Trayanova, N., Jiao, Z., & Bai, H. X. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00015-9 · Cited by 41
“Tosigni fi cantlyimprovetheperformanceofVLFMsin generating accurate radiology reports from 3D medical images, it is crucial to focus on both the development and re fi nement of model architectures tailored to the inherent complexities of 3D medical scans 9 .”
FaultExplainer: Leveraging large language models for interpretable fault detection and diagnosis
Khan, A., Nahar, R., Chen, H., Constante‐Flores, G. E., & Li, C. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109152 · Cited by 49
Self-citationA foundational architecture for AI agents in healthcare
Liu, F., Niu, Y., Zhang, Q., Wang, K., Dong, Z., Wong, I. N., Cheng, L., Li, T., Duan, L., Li, K., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102374 · Cited by 41
“44 Medical AI agents are uniquely positioned to address these multifaceted challenges through their advanced capabilities in data processing, autonomous decision-making, and adaptive learning.”
Large models in medical imaging: Advances and prospects
Fang, M., Wang, Z., Pan, S., Feng, X., Zhao, Y., Hou, D., Wu, L., Xie, X., Zhang, X.-Y., Tian, J., et al. (2025) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003699 · Cited by 43
“For instance, LLaVA-Med [20] uses GPT-4 to assess the accuracy, relevance, and detail richness of medical dialogues.”
InfluentialGuo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
Self-citationLiu, M., Hu, W., Ding, J., Xu, J., Li, X., Zhu, L., Bai, Z., Shi, X., Wang, B., Song, H., et al. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020044 · Cited by 41
Self-citationYi, Z., Xiao, T., & Albert, M. V. (2025) · Information · doi:10.3390/info16020136 · Cited by 43
Generative Models in Medical Visual Question Answering: A Survey
Dong, W., Shen, S., Han, Y., Tao, T., Wu, J., & Xu, H. (2025) · Applied Sciences · doi:10.3390/app15062983 · Cited by 35
A survey of deep-learning-based radiology report generation using multimodal inputs
Wang, X., Figueredo, G. P., Li, R., Zhang, W. E., Chen, W., & Chen, X. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103627 · Cited by 65
Danish, S., Sadeghi‐Niaraki, A., Khan, S. U., Dang, L. M., Tightiz, L., & Moon, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103623 · Cited by 45
Towards Generalist Foundation Model for RadiologyPreprint
Xie, W., Wu, C., Zhang, X., Zhang, Y., & Wang, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3324530/v1 · Cited by 33
PneumoLLM: Harnessing the power of large language model for pneumoconiosis diagnosis
Song, M., Wang, J., Yu, Z., Wang, J., Yang, L., Lu, Y., Li, B., Wang, X., Wang, X., Huang, Q., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103248 · Cited by 31
Self-citationOphGLM: An ophthalmology large language-and-vision assistant
Deng, Z., Gao, W., Chen, C. H., Niu, Z., Gong, Z., Zhang, R., Cao, Z., Li, F., Ma, Z., Wei, W., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103001 · Cited by 35
Multi-modal large language models in radiology: principles, applications, and potential
Shen, Y., Xu, Y., Ma, J., Rui, W., Zhao, C., Heacock, L. C., & Huang, C. (2024) · Abdominal Radiology · doi:10.1007/s00261-024-04708-8 · Cited by 34
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 654
“…models, there has been increased interest in their application in healthcare (Clusmann et al., 2023; Thirunavukarasu et al., 2023; Lee et al., 2023; Liu et al., 2023; Moor et al., 2023; Singhal et al., 2023; Omiye et al., 2024; Zakka et al., 2024; Nori et al., 2023; Cosentino et al., 2024; Shah et…”
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28
Lameesa, A., Silpasuwanchai, C., & Alam, M. S. B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128730 · Cited by 28
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483
K-PathVQA: Knowledge-Aware Multimodal Representation for Pathology Visual Question Answering
Naseem, U., Khushi, M., Dunn, A. G., & Kim, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3294249 · Cited by 30
“Apart from reliance on non-domain specific language models, these existing PathVQA approaches [3], [4], [5], [6], [7], [8], [10], [11], [12] also relied exclusively on the information within the image, and this caused wide failures when the visual content of a medical image was insufficient to answer…”
Deep learning applications for human embryo assessment using time-lapse imaging: scoping review
AlSaad, R., Abusarhan, L., Odeh, N., et al. (2025) · Frontiers in Reproductive Health · doi:10.3389/frph.2025.1549642 · Cited by 25
A systematic evaluation of GPT-4V's multimodal capability for chest X-ray image analysis
Liu, Y., Li, Y., Wang, Z., Liang, X., Liu, L., Wang, L., Cui, L., Tu, Z., Wang, L., & Zhou, L. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100099 · Cited by 25
Moglia, A., Georgiou, K. E., Cerveri, P., Mainardi, L. T., Satava, R. M., & Cuschieri, A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10849-5 · Cited by 27
Biomedical Natural Language Processing in the Era of Large Language Models
Usuyama, N., Wong, C., Zhang, S., Naumann, T., & Poon, H. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095406 · Cited by 22
Self-citationA Comprehensive Review of Multimodal Large Language Models for Medical Imaging and Omics Data
Vavekanand, R. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10504-y · Cited by 29
Alzubaidi, L., Fadhel, M. A., Hollman, F., Salhi, A., Santamaría, J., Duan, Y., Gupta, A., Cutbush, K., Abbosh, A. M., & Gu, Y. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10878-0 · Cited by 23
“Using the same approach, several published articles have shown the effectiveness of these solutions for medical images and other domains (Silva et al. 2021; Pham et al. 2020; Fan and Bu 2022; Al-kubaisi and Khamiss 2021; Cheng et al. 2021; Li et al. 2024).”
AI-driven multimodal colorimetric analytics for biomedical and behavioral health diagnostics
Hagos, D. H., Aryal, S. K., Ymele‐Leki, P., & Burge, L. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.05.015 · Cited by 23
Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework
Chang, Y., Yin, J., Li, J., Liu, C., Cao, L., & Lin, S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02132-5 · Cited by 20
“LLaVA-Med [162] constructs a multimodal conversation generation LLM for the medical field.”
Application of deep learning-based multimodal fusion technology in cancer diagnosis: A survey
Li, Y., Pan, L., Peng, Y., Li, X., Wang, X., Qu, L., Song, Q., Liang, Q., & Peng, S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109972 · Cited by 25
Task-Oriented Semantic Communication in Large Multimodal Models-Based Vehicle Networks
Du, B., Du, H., Niyato, D. T., & Li, R. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3564543 · Cited by 28
Opportunities and challenges in the application of large artificial intelligence models in radiology
Pan, L., Zhao, Z., Lu, Y., Tang, K., Fu, L., Liang, Q., & Peng, S. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100080 · Cited by 22
A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot
Du, F., Ma, X.-J., Yang, J., Liu, Y., Luo, C.-R., Wang, X., Jiang, H.-O., & Jing, X. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-3767-3 · Cited by 25
Self-citationPre-trained Multimodal Large Language Model Enhances Dermatological Diagnosis using SkinGPT-4Preprint
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., & Gao, X. (2023) · medRxiv · doi:10.1101/2023.06.10.23291127 · Cited by 20
Advances in artificial intelligence for meibomian gland evaluation: A comprehensive review
Li, L., Xiao, K., Shang, X., Hu, W., Yusufu, M., Chen, R., Wang, Y., Liu, J., Lai, T., Guo, L., et al. (2024) · Survey of Ophthalmology · doi:10.1016/j.survophthal.2024.07.005 · Cited by 21
Zhang, K., Li, P., & Wang, J. (2024) · Remote Sensing · doi:10.3390/rs16214113 · Cited by 30
“The meta caption [143] mentioned above can also achieve the effect of few-shot learning, as the model acquired meta-features from natural image classification and remote sensing image classification, so that it requires minimal data to train the RSIC model for generating accurate captions.”
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 40
Ryan, L., & Agaian, S. С. (2025) · Bioengineering · doi:10.3390/bioengineering12060639 · Cited by 26
Xu, X., & Sankar, R. (2025) · Information · doi:10.3390/info16100894 · Cited by 20
“Representative systems including LLaVA-Med [52] and GeneVerse [53] enable agents to interpret imaging and genetic data through joint embedding and instruction-following capabilities.”
PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology
Sun, Y., Wu, H., Zhu, C., Zheng, S., Chen, Q., Zhang, K., Zhang, Y., Wan, D., Lan, X., Zheng, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73033-7_4 · Cited by 36
Integrating language into medical visual recognition and reasoning: A survey
Lu, Y., & Wang, A. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103514 · Cited by 32
Wu, Y., Lu, Y., Zhou, Y., Ding, Y., Liu, J., & Ruan, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129999 · Cited by 19
Liu, F., Zhou, H., Wang, K., Yu, Y., Gao, Y., Sun, Z., Liu, S., Sun, S., Zou, Z., Li, Z., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102056 · Cited by 22
Large Language Models in Medical Image Analysis: A Systematic Survey and Future Directions
Urooj, B., Fayaz, M., Ali, S., Dang, L. M., & Kim, K. W. (2025) · Bioengineering · doi:10.3390/bioengineering12080818 · Cited by 26
Multimodal integration strategies for clinical application in oncology
Zhang, B., Wan, Z., Luo, Y., Zhao, X., Samayoa, J. A., Zhao, W., & Wu, S. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1609079 · Cited by 28
EndoChat: Grounded multimodal large language model for endoscopic surgery
Wang, G., Bai, L., Wang, J., Yuan, K., Li, Z., Jiang, T., He, X., Wu, J., Chen, Z., Lei, Z., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103789 · Cited by 54
Self-citationGuo, Y., & Wan, Z. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00080 · Cited by 19
DALL-M: Context-aware clinical data augmentation with large language models
Hsieh, C., Moreira, C., Nobre, I. B., Sousa, S. C., Ouyang, C., Brereton, M. F., Jorge, J. A., & Nascimento, J. C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110022 · Cited by 28
Trust it or not: Confidence-guided automatic radiology report generation
Wang, Y., Lin, Z., Xu, Z., Dong, H., Luo, J., Tian, J., Shi, Z., Huang, L., Zhang, Y., Jianping Fan, B., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127374 · Cited by 20
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., & Zhu, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02508 · Cited by 17
“Though effective in various fields, such as science [37] and biomedicine [33], full-parameter fine-tuning (FFT) is computationally demanding and storage-intensive, particularly for models with billions of parameters.”
BackgroundSelf-citationJack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
Pramanick, S., Han, G., Hou, R., Nag, S., Lim, S.-N., Ballas, N., Wang, Q., Chellappa, R., & Almahairi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01335 · Cited by 35
“To name a few, LLaVA [57], MiniGPT-4 [126], MM-REACT [113], BLIP2 [46], mPLUS-OWL [114], LLaMA-Adapter v2 [22], Otter [42], Instruct-BLIP [15], LLaVA-Med [43] have been proven to be effective.”
MammoVLM: A generative large vision–language model for mammography-related diagnostic assistance
Cao, Z., Deng, Z., Ma, J., Hu, J., & Ma, L. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102998 · Cited by 21
Generating Medical Reports With a Novel Deep Learning Architecture
Uçan, M., Kaya, B., & Kaya, M. (2025) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.70062 · Cited by 18
UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling
Yu, K., Zhou, Y., Bai, Y., Soh, Z. D., Xu, X., Goh, R. S. M., Cheng, C.-Y., & Liu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_70 · Cited by 17
Metadata
- Authors
- Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 28541-28564
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2306.00890
- OpenAlex
- W7133197087
- Cited by
- 590
- References
- 45
- Citations by year
- 2026: 288 · 2025: 179 · 2024: 115 · 2023: 8
- FWCI
- 63.10
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- multimodal language models, medical question answering, biomedicine, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
