The DOI matches its DataCite record; 47 of the 47 references it cites were found in a registry or an index.
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Open accessAbstract
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing multimodal medical data sets available for public use [84,85,148] are often relatively small in scale and demand the consolidation of numerous resources.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“LLaVA-Med Model Variants VQA-RAD SLAKE PathVQA Average Instruct Stage 1 Stage 2 FT Open Closed Open Closed Open Closed CLIP Vision Encoder [36], 7B Language Model 0 1 0 0 15.”
MethodBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“By contrast, LLaVA-Med uses a linear projection and a 7B LM [8, 43].”
MethodRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“Note that the image is not used to prompt GPT-4; we only show it here as a reference. original PubMed paper as additional context to the caption, inspired by the observations that external knowledge helps generalization [21, 28].”
BackgroundLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 1×
“To clarify, throughout the entire experiments, we only utilize the language-only GPT-4.”
MethodDomain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 1×
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundSelf-citationDon’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · cited 1×
“Similar to the don’t stop pre-training argument in [12], we consider a scalable pipeline to create domain-specific instruct data from large unlabelled data, and advocate don’t stop instruction-tuning to build customized LMM. ( iii ) Low serving cost.”
BackgroundJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1×
“Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations.”
BackgroundMIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · cited 2×
“This dataset is two orders of magnitude larger than the next largest public dataset, MIMIC-CXR [16], and covers a diverse image types.”
BackgroundMethodBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 1×
“Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.”
Background
Research works printed but not linked: 1
- Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2023. 2, 3, 6, 9 · arXiv
Cited by
Sorted by citations
Items 101–200 of 239
SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology
Kapse, S., Pati, P., Das, S., Zhang, J., Chen, C., Vakalopoulou, M., Saltz, J. H., Samaras, D., Gupta, R., & Prasanna, P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01067 · Cited by 24
“However these methods [20, 38] either suffer from post-hoc approximation [51], or are not scalable [32, 69] for gigapixel images.”
MethodBlenderAlchemy: Editing 3D Graphics with Vision-Language Models
Huang, I., Yang, G., & Guibas, L. J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73024-5_18 · Cited by 34
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Healthcare: A Growing Role for Large Language Models and Generative AI
Pahune, S. A., & Rewatkar, N. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.55573 · Cited by 19
Beyond Text: Frozen Large Language Models in Visual Signal Comprehension
Zhu, L., Wei, F., & Lu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02554 · Cited by 24
Engelke, M., Baldini, G., Kleesiek, J., Nensa, F., & Dada, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf165 · Cited by 15
MISS: A Generative Pre-training and Fine-Tuning Approach for Med-VQA
Chen, J., Yang, D., Jiang, Y., Lei, Y., & Zhang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72353-7_22 · Cited by 31
HumanVLM: Foundation for Human-Scene Vision-Language Model
Dai, D., Xu, L., Li, Y., Zhang, Y., & Xia, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103271 · Cited by 19
Radio DINO: A foundation model for advanced radiomics and AI-driven medical imaging analysis
Zedda, L., Loddo, A., & Ruberto, C. D. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110583 · Cited by 23
Medical multimodal large language models: A systematic review
Hu, Y., Xu, C., Lin, B., Yang, W., & Tang, Y. Y. (2025) · Intelligent Oncology · doi:10.1016/j.intonc.2025.09.005 · Cited by 14
MemeCraft: Contextual and Stance-Driven Multimodal Meme Generation
Wang, H., & Lee, R. K.-W. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3648151 · Cited by 18
Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis
Verma, P., Van, M.-H., & Wu, X. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825000 · Cited by 20
“Built upon the success of LLaVA, LLaVA-Med [23] is a step towards building a multimodal chat assistant for medical data upon previous LLaVA architecture.”
GPT-4 as a biomedical simulator
Schaefer, M., Reichl, S., ter Horst, R., Nicolas, A. M., Krausgruber, T., Piras, F., Stepper, P., Bock, C., & Matthias Samwald, A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108796 · Cited by 13
Liu, X., Gong, W., Chen, X., Li, Z., Li, Z., Wang, L., Liu, Q., Sun, X., Liu, X., Chen, X., et al. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111646 · Cited by 14
Tung, Y.-H., Yang, Z.-R., Shen, M.-W., Chang, C., Chen, C., & Ho, L.-C. (2025) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2025.105371 · Cited by 17
Guo, M., Wu, M., Shen, Y., Li, H., & Tao, C. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111672 · Cited by 18
“In the biomedical field, Li et al. [18] developed a biomedical image instruction-following dataset based on medical images and corresponding descriptions.”
Wang, D., Cheng, T., Wang, S., Chen, Y., & Yin, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104297 · Cited by 14
Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning
Shen, C., Lian, C., Zhang, W., Wang, F., Zhang, J., Fan, S., Wei, X., Wang, G., Li, K., Mu, H., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62060-x · Cited by 18
MEDITRON: Open Medical Foundation Models Adapted for Clinical PracticePreprint
Chen, Z., Romanou, A., Bonnet, A., Hernández-Cano, A., AlKhamissi, B., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4139743/v1 · Cited by 16
ChatNT: A Multimodal Conversational Agent for DNA, RNA and Protein TasksPreprint
Richard, G., de Almeida, B. P., Dalla-Torre, H., Blum, C. F., Hexemer, L., Pandey, P., Laurent, S., Lopez, M., Laterre, A., Lang, M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.30.591835 · Cited by 14
Exp-VQA: Fine-grained facial expression analysis via visual question answering
Yuan, Y., Zeng, J., & Shan, S. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111783 · Cited by 12
Lin, M., Holste, G., Wang, S., Zhou, Y., Wei, Y., Banerjee, I., Chen, P., Dai, T., Du, Y., Dvornek, N. C., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103739 · Cited by 22
Miao, Z., Zhang, Y., Fabian, Z., Celis, A. H., Beery, S., Li, C., Liu, Z., Gupta, A., Nasir, M., Li, W., et al. (2025) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.70120 · Cited by 25
Self-citationPath to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest CostPreprint
Zhou, J., Chen, X., & Gao, X. (2023) · medRxiv · doi:10.1101/2023.06.23.23291802 · Cited by 21
Truong, T.-D., Nguyen, H.-Q., Nguyen, X.-B., Dowling, A. P. G., Li, X., & Luu, K. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02521-4 · Cited by 17
“…present their robustness, large-scale Besides the general-purpose foundation models, the large multimodal models for specific fields, e.g., biomedicine (Li et al., 2024), social analysis (Zhang et al., 2024), or agri-culture (Gharaee et al., 2023), remain a significant research topic because of their…”
Navigation Instruction Generation with BEV Perception and Large Language Models
Fan, S., Liu, R., Wang, W., & Yang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72670-5_21 · Cited by 17
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
Pi, R., Han, T., Xiong, W. H., Zhang, J., Liu, R., Pan, R., & Zhang, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_22 · Cited by 43
ArtEyer: Enriching GPT-based agents with contextual data visualizations for fine art authentication
Tang, T., Wu, Y., Gao, J., Ruan, K. Q., Zhang, Y., Ye, S., Wu, Y., & Chen, X. (2024) · Visual Informatics · doi:10.1016/j.visinf.2024.11.001 · Cited by 10
Self-citationRa, S., Kim, J., Na, I., Ko, E. S., & Park, H. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108765 · Cited by 13
Panacea: A foundation model for clinical trial search, summarization, design, and recruitmentPreprint
Lin, J., Xu, H., Wang, Z., Wang, S., & Sun, J. (2024) · medRxiv · doi:10.1101/2024.06.26.24309548 · Cited by 15
Vision-BioLLM: Large vision language model for visual dialogue in biomedical imagery
AlShibli, A., Bazi, Y., Rahhal, M. M. A., & Zuair, M. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107437 · Cited by 11
IR-GPT: AI Foundation Models to Optimize Interventional Radiology
Brenner, J., Anibal, J., Hazen, L., Song, M. J., Huth, H. B., Xu, D., Xu, S., & Wood, B. J. (2025) · CardioVascular and Interventional Radiology · doi:10.1007/s00270-024-03945-0 · Cited by 10
The Development Landscape of Large Language Models for Biomedical Applications
Cao, Z., Keloth, V. K., Xie, Q., Qian, L., Liu, Y., Wang, Y., Shi, R., Zhou, W., Yang, G. F., Zhang, J., et al. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-102224-074736 · Cited by 16
Ultrasound report generation with fuzzy knowledge and multi-modal large language model
Li, Z., Li, M., Wang, W., & Huang, Q. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128555 · Cited by 13
Wang, Y., Gu, Y., Zhang, X., Wang, B., Wang, R., Li, X., Liu, Y., Qu, F., Ren, F., Yan, R., et al. (2025) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003790 · Cited by 9
“When evaluated using the diagnostic task benchmark PathQABench, PathChat achieved an accuracy of 89.5%, outperforming leading AI models such as LLaVA [53] and GPT-4V.”
InfluentialChEX: Interactive Localization and Region Description in Chest X-Rays
Müller, P., Kaissis, G. A., & Rueckert, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72664-4_6 · Cited by 19
Advancing holistic educational goals through generative language-based technologies
Nussbaum, M., & Bekerman, Z. (2024) · Learning and Individual Differences · doi:10.1016/j.lindif.2024.102600 · Cited by 10
Voice EHR: introducing multimodal audio data for health
Anibal, J., Huth, H. B., Li, M., Hazen, L., Daoud, V., Ebedes, D. M., Yen, L. M., Nguyen, H. T. T., Hong, P. V., Kleinman, M., et al. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1448351 · Cited by 7
Afzaal, U., Su, Z., Sajjad, U., Stack, T., Lü, H., Niu, S., Akbar, A., Gürcan, M. N., & Niazi, M. K. K. (2025) · Patterns · doi:10.1016/j.patter.2025.101284 · Cited by 9
“9,52,53 However, the challenge lies in the noise in these datasets, where the text may not accurately represent the image or vice versa.”
Explainable medical visual question answering via chain of evidence
Qiu, C., Huang, K., Xie, Z., Liu, M., Gu, J., & Zong, X. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113672 · Cited by 8
How well do multimodal LLMs interpret CT scans? An auto-evaluation framework for analyses
Zhu, Q., Hou, B., Mathai, T. S., Mukherjee, P., Jin, Q., Chen, X., Wang, Z., Cheng, R., Summers, R. M., & Lu, Z. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104864 · Cited by 9
Muhammad, D., Salman, M., & Bendechache, M. (2025) · Computational and Structural Biotechnology Reports · doi:10.1016/j.csbr.2025.100066 · Cited by 11
Gautam, S., Riegler, M. A., & Halvorsen, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-08009-7_6 · Cited by 9
The Role of Large Language Models ( LLMs ) in Breast Imaging Today and in the Near Future
Schiaffino, S., Zhang, T., Mann, R. M., & Pinker, K. (2025) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.29807 · Cited by 8
Distributed VLMs: Efficient Vision-Language Processing through Cloud-Edge Collaboration
Li, Y., Gumaste, D., Türkcan, M. K., Ghaderi, J., Zussman, G., & Kostić, Z. (2025) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops65533.2025.00078 · Cited by 8
“From autonomous vehicles [6] and robotic navigation to augmented reality and assistive technologies [7], VLMs are enabling machines to understand and generate language from visual data with remarkable accuracy and versatility.”
Pan, X., Bai, Y., Zou, K., Zhou, Y., Zhou, J., Fu, H., Tham, Y. C., & Liu, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103631 · Cited by 8
“For biomedical image analysis, LLaVA-Med [33] delivers superior multimodal conversational capabilities with low training costs, while MENDER [34] excels in differential-aware medical VQA through cross-modal knowledge diffusion and multi-scale medical knowledge fusion.”
Inamullah, I., Razzak, I., & Jameel, S. (2025) · The Journal of Precision Medicine Health and Disease · doi:10.1016/j.premed.2025.100023 · Cited by 8
“At the same time, multi-system screening models such as DeepSLE preview a future where a single retinal image could support simultaneous risk profiling across cardiovascular, neurological, and immune conditions.”
A preliminary study into the emergence of tendon microstructure during postnatal development
Raymond-Hayling, H., Lu, Y., Tom Shearer, N., & Kadler, K. E. (2024) · Matrix Biology Plus · doi:10.1016/j.mbplus.2024.100142 · Cited by 7
“For instance, through the implementation of joint vision and language models, as demonstrated in [42], images of collagen fibrils from diverse research sources can be efficiently interpreted and described with scientific language.”
BackgroundMethodMetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q., Liu, Y., Zhou, P., Wan, Y., Gong, N. Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03128 · Cited by 7
Physically-guided open vocabulary segmentation with weighted patched alignment loss
Liu, W., Lou, J., Wang, X., Zhou, W., Cheng, J. T., & Yang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128788 · Cited by 7
MEDSQ: Towards personalized medical education via multi-form interaction guidance
Ouyang, Y., Gao, W., Wang, H., Chen, L., Wang, J., & Zeng, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126138 · Cited by 8
Xu, J., Wang, J., Li, J., Zhu, Z., Xiao, F., Cai, W., Song, R., Wang, T. F., & Li, H. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02192-1 · Cited by 9
“…this study assessed only three representative LLMs for handling medical imaging inputs, excluding newer models such as Claude 3 Opus (39), LLaVA(40), and Grok(41), as well as recently updated versions of existing LLMs. Future research should comprehensively evaluate a broader range of LLMs and…”
LaTP: LiDAR-aided multimodal token pruning for efficient trajectory prediction of autonomous driving
Lu, Y., Sun, S., Liu, N., Jiang, B., Li, Y., Chen, J., Zhang, Y., Zhu, Y., & Velipasalar, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107673 · Cited by 7
Knowledge fusion in deep learning-based medical vision-language models: A review
Xu, D., Chen, Y., Chai, Z., Xiao, Y., Yan, Y., Ding, W., Wang, H., Jin, Z. J., Jiao, W., Yue, W., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103455 · Cited by 6
Hao, C., Wang, Z., Mao, X., He, S., Li, B., Liu, H., Peng, F., & Li, W. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104302 · Cited by 7
Self-citationA survey for wearable sensors empowering smart healthcare in the era of large language models
Liu, Y., Li, J., Lian, C., Zhang, X., Gong, J., Wang, Y., & Zhao, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103409 · Cited by 8
Detours for Navigating Instructional Videos
Ashutosh, K., Xue, Z., Nagarajan, T., & Grauman, K. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01779 · Cited by 7
“…using a video encoder f v (e.g., InternVideo [72]), extracted at one feature per second [6, 75] Next, we use a low-parameter visual mapper f vm [27, 39, 46, 51, 85] to convert the video features into visual tokens v = f vm ( f v ( V )) that are in the same embedding space as the text tokens,…”
MethodRevolutionizing Marketing Research with a Large Language Model: A Hybrid AI-Human ApproachPreprint
Arora, N. K., Chakraborty, I., & Nishimura, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4683054 · Cited by 7
From pixels to patients: the evolution and future of deep learning in cancer diagnostics
Yang, Y., Shen, H., Chen, K., & Li, X. (2024) · Trends in Molecular Medicine · doi:10.1016/j.molmed.2024.11.009 · Cited by 6
Cost-effective instruction learning for pathology vision and language analysis
Chen, K., Liu, M., Yan, F., Ма, Л., Shi, X., Wang, L., Wang, X., Zhu, L., Wang, Z., Zhou, M., et al. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00818-5 · Cited by 6
“CLOVER’s performance is twice that of BLIP-2 and approaches five times that of LLaVA in open-ended question scenarios.”
InfluentialSelf-citationQin, Z., He, Z., Zhang, Y., Shen, Y., & Li, K. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112178 · Cited by 5
Insight: A Multi-modal Diagnostic Pipeline Using LLMs for Ocular Surface Disease Diagnosis
Yeh, C.-H., Wang, J., Graham, A. D., Liu, A. J., Tan, B., Chen, Y., Ma, Y., & Lin, M. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72378-0_66 · Cited by 5
“MLLMs, when utilized for diagnostic purposes [9,12,33], are capable of producing hundreds to thousands of free-form answers with clinical reasoning, instead of being limited to closed-set predictions.”
Foundation Models for Translational Cancer Biology
Tsang, K. K., Kivelson, S., Cortina, J. M. A., Kuchi, A., Berkowitz, J. S., Liu, H., Srinivasan, A., Friedrich, N. A., Fatapour, Y., & Tatonetti, N. P. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095633 · Cited by 5
Self-citationAccelerating primer design for amplicon sequencing using large language model-powered agents
Wang, Y., Hou, Y., Yang, L., Li, S., Tang, W., Tang, H. R., He, Q., Lin, S., Zhang, Y., Li, X., et al. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01455-z · Cited by 8
“To enable the VLM to accurately detect the aforementioned anomalies, we implemented a two-step fine-tuning strategy 79 .”
Symmetry-Aware Advances in Multimodal Large Language Models: Architectures, Training, and Evaluation
Liu, X., & Liu, H. (2025) · Symmetry · doi:10.3390/sym17091400 · Cited by 7
Self-citationAdopting Generative AI with Precaution in Dentistry: A Review and Reflection
Xu, M., Ye, C., Zeng, Z., Chang, C., Qi, S., Wu, Y., Yang, H., Chen, Y., Huang, H., Liu, L., et al. (2024) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh62654.2024.00047 · Cited by 5
“LLaVA-med [38] adopts a linear mapping layer to project medical image features into text space, enabling the LLM to comprehend them.”
BackgroundPerformance of vision language models for optic disc swelling identification on fundus photographs
Li, K. Z., Nguyen, T. T., & Moss, H. E. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1660887 · Cited by 4
CKE-Former: A clinical knowledge-enhanced transformer for disease classification in telemedicine
Peng, Q., Cai, Y., Liu, J., Chen, X., Wang, Z., Xie, J., & Li, Q. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113259 · Cited by 4
SigPhi-Med: A lightweight vision-language assistant for biomedicine
Zhou, F., Liu, X., Zeng, Q., Li, Z., & Xiao, H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104849 · Cited by 6
New frontiers in AI for biodiversity research and conservation with multimodal language modelsPreprint
Miao, Z., Zhang, Y., Fabian, Z., Celis, A. H., Beery, S., Li, C., Liu, Z., Gupta, A., Nasir, M., Li, W., et al. (2024) · doi:10.32942/x22s6f · Cited by 4
Self-citationDomain Adaptation and Generalization Using Foundation Models in Healthcare ImagingPreprint
Ying, H., Lia, Y., & Fu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5345726 · Cited by 4
From task-specific to foundation models: A paradigm shift in medical vision-language analysis
Ali, M. U., Zafar, A., Kim, S., Kim, K. S., & Lee, S. W. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100831 · Cited by 6
Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives
Queiroz, D., Carlos, A., Anjos, A., & Berton, L. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3793542 · Cited by 9
“Recent research demonstrates, however, that effective bias and hallucinations eliminations techniques can be implemented without requiring access to internal model parameters, offering viable solutions for both open-source and proprietary model frameworks [148, 100].”
Enhancing Visual Question Answering for Arabic Language Using LLaVa and Reinforcement Learning
ElMaghraby, A., Maged, S., Essawey, M., et al. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.10.207 · Cited by 3
LLaVA-NeXT-Med: Medical Multimodal Large Language Model
Guo, Y., & Huang, W. X. (2025) · Asia-Europe Conference on Cybersecurity, Internet of Things and Soft Computing (CITSC) · doi:10.1109/citsc64390.2025.00092 · Cited by 3
“LLaVA-Med is a multimodal model specifically designed for the biomedical field [8].”
InfluentialMedKit: Multi-level feature distillation with knowledge injection for radiology report generation
Su, Z., Song, H., Lin, Y., Wu, Y., Weng, X., Mao, Z., Liu, B., Yin, H., & Yang, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129003 · Cited by 4
Self-citationB., C. E. P., R., O. I. I., A., M. D. L., & M., C. G. Q. (2025) · Systems · doi:10.3390/systems13090753 · Cited by 3
“The study in [22] introduces LLaVA-Med, a vision–language conversational assistant designed to answer open-ended biomedical research questions by combining multimodal AI with domain-specific datasets.”
Sharma, K., Vats, V., Singh, A., et al. (2024) · doi:10.1109/ijcnn60899.2024.10651096 · Cited by 3
VISLIX: An XAI Framework for Validating Vision Models with Slice Discovery and Analysis
Yan, X., Xuan, X., Ono, J. P., Guo, J., Mohanty, V., Shekar, A. K., Gou, L., Wang, B., & Ren, L. (2025) · Computer Graphics Forum · doi:10.1111/cgf.70125 · Cited by 2
“At the project’s inception, we utilized VLMs like BLIP-2 [LLSH23] and LLaVA [LWZ ∗ 24] and LLMs”
Medical image segmentation using dual-decoder mutual teaching with a mean teacher framework
Zhang, J., Jiang, G., Li, Z., Tian, B., Yu, S., Yu, Q., Zhou, J., Chen, H., Pu, J., Yi, Q., et al. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112184 · Cited by 3
CaRe-Ego: Contact-aware relationship modeling for egocentric interactive hand-object segmentation
Su, Y., Wang, Y., & Chau, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129148 · Cited by 2
Domain-graph enhanced multi-task multimodal large models for urban planning
Wang, C., Wang, Y., & Zhao, W. (2025) · Indoor and Built Environment · doi:10.1177/1420326x251368756 · Cited by 2
radio-llava: Advancing vision-language models for radio astronomical source analysis
Riggi, S., Cecconello, T., Pilzer, A., Palazzo, S., Gupta, N., Hopkins, A., Trigilio, C., & Umana, G. (2025) · Publications of the Astronomical Society of Australia · doi:10.1017/pasa.2025.10082 · Cited by 14
Clinical knowledge enhanced medical image classification
Xu, Z., Liang, J., Wei, Z., Yue, X., & Li, D. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112414 · Cited by 2
Zeng, L., Lin, X., & Yu, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103783 · Cited by 3
Patrício, C., Rio-Torto, I., Cardoso, J. S., Teixeira, L. F., & Neves, J. C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111145 · Cited by 6
Yamamoto, K., & Kikuchi, T. (2025) · Bioengineering · doi:10.3390/bioengineering12101093 · Cited by 2
“To scale training, most studies mine figures and captions from large bibliographic databases and then employ large language models (LLMs) to synthesize visual question answering (VQA) pairs [7,8].”
JW-VL: A Vision-language Model for Solar Physics
Shao, M., Wang, H., Tong, L., Li, Y., Wang, C., Lin, J. B., Suo, L., Xu, H., Zhang, Y., & Huang, J. (2026) · Research in Astronomy and Astrophysics · doi:10.1088/1674-4527/ae5a5d · Cited by 2
Knowledge Distillation in Mixture of Experts for Multi-Modal Medical LLMs
Nathani, M., Soni, R., & Mishra, R. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825733 · Cited by 3
“Advancements in Medical Multimodal Large Language Models (MLLMs), including Med-Flamingo [7], Med-PaLM M [8], and LLaVA-Med [9], have greatly influenced medical diagnostics and patient care.”
Efficient annotation bootstrapping for cell identification in follicular lymphoma
Krawczyk, A., Osowska-Kurczab, A. M., Pakuło, S., Kotłowski, W., & Świderska-Chadaj, Ż. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108728 · Cited by 3
Cerit, M., Zelikman, E., Cho, M.-J., Robinson, T. N., Reeves, B. B., Ram, N., & Haber, N. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720055 · Cited by 4
“Some clusters were app-specific, such as “Candy Crush Saga,” while others distinguished content within apps, including “TikTok Kitchen Content.”
I-Viewer: An Online Digital Pathology Analysis Platform with Agentic-RAG AI CopilotPreprint
Rong, R., Luo, D., Gu, Z., Quan, P., Villanueva-Miranda, I., Wang, J., Yang, S., Chi, Z., Leavey, P. J., Yang, D. M., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-5404747/v1 · Cited by 2
NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context
Yao, B., Li, Q., Zhang, Y., Yang, S., Zhang, B., Tiwari, P., & Qin, J. (2025) · arXiv · doi:10.48550/arxiv.2505.08734 · Cited by 1
“Through post-training on synthetic medical data, large language models (LLMs) have gained the understanding and conversational ability in the medical domain [31, 44, 46].”
Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components
Warr, H., Ibrahim, Y., McGowan, D. R., & Kamnitsas, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73158-7_7 · Cited by 1
Language-Enhanced Local-Global Aggregation Network for Multi-organ Trauma Detection
Yu, J., Hu, Q., Jiang, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72086-4_37 · Cited by 1
Self-citationLarge multimodal model assisted underground tunnel damage inspection and human-machine interaction
Qi, Y., Ding, Z., & Luo, Y. (2025) · Journal of Infrastructure Intelligence and Resilience · doi:10.1016/j.iintel.2025.100154 · Cited by 1
Liu, L., Su, X., Wang, X., & Gao, G. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108308 · Cited by 1
CorGPT: Coronary Angiography Imaging Analysis Using Large Medical Vision-Language Models
Huang, B., Zhang, H., Liu, Z., Wang, J., & Zhou, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0027-7_24 · Cited by 1
Patchwise Cooperative Game-based Interpretability Method for Large Vision-language Models
Zhu, Y. T., Zhang, Y., Wang, Z., Yan, X. J., Sun, P., & Ji, X. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00756 · Cited by 2
Review of generative AI for lesion localization and automatic report generation
Sun, K., Wang, Y., Huang, S., Liu, Y., Chen, H., Lian, Z., Tan, M., Lin, X., Zhou, W., Qiu, Z., et al. (2026) · Med-X · doi:10.1007/s44258-026-00076-8 · Cited by 1
“In contrast to vanilla VLMs such as LLaVA [160], Fla-mingo [161], and their variants [19, 201, 202], grounded report generation models [153, 165, 166, 171] intend to explicitly align report narratives with the corresponding visual evidence.”
Metadata
- Authors
- Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 28541-28564
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2306.00890
- OpenAlex
- W7133197087
- Cited by
- 590
- References
- 45
- Citations by year
- 2026: 288 · 2025: 179 · 2024: 115 · 2023: 8
- FWCI
- 63.10
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- multimodal language models, medical question answering, biomedicine, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
