The DOI matches its DataCite record; 47 of the 47 references it cites were found in a registry or an index.
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Open accessAbstract
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing multimodal medical data sets available for public use [84,85,148] are often relatively small in scale and demand the consolidation of numerous resources.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [84]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“LLaVA-Med Model Variants VQA-RAD SLAKE PathVQA Average Instruct Stage 1 Stage 2 FT Open Closed Open Closed Open Closed CLIP Vision Encoder [36], 7B Language Model 0 1 0 0 15.”
MethodBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“By contrast, LLaVA-Med uses a linear projection and a 7B LM [8, 43].”
MethodRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“Note that the image is not used to prompt GPT-4; we only show it here as a reference. original PubMed paper as additional context to the caption, inspired by the observations that external knowledge helps generalization [21, 28].”
BackgroundLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 1×
“To clarify, throughout the entire experiments, we only utilize the language-only GPT-4.”
MethodDomain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 1×
“Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
BackgroundSelf-citationDon’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · cited 1×
“Similar to the don’t stop pre-training argument in [12], we consider a scalable pipeline to create domain-specific instruct data from large unlabelled data, and advocate don’t stop instruction-tuning to build customized LMM. ( iii ) Low serving cost.”
BackgroundJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1×
“Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations.”
BackgroundMIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · cited 2×
“This dataset is two orders of magnitude larger than the next largest public dataset, MIMIC-CXR [16], and covers a diverse image types.”
BackgroundMethodBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 1×
“Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.”
Background
Research works printed but not linked: 1
- Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2023. 2, 3, 6, 9 · arXiv
Cited by
Sorted by citations
Items 201–239 of 239
Artificial Intelligence in Cutaneous Lesions: Where do we Stand and What is Next?
(2024) · Frontiers research topics · doi:10.3389/978-2-8325-4911-7 · Cited by 1
Afifi, S. M., Faragallah, M. H., Taha, R., Baig, M., Ullah, E., GholamHosseini, H., & Hassanein, S. I. (2024) · Journal of Engineering and Science in Medical Diagnostics and Therapy · doi:10.1115/1.4067302 · Cited by 3
Grasp-Sym: A Neuro-Symbolic Approach to Robust Grasp Classification
Zandigohar, M., & Schirner, G. (2025) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc64931.2025.11077518 · Cited by 1
“To be able to propose an approach that can improve accuracy on unseen objects through reasoning while avoiding the use of a VLM, we first need to understand VLMs such as LLaVA [12].”
MITS: A large-scale multimodal benchmark dataset for Intelligent Traffic Surveillance
Zhao, K., Liu, Z., Wang, P., Wang, X., Ma, Z., Xu, Y., Zhang, W., Nan, Y., Wang, K., & Lian, S. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105736 · Cited by 1
“While general-domain LMMs [13, 14, 15, 16] demonstrate remarkable adaptability, specialized applications in vertical domains such as medical imaging [17, 18, 33, 34, 35, 36] and autonomous driving [19, 20, 21] often require domain-specific models.”
InfluentialSummarizing Clinical Notes using LLMs for ICU Bounceback and Length-of-Stay PredictionPreprint
Choudhuri, A., Polgreen, P. M., Segre, A. M., et al. (2025) · medRxiv · doi:10.1101/2025.01.19.25320797 · Cited by 1
“…utilizes prompt crafting with a Pydantic Schema, in-context learning with few-shot examples, and leverages two specific open-source LLMs: Mistral 7B and Gemma 7B. [55] introduces LLaVA-Med, a novel method for creating a biomedical visual instruction-following model using a data-centric paradigm.”
Feng, W., Yazdani, A., Bornet, A., et al. (2025) · medRxiv · doi:10.1101/2025.03.17.25324103 · Cited by 1
Yılmaz, A., Yuceyalcin, F., Varol, R., Gökyayla, E., Erdem, O., Choi, D., Demirçalı, A. A., Gençoğlan, G., Posma, J. M., & Temelkuran, B. (2025) · medRxiv · doi:10.1101/2025.05.17.25327785 · Cited by 4
“A similar framework, LLaVa-Med, developed in the last few years [33] used 8 × A100 GPUs for training, costing an estimated $400–600 for training alone and significant labour force.”
The Illusion of Readiness in Health AI
Gu, Y., Fu, J., Liu, X., Valanarasu, J. M. J., Codella, N. C., Tan, R., Liu, Q., Jin, Y., Zhang, S., Wang, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.18234 · Cited by 1
Self-citationHolste, G., Zhou, Y., Wang, S., et al. (2023) · PubMed · doi:10.48550/arxiv.2310.16112
Xu, H., Liu, M., Yao, H., & Meng, J. H. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-97-8820-0_31
CTGLM: A Vision-Language Model for Automated Chinese Chest CT Report Generation
Zhang, K., Li, Y., Gao, Q., Huang, X., Hua, X., & Li, R. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-3755-3_11
Domain specific models outperform large vision language models on cytomorphology tasksPreprint
Kukuljan, I., Dasdelen, M. F., Schäfer, J., Buck, M. C., Götze, K. S. S., & Marr, C. (2025) · medRxiv · doi:10.1101/2025.05.05.25326989
Geração de laudos de retinografia utilizando Contrastive Captioner
Pimentel, P. O., Almeida, M. M., de Almeida, J. D. S., de Lemos, V. H. B., & Martins, L. E. S. C. (2025) · doi:10.5753/sbcas.2025.7824
Iftee, M. A. R., Faija, T., Shoumya, P. C., & Das, S. (2024) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit64611.2024.11022489
“Our approach combines the strengths of LLaVA-MED [4] and MED-SAM [5] to create an interactive medical image segmentation system capable of responding to user queries with reliable performance.”
Wang, C., Li, C., & Lin, G. (2025) · Journal of Imaging · doi:10.3390/jimaging11070240
“Additionally, compared with existing large models, such as LLaVA-Med [42], the proposed model requires significantly fewer computational resources (see Appendix A.4, Figure A2), making it highly suitable for deployment in resource-limited environments.”
Self-citationRedefining medical visual question answering using conditional generative diffusion models
Liu, B., Liu, L., Yang, X., Peng, W., & Liu, L. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108222
Momenian, M., & Shojaedini, S. V. (2025) · Research Square · doi:10.21203/rs.3.rs-7166904/v1
Enhancing Clinical Reasoning in Medical Vision-Language Model through Structured PromptsPreprint
Prakash, K. D., Kim, K., & Han, Y. (2025) · medRxiv · doi:10.1101/2025.07.31.25331901
“Additional developments include instruction-tuned models that use CoT reasoning to provide clinically appropriate outputs, including LLaVA-Med [16], Med-PaLM 2 [17], and Med-R1 [18].”
Hazarika, A., & Rahmati, M. (2025) · IEEE World AI IoT Congress (AIIoT) · doi:10.1109/aiiot65859.2025.11105279
“Inspired by recent LLM advancements in multi-modal reasoning and UAV decision-making [14], we employ Large Language and Vision Assistant (LLaVA) [15], a lightweight, vision-language model pre-trained on diverse datasets and fine-tuned for wildfire contexts.”
Коньшина, О., Першин, А., Кулябин, М., et al. (2025) · doi:10.24108/preprints-3113672
Zhou, M., Qi, Z., Zhu, T., Vijg, J., & Huang, X. (2025) · Mathematics · doi:10.3390/math13172739
BrainDEC: A Multimodal LLM for the Non-Invasive Decoding of Text from Brain Recordings
Hmamouche, Y., Chihab, I., Kdouri, L., & Seghrouchni, A. E. F. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103589 · Cited by 4
ConVLM: Context-guided vision-language model for fine-grained histopathology image classification
Sohail, A., Ganapathi, I. I., Alawode, B., Javed, S., Bennamoun, M., & Mahmood, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103737 · Cited by 2
Zhang, Y., Chen, L., Zhao, W., Zhang, H., Qiao, C., Liu, Z. C., Chung, C. H., Tan, M. C. J., Wang, M., Tham, Y. C., et al. (2026) · medRxiv · doi:10.64898/2026.01.15.26344200
Peng, C., Gao, J., Yang, N., et al. (2026) · JMIR Medical Informatics · doi:10.2196/87237
“Med-VQA interpretability by localizing image preference regions [39] enhanced the performance of biomedical VLMs by using instruction fine-tuning and multi-image training, respectively [40,41].”
Self-citationChao, C., Asadi, M., Li, L., Ramasamy, G., Pecco, N., Wang, Y.-C., Poterucha, T. J., Arsanjani, R., Kane, G. C., Oh, J. K., et al. (2026) · medRxiv · doi:10.64898/2026.03.14.26348388
Large language models in NLP: evolution, architectural trends, and open challenges
Javed, H., Shah, B., Ali, F., & Kwak, D. (2026) · Journal Of Big Data · doi:10.1186/s40537-026-01429-1 · Cited by 1
“LLaVA (Large Language and Vision Assistant) introduced effective instruction tuning techniques for vision-language tasks, making multimodal large language models (MLLMs) capabilities more accessible through open-source implementations [65].”
Hamma, M. B., Merrouni, Z. A., Frikh, B., & Ouhbi, B. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11556-z · Cited by 1
Multimodal Fusion Architectures for Clinical Vision Language Models
Ghosh, S., & Raj, D. R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7315-7.ch003
Tanvir, K., Biswas, R., Gomes, D., Rahman, M. M., Dey, N., & Islam, M. R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7315-7.ch009
Bridging the Patient Comprehension Gap: An Integrated Augmented Reality and Multimodal AI Framework
Lin, X., Alam, N. A., Jang, S., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26352-0_4
Saadi, S. B., Berne, J. V., Fontenele, R. C., Claes, P. D., & Jacobs, R. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02086-9
Wasi, A. T., Anik, M. A., Islam, M. S., & Ahsan, M. M. (2026) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2026.2703103
CARR: Cross-Modal Adaptation with Reverse Reasoning for Medical Visual Question Answering
Yan, W., Zhu, J., Zhang, L., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3378-6_15
Identifying the presence of disc herniations in lumbar spine MRI using Gemini 3.1 Pro
Rodríguez, L., Naidu, T., Chetla, N., Mallepally, A., Kumar, R., Duggan, S., Bouras, A., Sporn, K., Carey, M., Raja, V., et al. (2026) · European Spine Journal · doi:10.1007/s00586-026-10285-9
Nguyen, B.-D., Hoang, V.-D., & Nguyen, H. D. (2026) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-026-00871-8
DermAgent: A Multi-tool Agentic System for Evidence-Grounded and Traceable Reasoning in Dermatology
Liu, Y., Yan, S., Hu, M., Ju, L., Li, X., Tang, F., Feng, W., & Ge, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38062-3_16
Wang, H., & Li, Z. (2026) · arXiv · doi:10.48550/arxiv.2605.20469
“Generative models such as LLaVA-Med [5], Med-Flamingo [7], CheXagent [6], RadFM [16], InternVL2 [3], and Qwen2-VL [2] produce free-text reports with richer clinical detail but introduce the risk of open-ended hallucination.”
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Metadata
- Authors
- Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 28541-28564
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2306.00890
- OpenAlex
- W7133197087
- Cited by
- 590
- References
- 45
- Citations by year
- 2026: 288 · 2025: 179 · 2024: 115 · 2023: 8
- FWCI
- 63.10
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- multimodal language models, medical question answering, biomedicine, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
