PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Open accessAbstract
Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery by leveraging artificial intelligence to interpret and answer questions based on medical images. In this study, we reframe the problem of MedVQA as a generation task that naturally follows the human-machine interaction and propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model. We establish a scalable pipeline to construct a large-scale medical visual question-answering dataset, named PMC-VQA, which contains 227k VQA pairs of 149k images that cover various modalities or diseases. We train the proposed model on PMC-VQA and then fine-tune it on multiple public benchmarks, e.g., VQA-RAD, SLAKE, and Image-Clef-2019, significantly outperforming existing MedVQA models in generating relevant, accurate free-form answers. In addition, we propose a test set that has undergone manual verification, which is significantly more challenging, serving to better monitor the development of generative MedVQA methods. To facilitate comprehensive evaluation and comparison, we have maintained a leaderboard at https://paperswithcode.com/paper/pmc-vqa-visual-instruction-tuning-for-medical, offering a centralized resource for tracking progress and benchmarking state-of-the-art approaches. The PMC-VQA dataset emerges as a vital resource for the field of research, and the MedVInT presents a significant breakthrough in the area of MedVQA.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing multimodal medical data sets available for public use are often relatively small in scale and demand the consolidation of numerous resources.
“Existing multimodal medical data sets available for public use [84,85,148] are often relatively small in scale and demand the consolidation of numerous resources.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [85]
Section 4.1 identifies a lack of large-scale, multimodal medical VQA datasets, and Section 4.3.1 gives the relatively small sizes of three public benchmarks. The paper does not say that using these datasets demands consolidating numerous resources. (LLM, conf=0.84)
Experimental — accuracy not yet measuredPMC-VQA and LLaVA-Med include text and image modalities for medical visual question answering.
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [85]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Self-citationBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2× · see citing works
“2 Vision Backbone CLIP [30] is a model trained from scratch on a dataset of 400 million image-text pairs collected from the internet with contrastive loss.”
MethodDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 2× · see citing works
“Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix ® database 0.3k 3.5k PathVQA [12] Pathology PEIR Digital Library [14] 5k 32.8k SLAKE [23] Radiology MSD [3], ChestX-ray8 [36], CHAOS [15] 0.7k 14k VQA-Med-2021 [5] Radiology MedPix ® database 5k 5k PMC-VQA Mixture * PubMed Central ® 149k 227k Figure 1: The…”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 6× · see citing works
“We fine-tuned LLaMA-7B [35] on this training set, and our model achieved an accuracy of 81.”
BackgroundMethodInfluentialKung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1× · see citing works
“Notably, even in domains where expert knowledge is supposed to play a critical role, like medical diagnosis, these language models have also achieved impressive success, passing the United States Medical Licensing Examination (USMLE) [13, 17, 26, 32].”
BackgroundLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1× · see citing works
“Notably, even in domains where expert knowledge is supposed to play a critical role, like medical diagnosis, these language models have also achieved impressive success, passing the United States Medical Licensing Examination (USMLE) [13, 17, 26, 32].”
BackgroundLarge Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 1× · see citing works
“Existing MedVQA methods [25, 22, 6, 20] typically treat the problem as a retrieval task with a limited answer base and train multi-modal vision-language models with contrastive or classification objectives.”
MethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“Large Language Models (LLMs) have recently achieved tremendous success [28, 27, 1] in generating high-quality text for various tasks such as language translation, summarization, and question answering.”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1× · see citing works
“Large language models (LLMs), such as ChatGPT [1], PaLM [8], LLaMA [35], have recently demonstrated remarkable progress in a wide range of natural language processing (NLP) tasks, such as question answering, text classification, and interactive dialog.”
BackgroundDomain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 1× · see citing works
“PubMedBERT [11] is an encoder-based BERT-like model that is trained from scratch using abstracts from PubMed and full-text articles from PubMedCentral in the corpus “The Pile” [10].”
MethodArtificial intelligence in healthcare: transforming the practice of medicine
Bajwa, J., Munir, U., Nori, A., & Williams, B. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0095 · Cited by 1,856 · see citing works
The Medical Segmentation Decathlon
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B. A., Litjens, G., Menze, B., Ronneberger, O., Summers, R. M., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30695-9 · Cited by 1,384 · cited 2× · see citing works
“Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix ® database 0.3k 3.5k PathVQA [12] Pathology PEIR Digital Library [14] 5k 32.8k SLAKE [23] Radiology MSD [3], ChestX-ray8 [36], CHAOS [15] 0.7k 14k VQA-Med-2021 [5] Radiology MedPix ® database 5k 5k PMC-VQA Mixture * PubMed Central ® 149k 227k Figure 1: The…”
Self-citationBurbidge, D., & Raji, T. (2020) · The Oxford Handbook of Kenyan Politics · doi:10.1093/oxfordhb/9780198815693.013.47 · Cited by 1,162 · cited 1×
“The total possible answers for PMC-VQA are over 100 K, which makes the traditional retrieval-based approach limited in usefulness for the answer set of such a level.”
BackgroundThe future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · cited 3× · see citing works
“Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix ® database 0.3k 3.5k PathVQA [12] Pathology PEIR Digital Library [14] 5k 32.8k SLAKE [23] Radiology MSD [3], ChestX-ray8 [36], CHAOS [15] 0.7k 14k VQA-Med-2021 [5] Radiology MedPix ® database 5k 5k PMC-VQA Mixture * PubMed Central ® 149k 227k Figure 1: The…”
BackgroundLi, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · cited 4× · see citing works
“Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix ® database 0.3k 3.5k PathVQA [12] Pathology PEIR Digital Library [14] 5k 32.8k SLAKE [23] Radiology MSD [3], ChestX-ray8 [36], CHAOS [15] 0.7k 14k VQA-Med-2021 [5] Radiology MedPix ® database 5k 5k PMC-VQA Mixture * PubMed Central ® 149k 227k Figure 1: The…”
BackgroundMethodInfluentialCHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation
Kavur, A. E., Gezer, N. S., BARIŞ, M. M., Aslan, S., Conze, P.-H., Groza, V., Pham, D. D., Chatterjee, S., Ernst, P., Özkan, S., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101950 · Cited by 828 · cited 2× · see citing works
“Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix ® database 0.3k 3.5k PathVQA [12] Pathology PEIR Digital Library [14] 5k 32.8k SLAKE [23] Radiology MSD [3], ChestX-ray8 [36], CHAOS [15] 0.7k 14k VQA-Med-2021 [5] Radiology MedPix ® database 5k 5k PMC-VQA Mixture * PubMed Central ® 149k 227k Figure 1: The…”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718 · cited 1× · see citing works
“Notably, even in domains where expert knowledge is supposed to play a critical role, like medical diagnosis, these language models have also achieved impressive success, passing the United States Medical Licensing Examination (USMLE) [13, 17, 26, 32].”
BackgroundConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · cited 1× · see citing works
“Notably, even in domains where expert knowledge is supposed to play a critical role, like medical diagnosis, these language models have also achieved impressive success, passing the United States Medical Licensing Examination (USMLE) [13, 17, 26, 32].”
BackgroundThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 1× · see citing works
“PubMedBERT [11] is an encoder-based BERT-like model that is trained from scratch using abstracts from PubMed and full-text articles from PubMedCentral in the corpus “The Pile” [10].”
MethodMiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · see citing works
MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis
Yang, J., Shi, R., & Ni, B. (2021) · 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi48211.2021.9434062 · Cited by 414 · cited 1× · see citing works
“Same with MedVInT-TE, it also encodes the question input into embedding features q and can perform multi-choice or blank through different prompts.”
MethodPMC-CLIP: Contrastive Language-Image Pre-training using Biomedical DocumentsPreprint
Lin, W.-X., Zhao, Z.-H., Zhang, X., et al. (2023) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.48550/arxiv.2303.07240 · Cited by 363 · cited 9× · see citing works
“Method Language Backbone Vision Backbone Blanking Choice ACC Bleu-1 ACC Zero-shot PMC-CLIP [20] PMC-CLIP [20] PMC-CLIP [20] – – 24.”
MethodInfluentialSelf-citationSlake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering
Liu, B., Zhan, L.-M., Xu, L., Ma, L., Yang, Y., & Wu, X.-M. (2021) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi48211.2021.9434010 · Cited by 351 · cited 4× · see citing works
“MedVQA is a task of answering natural language questions about medical visual content, typically images or videos obtained from medical devices like X-ray, CT, MRI, or microscopy, etc. Specifically, our goal is to train a model that can output the corresponding answer for a given question, which…”
BackgroundInfluentialPubMed Central: The GenBank of the published literature.
Roberts, R. J. (2001) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.98.2.381 · Cited by 261 · cited 1× · see citing works
“6 million image-text pairs collected from PubMedCentral (PMC)’s OpenAccess subset [31], which covers 2.”
MethodTowards Generalist Foundation Model for RadiologyPreprint
Wu, C., Zhang, X., Zhang, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.02463 · Cited by 236 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundSelf-citationRadiology Objects in COntext (ROCO): A Multimodal Image Dataset
Pelka, O., Koitka, S., Rückert, J., Nensa, F., & Friedrich, C. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01364-6_20 · Cited by 221 · see citing works
Overcoming Data Limitation in Medical Visual Question Answering
Nguyen, B. D., Do, T.-T., Nguyen, B. X., Do, T., Tjiputra, E., & Tran, Q. D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9_57 · Cited by 220 · cited 3× · see citing works
“We compare with various baselines on these two MedVQA datasets, namely, MEVF-BAN [25], CPRD-BAN [22], M3AE [6], PMC-CLIP [20].”
MethodMedical visual question answering: A survey
Lin, Z., Zhang, D., Tao, Q., Shi, D., Haffari, G., Wu, Q., He, M., & Ge, Z. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102611 · Cited by 186 · cited 2× · see citing works
“The field of MedVQA has gained significant interest in recent years, with a growing number of studies [21].”
BackgroundMulti-modal Masked Autoencoders for Medical Vision-and-Language Pre-training
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_65 · Cited by 160 · cited 3× · see citing works
“1 M3AE [6] ROCO [29], MedICaT [33] 67.”
MethodSilkie: Preference Distillation for Large Visual Language ModelsPreprint
Li, L., Xie, Z.-H., Li, M., Chen, S.-N., Wang, P., Chen, L., Yang, Y.-Z., Wang, B.-Y., & Kong, L.-P. (2023) · arXiv.org · doi:10.48550/arxiv.2312.10665 · Cited by 139 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundLiu, B., Zhan, L.-M., & Wu, X.-M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_20 · Cited by 120 · see citing works
Med-Flamingo: a Multimodal Medical Few-shot LearnerPreprint
Moor, M., Huang, Q., Wu, S., Yasunaga, M., Zakka, C., Dalmia, Y., Reis, E. P., Rajpurkar, P., & Leskovec, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.15189 · Cited by 111 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundZhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R. P. N., Wei, M., Valluri, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.00915 · Cited by 102 · see citing works
Self-citationMedICaT: A Dataset of Medical Images, Captions, and Textual References
Subramanian, S., Wang, L. L., Bogin, B., Mehta, S., van Zuylen, M., Parasa, S., Singh, S., Gardner, M., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.191 · Cited by 94 · see citing works
Park, J., Oh, K., Han, K., & Lee, Y. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-63824-z · Cited by 94 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundOmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
Hu, Y., Li, T., Lu, Q., Shao, W., He, J., Qiao, Y., & Luo, P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02093 · Cited by 85 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language ModelsPreprint
Wang, S., Zhao, Z., Ouyang, X., Wang, Q., & Shen, D. G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.07257 · Cited by 74 · cited 1× · see citing works
“Additionally, acknowledging that the encoder-based language models lack casual masking, we reform the generation task as a mask language modeling task, i.e. , the question input is padded with several ‘[MASK]’ token and the decoder module learns to generate the prediction for the masked token.”
Self-citationSeyfioğlu, M. S., Ikezogwo, W. O., Ghezloo, F., Krishna, R., & Shapiro, L. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01252 · Cited by 57 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundTowards Visual Question Answering on Pathology Images
He, X., Cai, Z., Wei, W., Zhang, Y., Mou, L., Xing, E. P., & Xie, P. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-short.90 · Cited by 55 · cited 2× · see citing works
“Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix ® database 0.3k 3.5k PathVQA [12] Pathology PEIR Digital Library [14] 5k 32.8k SLAKE [23] Radiology MSD [3], ChestX-ray8 [36], CHAOS [15] 0.7k 14k VQA-Med-2021 [5] Radiology MedPix ® database 5k 5k PMC-VQA Mixture * PubMed Central ® 149k 227k Figure 1: The…”
Self-citationPMC-LLaMA: Towards Building Open-source Language Models for MedicinePreprint
Wu, C., Lin, W., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14454 · Cited by 48 · cited 2× · see citing works
“Notably, while all the following models can be used in our architecture, by default, we use the “PMC-LLaMA” (or “PMC-LLaMA-ENC”) and “PMC-CLIP” as backbones, since they are known to be more suitable for medical data according to previous works.”
BackgroundMethodSelf-citationThe impact of ChatGPT and LLMs on medical imaging stakeholders: Perspectives and use cases
Yang, J., Li, H. B., & Wei, D. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100007 · Cited by 46 · cited 1× · see citing works
“Existing MedVQA methods [25, 22, 6, 20] typically treat the problem as a retrieval task with a limited answer base and train multi-modal vision-language models with contrastive or classification objectives.”
MethodTowards Visual Dialog for Radiology
Kovaleva, O., Shivade, C., Kashyap, S., Kanjaria, K., Wu, J., Ballah, D., Coy, A., Karargyris, A., Guo, Y., Beymer, D. B., et al. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.6 · Cited by 38 · cited 2× · see citing works
“Existing publicly available MedVQA datasets have limitations on diversity, or dataset scale, for example, RadVisDial [16] only contains samples on chest x-ray images, VQA-Med [5], VQA-RAD [18], and SLAKE [23] have less than 10K images.”
Survey of Multimodal Medical Question Answering
Demirhan, H., & Zadrozny, W. (2023) · BioMedInformatics · doi:10.3390/biomedinformatics4010004 · Cited by 32 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundA Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray InterpretationPreprint
Chen, Z., Varma, M., Xu, J., Paschali, M., Veen, D. V., Johnston, A., Youssef, A., Blankemeier, L., Blüthgen, C., Altmayer, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.12208 · Cited by 26 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundHallucination Benchmark in Medical Visual Question AnsweringPreprint
Wu, J., Kim, Y., & Wu, H.-H. (2024) · Tiny Papers @ ICLR · doi:10.48550/arxiv.2401.05827 · Cited by 25 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundUniDCP: Unifying Multiple Medical Vision-Language Tasks via Dynamic Cross-Modal Learnable Prompts
Zhan, C., Zhang, Y., Lin, Y., Wang, G., & Wang, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3397191 · Cited by 25 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundSelf-citationA Self-Adaptive Discriminative Autoencoder for Medical Applications
Ge, X., Qu, Y., Shang, C., Yang, L., & Shen, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3195727 · Cited by 17 · see citing works
Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General HealthcarePreprint
Liu, J., Wang, Z., Ye, Q., Chong, D., Zhou, P., & Hua, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17956 · Cited by 16 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundMathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2310.02255 · Cited by 12 · cited 1× · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundCheng, J., Kuang, H., Zhao, Q., Wang, Y., Xu, L., Liu, J., & Wang, J. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.04.025 · Cited by 8 · see citing works
Self-citationSeyfioğlu, M. S., Ikezogwo, W. O., Ghezloo, F., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.04746 · Cited by 2 · see citing works
OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMPreprint
Hu, Y., Li, T., Lu, Q., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.09181 · Cited by 2 · see citing works
UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable PromptsPreprint
Zhan, C., Zhang, Y., Yu, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.11171 · Cited by 1 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
Research works printed but not linked: 8
- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
- Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina Demner-Fushman, and Henning Müller. Vqa- med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019, 2019.
- Asma Ben Abacha, Mourad Sarrouti, Dina Demner-Fushman, Sadid A Hasan, and Henning Müller. Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain. In Proceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum-working notes. 21-24 September 2021, 2021.
- Xiaolan Chen, Pusheng Xu, Yao Li, Weiyi Zhang, Fan Song, Ying-Feng Zheng, Danli Shi, and Mingguang He. Chatffa: Interactive visual question answering on fundus fluorescein angiography image using chatgpt. Available at SSRN 4578568.
- Jianwei Feng and Dong Huang. Optimal gradient checkpoint search for arbitrary computation graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11433-11442, 2021.
- Kristopher N Jones, Dwain E Woode, Kristina Panizzi, and Peter G Anderson. Peir digital library: Online resources and authoring system. In Proceedings of the AMIA Symposium, page 1075. American Medical Informatics Association, 2001.
- Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
- Aaron Nicolson, Jason Dowling, and Bevan Koopman. A concise model for medical image captioning. In CLEF (Working Notes), pages 1611-1619, 2023.
Cited by
Sorted by citations
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
A generalist vision-language foundation model for diverse biomedical tasks
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333 · see citing works
A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications
Jiao, T., Guo, C., Feng, X., Chen, Y., & Song, J. (2024) · Computers, Materials & Continua · doi:10.32604/cmc.2024.053204 · Cited by 158 · see citing works
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 133 · see citing works
Self-citationCLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 175 · see citing works
“Hence, Zhang et al. (2023e) leveraged CLIP’s image encoder and text encoder for question and image understanding, respectively, with a subsequent language decoder for answer generation.”
Self-citationA comprehensive evaluation of large Language models on benchmark biomedical text processing tasks
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108189 · Cited by 116 · see citing works
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 100 · see citing works
“…questions on realistic images captured by visually impaired people Medical VQA VQA-Med [97] 2018 Open-ended QA on radiology images PMC-VQA [162] 2023 MCQ-based questions on a mixture of modalities b Image Set IS-VQA [2] 2021 Open-ended QA on indoor and outdoor scenes Change Detection…”
BackgroundMethodA comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 130 · see citing works
“PubMedVision-Alignment [117] Instruction-Following Data 647,031 instruction-following data.”
InfluentialMultimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 106 · see citing works
“5B) [50,62].”
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98 · see citing works
ChatCAD+: Toward a Universal and Reliable Interactive CAD Using LLMs
Zhao, Z., Wang, S., Gu, J., Zhu, Y., Mei, L., Zhuang, Z., Cui, Z., Wang, Q., & Shen, D. G. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3398350 · Cited by 83 · see citing works
“For instance, RadFM [57] and PMC-VQA [56] are trained based on Flamingo and BLIP-2, respectively.”
MethodSelf-citationRULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models
Xia, P., Zhu, K., Li, H., Zhu, H., Li, Y., Li, G., Zhang, L., & Yao, H. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.62 · Cited by 67 · see citing works
ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset
Rückert, J., Bloch, L., Brüngel, R., Idrissi-Yaghir, A., Schäfer, H., Schmidt, C. S., Koitka, S., Pelka, O., Abacha, A. B., de Herrera, A. G. S., et al. (2024) · Scientific Data · doi:10.1038/s41597-024-03496-6 · Cited by 75 · see citing works
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.510 · Cited by 54 · see citing works
Self-citationThe role of large language models in agriculture: harvesting the future with LLM intelligence
Shaikh, T. A., Rasool, T., Veningston, K., & Yaseen, S. M. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00359-4 · Cited by 61 · see citing works
RegionGPT: Towards Region Understanding Vision Language Model
Guo, Q., Mello, S. D., Yin, H., Byeon, W., Cheung, K. C., Yu, Y., Luo, P., & Liu, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01309 · Cited by 54 · see citing works
“Some other interesting works have made remarkable progress by extending LLM to audio [7, 20], medical VQA [31, 57] and control systems [16, 32].”
BackgroundInterpretable medical image Visual Question Answering via multi-modal relationship graph learning
Hu, X., Gu, L., Kobayashi, K., Liu, L., Zhang, M., Harada, T., Summers, R. M., & Zhu, Y. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103279 · Cited by 48 · see citing works
Advancing medical imaging with language models: featuring a spotlight on ChatGPT
Hu, M., Qian, J. Y., Pan, S., Li, Y., Qiu, R. L. J., & Yang, X. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad387d · Cited by 74 · see citing works
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 50 · see citing works
“…[85] WenLan, [86] RedCaps, [87] WIT, [88] Medical Model Architecture PMC-CLIP, [93] MedBLIP, [89] LLaVA-Med, [62] Med-Flamingo, [90] PMC-VQA, [91] MUMC, [92] PMC-CLIP, [93] PMC-LLaMA, [94] MedAlpaca, [62] Meditron, [95] Taiyi, [290] BioMedGPT, [97] Data CheXpert, [98] ROCO, [99] PMC-OA, [93]…”
Towards Generalist Foundation Model for RadiologyPreprint
Xie, W., Wu, C., Zhang, X., Zhang, Y., & Wang, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3324530/v1 · Cited by 33 · see citing works
Self-citationKI-MAG: A knowledge-infused abstractive question answering system in medical domain
Zafar, A., Sahoo, S. K., Bhardawaj, H., Das, A., & Ekbal, A. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127141 · Cited by 33 · see citing works
Lameesa, A., Silpasuwanchai, C., & Alam, M. S. B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128730 · Cited by 28 · see citing works
Large language model for medical images: a survey of taxonomy, systematic review, and future trends
Wang, P., Lu, W., Lu, C., Zhou, R., Li, M., & Qin, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020090 · Cited by 21 · see citing works
Chen, X., Xu, P., Li, Y., Zhang, W., Song, F., He, M., & Shi, D. (2024) · iScience · doi:10.1016/j.isci.2024.110021 · Cited by 20 · see citing works
“11,12 Our study extends this approach to ophthalmic imaging.”
UniDCP: Unifying Multiple Medical Vision-Language Tasks via Dynamic Cross-Modal Learnable Prompts
Zhan, C., Zhang, Y., Lin, Y., Wang, G., & Wang, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3397191 · Cited by 25 · see citing works
“…ablation study, it is evident that even without specialized pre-training on medical text datasets like PubMed abstracts (PubMed) [44] and PubMed Central full-text articles (PMC) [38], our model can achieve remarkable performance compared with the SOTA methods like BioELECTRA [66] and SciFive [67].”
MethodInfluentialSelf-citationKnowledge graph based reasoning in medical image analysis: A scoping review
Huang, Q., & Li, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109100 · Cited by 19 · see citing works
Wu, Y., Lu, Y., Zhou, Y., Ding, Y., Liu, J., & Ruan, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129999 · Cited by 18 · see citing works
MiniMedGPT: Efficient Large Vision–Language Model for medical Visual Question Answering
Alsabbagh, A. R., Mansour, T., Al-Kharabsheh, M., Ebdah, A. S., Al-Emaryeen, R., Al-Nahhas, S., Mahafza, W. S., & Al-Kadi, O. S. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.01.001 · Cited by 17 · see citing works
Healthcare: A Growing Role for Large Language Models and Generative AI
Pahune, S. A., & Rewatkar, N. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.55573 · Cited by 17 · see citing works
Beyond Text: Frozen Large Language Models in Visual Signal Comprehension
Zhu, L., Wei, F., & Lu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02554 · Cited by 17 · see citing works
Huang, X., Wang, X., Zhang, H., Zhu, Y., Xi, J., An, J., Wang, H., Liang, H., & Pan, C. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i4.32396 · Cited by 14 · see citing works
MEDITRON: Open Medical Foundation Models Adapted for Clinical PracticePreprint
Chen, Z., Romanou, A., Bonnet, A., Hernández-Cano, A., AlKhamissi, B., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4139743/v1 · Cited by 11 · see citing works
HRVDA: High-Resolution Visual Document Assistant
Liu, C., Yin, K., Cao, H., Jiang, X., Li, X., Liu, Y., Jiang, D., Sun, X. W., & Xu, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01471 · Cited by 14 · see citing works
“The experimental results are provided in the Appendix A. Projection-based methods involve directly mapping visual tokens with the LLM’s input space [24, 39, 44, 58, 65, 75].”
MethodRaTEScore: A Metric for Radiology Report GenerationPreprint
Zhao, W., Wu, C., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2024) · medRxiv · doi:10.1101/2024.06.24.24309405 · Cited by 19 · see citing works
Self-citationExplainable medical visual question answering via chain of evidence
Qiu, C., Huang, K., Xie, Z., Liu, M., Gu, J., & Zong, X. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113672 · Cited by 8 · see citing works
Gautam, S., Riegler, M. A., & Halvorsen, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-08009-7_6 · Cited by 9 · see citing works
Structure Causal Models and LLMs Integration in Medical Visual Question Answering
Xu, Z., Li, Q., Nie, W., Wang, W., & Liu, A.-A. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3564320 · Cited by 11 · see citing works
“In addition, Zhang et al. [9] collect an extensive data set and design a projection module to align visual and textual attributes.”
InfluentialHe, Z., Wong, A. N. N., & Yoo, J. S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110625 · Cited by 7 · see citing works
Knowledge fusion in deep learning-based medical vision-language models: A review
Xu, D., Chen, Y., Chai, Z., Xiao, Y., Yan, Y., Ding, W., Wang, H., Jin, Z. J., Jiao, W., Yue, W., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103455 · Cited by 6 · see citing works
A Review on VQA: Methods, Tools and Datasets
Agrawal, M., Jalal, A. S., & Sharma, H. (2023) · doi:10.1109/cset58993.2023.10346815 · Cited by 6 · see citing works
“The authors of this study [31] have suggested MedVInT, a generative model designed to progress this important medical endeavour.”
BackgroundChatFFA: Interactive Visual Question Answering on Fundus Fluorescein Angiography Image Using ChatGPTPreprint
Chen, X., Xu, P., Li, Y., Zhang, W., Song, F., Zheng, Y., Shi, D., & He, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4578568 · Cited by 6 · see citing works
XAI-based Data Visualization in Multimodal Medical DataPreprint
Sharma, S., Singh, M., Mcdaid, L. J., & Bhattacharyya, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.11.664302 · Cited by 6 · see citing works
PathVLM-Eval: Evaluation of open vision language models in histopathology
Gilal, N. U., Zegour, R., Al-Thelaya, K., Özer, E., Agus, M., Schneider, J., & Boughorbel, S. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100455 · Cited by 7 · see citing works
Hùng, B. T., & Duy, H. V. H. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110439 · Cited by 5 · see citing works
SigPhi-Med: A lightweight vision-language assistant for biomedicine
Zhou, F., Liu, X., Zeng, Q., Li, Z., & Xiao, H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104849 · Cited by 6 · see citing works
Domain Adaptation and Generalization Using Foundation Models in Healthcare ImagingPreprint
Ying, H., Lia, Y., & Fu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5345726 · Cited by 5 · see citing works
Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives
Queiroz, D., Carlos, A., Anjos, A., & Berton, L. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3793542 · Cited by 8 · see citing works
Interpretable Pulmonary Disease Diagnosis with Graph Neural Network and Counterfactual Explanations
Li, J., Chen, Y., Wang, Y., Ye, Y., Sun, M., Ren, H., Cheng, W., & Zhang, H. (2023) · International Conference on Sensing, Measurement, Communication and Internet of Things Technologies (SMC-IoT) · doi:10.1109/smc-iot62253.2023.00034 · Cited by 3 · see citing works
“Medical Vision-Language Models (VLM) such as Visual-Med-Alpaca, XrayGLM [24] and PMC-VQA [25] have gained the ability to understand chest X-ray images by training on large-scale chest X-ray images and corresponding reports.”
BackgroundSelf-citationFine-tune language models as multi-modal differential equation solvers
Yang, L., Liu, S., & Osher, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107455 · Cited by 12 · see citing works
“Additionally, the expressivity and training complexity could be limited in these cases.”
Liu, G., Li, H., He, Z., & Zhong, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446378 · Cited by 2 · see citing works
Priority-Encoder Ensemble for Speech Recognition
Ivan, D. F., Darlan, D., Adedigba, A. P., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3454221 · Cited by 1 · see citing works
End-to-End 3D point cloud question answering via state space models
Qorsham, H., El-Deeb, R., & Essa, E. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113404 · Cited by 1 · see citing works
Multimodal LLM with Hierarchical Mixture-of-Experts for VQA on 3D Brain MRI
Vepa, A., Yu, Y., Gan, J., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818865 · Cited by 1 · see citing works
DIEM: Decomposition-Integration Enhancing Multimodal Insights
Jiang, X., Wang, G., Guo, J., Li, J., Zhang, W., Lu, R., & Tang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02578 · Cited by 1 · see citing works
“By integrating textual and visual modalities, MLLMs can transcend the limitations inherent in textual information [7, 14, 44, 50].”
BackgroundA Survey of Generalization and Adaptation in Medical Imaging Foundation ModelsPreprint
Ying, H., Lia, Y., & Fu, Z. (2025) · Preprints.org · doi:10.20944/preprints202507.0942.v1 · Cited by 2 · see citing works
The Illusion of Readiness in Health AI
Gu, Y., Fu, J., Liu, X., Valanarasu, J. M. J., Codella, N. C., Tan, R., Liu, Q., Jin, Y., Zhang, S., Wang, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.18234 · Cited by 1 · see citing works
A Comprehensive Review and Open Challenges on Visual Question Answering Models
Shaik, F. A., Koshti, D., Gupta, A., Kalla, M., & Sharma, A. (2023) · Aibi revista de investigación administración e ingeniería · doi:10.15649/2346030x.3370
Cross-Modal Multi-Source Public Data Fusion and Retrieval using Knowledge Distillation Method
Huang, Z., Li, S., Yang, J., et al. (2023) · International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc59590.2023.10507453
“Our approach centers around the translation and adaptation of the PMC-VQA dataset [13], a well-established corpus in the medical visual question-answering domain.”
MethodSelf-citationFine Tuning Idefic 9b With LORA for Multimodal Medical VQA
Punneshetty, S., Ashok, S., Niranjanamurthy, M., & Murthy, S. V. N. (2024) · International Conference on Knowledge Engineering and Communication Systems (ICKECS) · doi:10.1109/ickecs61492.2024.10616779
Deep learning-based visual question answering: Survey
Mohammed, N. F., & Ali, I. H. (2025) · AIP conference proceedings · doi:10.1063/5.0258482
Kumar, H., Deepak, G., & Santhanavijayan, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-88992-9_16
Redefining medical visual question answering using conditional generative diffusion models
Liu, B., Liu, L., Yang, X., Peng, W., & Liu, L. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108222
Enhancing Clinical Reasoning in Medical Vision-Language Model through Structured PromptsPreprint
Prakash, K. D., Kim, K., & Han, Y. (2025) · medRxiv · doi:10.1101/2025.07.31.25331901
“We utilize the HuatuoGPT-Vision dataset [28], a unified benchmark combining over 17,000 multiple-choice VQA samples from sources such as PMC-VQA [29], SLAKE [30], VQA-RAD [19], OmniMedVQA [31], and PathVQA PREPRINT [20].”
Alotaibi, S., Al-Hadhrami, S. A., & Al-Ahmadi, S. A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1646176
CARR: Cross-Modal Adaptation with Reverse Reasoning for Medical Visual Question Answering
Yan, W., Zhu, J., Zhang, L., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3378-6_15
- Authors
- Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.10415
- OpenAlex
- W4377121462
- Cited by
- 66
- References
- 57
- Citations by year
- 2026: 3 · 2025: 24 · 2024: 31 · 2023: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- medical visual question answering, visual instruction tuning, generative question answering, medical image interpretation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
