CoChat

Vision-language models for medical report generation and visual question answering: a review

Open access
Hartsock, Iryna, Rasool, Ghulam · 2024 · Frontiers in Artificial Intelligence
Cited by 229

The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.

Abstract

Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on publicly available models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs, with visual and language data often fused using Transformer-based architectures to enable effective learning from multimodal data. Key areas we address include the exploration of 18 public medical vision-language datasets, in-depth analyses of the architectures and pre-training strategies of 16 recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges facing medical VLM development, including limited data availability, concerns with data privacy, and lack of proper evaluation metrics, among others, while also proposing future directions to address these obstacles. Overall, our review summarizes the recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/frai.2024.1430984

Cites

93 of 93 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 110

  1. Abacha, 2020, “Overview of the VQA-Med task at ImageCLEF 2020: visual question answering and generation in the medical domain,”
  2. Abacha, 2019, “VQA-Med: overview of the medical visual question answering task at imageclef 2019,”
  3. Allan, 2020, 2018 robotic scene segmentation challenge, arXiv Preprint arXiv:2001.11190, 10.48550/arXiv.2001.11190 · DOI
  4. Allan, 2019, 2017 robotic instrument segmentation challenge, arXiv Preprint arXiv:1902.06426, 10.48550/arXiv.1902.06426 · DOI
  5. Bai, “CAT-ViL: co-attention gated vision-language embedding for visual question localized-answering in robotic surgery,”, 397
  6. Bai, 2018, A survey on automatic image caption generation, Neurocomputing, 311, 80, 10.1016/j.neucom.2018.05.080 · DOI · Paper page
  7. Baldi, 2021, Deep Learning in Science, 10.1017/9781108955652 · DOI
  8. Banerjee, 2005, “METEOR: an automatic metric for MT evaluation with improved correlation with human judgments,”, ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 65
  9. Barhoumi, 2023, Efficient scopeformer: toward scalable and rich feature extraction for intracranial hemorrhage detection, IEEE Access, 11, 81656, 10.48550/arXiv.2302.00220 · arXiv · doi:10.48550/arxiv.2302.00220
  10. Bazi, 2023, Vision—language model for visual question answering in medical imagery, Bioengineering, 10, 380, 10.3390/bioengineering10030380 · DOI · Paper page
  11. Beam, 2020, Clinical concept embeddings learned from massive sources of multimodal medical data, Pacif. Symp. Biocomput, 25, 295, 10.1142/9789811215636_0027 · DOI · Paper page
  12. Ben Abacha, 2019, “Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering,”, BioNLP Workshop and Shared Task, 370, 10.18653/v1/W19-5039 · DOI · Paper page
  13. Bigolin Lanfredi, 2022, Reflacx, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays, Sci. Data, 9, 1441, 10.1038/s41597-022-01441-z · DOI
  14. Boecking, 2022, Making the most of text semantics to improve biomedical vision–language processing, Comput. Vis, 5, 1, 10.1007/978-3-031-20059-5 · doi:10.1007/978-3-031-20059-5
  15. Cai, 2024, Dynamic transformer architecture for continual learning of multimodal tasks, arXiv Preprint arXiv:2401.15275, 10.48550/arXiv.2401.15275 · DOI
  16. Chen, 2023, VLP: a survey on vision-language pre-training, Machine Intell. Res, 20, 38, 10.1007/s11633-022-1369-5 · DOI · Paper page
  17. Chen, A simple framework for contrastive learning of visual representations, arXiv Preprint arXiv:2002.05709, 10.48550/arXiv.2002.05709 · DOI · Paper page
  18. Cho, 2021, “Unifying vision-and-language tasks via text generation,”, 1931
  19. Clunie, 2024, Summary of the National Cancer Institute 2023 virtual workshop on medical image de-identification—part 2: pathology whole slide image de-identification, de-facing, the role of AI in image de-identification, and the NCI MIDI datasets and pipeline, J. Imag. Informat. Med, 24, 1183, 10.1007/s10278-024-01183-x · DOI
  20. Dai, 2023, InstructBLIP: towards general-purpose vision-language models with instruction tuning, arXiv Preprint arXiv:2305.06500, 10.48550/arXiv.2305.06500 · arXiv · doi:10.48550/arxiv.2305.06500 · Paper page
  21. Dosovitskiy, 2021, “An image is worth 16x16 words: transformers for image recognition at scale,”
  22. Dou, 2022, “An empirical study of training end-to-end vision-and-language transformers,”, 18145
  23. Dwork, 2006, “Differential privacy,”, 1, 10.1007/11787006_1 · DOI · Paper page
  24. Eslami, 2023, PubmedCLIP: how much does clip benefit visual question answering in the medical domain?, Find. Assoc. Comput. Linguist, 88, 1181, 10.18653/v1/2023.findings-eacl.88 · DOI · Paper page
  25. Gan, 2022, Vision-language pre-training: basics, recent advances, and future trends, arXiv Preprint arXiv:2210.09263, 10.48550/arXiv.2210.09263 · DOI
  26. Goodfellow, 2016, Deep Learning
  27. Han, 2023, MedAlpaca—an open-source collection of medical conversational AI models and training data, arXiv Preprint arXiv:2304.08247, 10.48550/arXiv.2304.08247 · arXiv · doi:10.48550/arxiv.2304.08247 · Paper page
  28. Hao, 2020, Probabilistic predictions of people perusing: evaluating metrics of language model performance for psycholinguistic modeling, arXiv Preprint arXiv:2009.03954, 10.18653/v1/2020.cmcl-1.10 · DOI
  29. He, SA-Attack: improving adversarial transferability of vision-language pre-training models via self-augmentation, arXiv Preprint arXiv:2312.04913, 10.48550/arXiv.2312.04913 · DOI
  30. Hu, 2022, “LoRA: low-rank adaptation of large language models,”
  31. Huang, 2022, Convolutional networks with dense connectivity, IEEE Trans. Pat. Anal. Machine Intell, 44, 8704, 10.1109/TPAMI.2019.2918284 · DOI · Paper page
  32. Huang, 2021, “What makes multimodal learning better than single (provably),”
  33. Ionescu, 2021, Overview of the imageclef 2021: multimedia retrieval in medical, nature, internet and social media applications, Exp. IR Meets Multilingual. Multimodal. Interact, 23, 345, 10.1007/978-3-030-85251-1_23 · DOI
  34. Irvin, 2019, CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison, AAAI Conf. Artif. Intell, 33, 590, 10.1609/aaai.v33i01.3301590 · DOI · Paper page
  35. Jeong, 2023, Multimodal image-text matching improves retrieval-based chest x-ray report generation, arXiv Preprint arXiv:2303.17579, 10.48550/arXiv.2303.17579 · DOI
  36. Ji, 2020, 5—computer vision applications, Comput. Vis. Pat. Recogn, 10, 191, 10.1016/B978-0-12-803467-5.00010-1 · DOI
  37. Khan, 2023, “The importance of robust features in mitigating catastrophic forgetting,”, 752, 10.1109/ISCC58397.2023.10218203 · DOI
  38. Khan, 2024, Brain-inspired continual learning: robust feature distillation and re-consolidation for class incremental learning, IEEE Access, 2024, 14588, 10.48550/arXiv.2404.14588 · arXiv · doi:10.48550/arxiv.2404.14588
  39. Kim, 2018, Bilinear attention networks, Adv. Neural Inform. Process. Syst, 31, 1564, 10.48550/arXiv.1805.07932 · DOI · Paper page
  40. Kingma, 2014, “ADAM: a method for stochastic optimization,”
  41. Koutsoubis, Future-Proofing Medical Imaging with Privacy-Preserving Federated Learning and Uncertainty Quantification: A Review
  42. Koutsoubis, Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation
  43. Kwon, 2023, Masked vision and language modeling for multi-modal representation learning, arXiv Preprint arXiv:2208.02131, 10.48550/arXiv.2208.02131 · DOI
  44. Lau, 2018, A dataset of clinically generated visual questions and answers about radiology images, Sci. Data, 5, 180251, 10.1038/sdata.2018.251 · DOI · Paper page
  45. Li, 2022, “Cross-modal clinical graph transformer for ophthalmic report generation,”, 20624
  46. Li, “Masked vision and language pre-training with unimodal and multimodal contrastive losses for medical visual question answering,”, Medical Image Computing and Computer Assisted Intervention (MICCAI), 374
  47. Lin, 2004, “ROUGE: a package for automatic evaluation of summaries,”, 74
  48. Lin, PMC-CLIP: contrastive language-image pre-training using biomedical documents, arXiv Preprint arXiv:2303.07240, 10.48550/arXiv.2303.07240 · DOI
  49. Lin, Medical visual question answering: a survey, Artif. Intell. Med, 143, 102611, 10.1016/j.artmed.2023.102611 · DOI · Paper page
  50. Liu, SLAKE: a semantically-labeled knowledge-enhanced dataset for medical visual question answering, IEEE 18th International Symposium on Biomedical Imaging (ISBI), 1650
  51. Liu, Medical visual question answering via conditional reasoning and contrastive learning, IEEE Trans. Med. Imag, 42, 1532, 10.1109/TMI.2022.3232411 · DOI
  52. Liu, A systematic review of deep learning-based research on radiology report generation, arXiv Preprint arXiv:2311.14199, 10.48550/arXiv.2311.14199 · DOI
  53. Liu, 2022, DePlot: one-shot visual language reasoning by plot-to-table translation, arXiv Preprint arXiv:2212.10505, 10.48550/arXiv.2212.10505 · DOI
  54. Liu, “Swin transformer: hierarchical vision transformer using shifted windows,”, 9992
  55. Mabotuwana, 2020, Framework for extracting critical findings in radiology reports, J. Digit. Imag, 33, 988, 10.1007/s10278-020-00349-7 · doi:10.1007/s10278-020-00349-7
  56. Manzari, 2023, MedViT: a robust vision transformer for generalized medical image classification, Comput. Biol. Med, 157, 106791, 10.1016/j.compbiomed.2023.106791 · DOI · Paper page
  57. Masci, 2011, “Stacked convolutional auto-encoders for hierarchical feature extraction,”, 52
  58. Mishra, 2021, “VT-ADL: a vision transformer network for image anomaly detection and localization,”, 01
  59. Mohsan, 2023, Vision transformer and language model based radiology report generation, IEEE Access, 11, 1814, 10.1109/ACCESS.2022.3232719 · DOI
  60. Monshi, 2020, Deep learning in generating radiology reports: a survey, Artif. Intell. Med, 106, 101878, 10.1016/j.artmed.2020.101878 · DOI · Paper page
  61. Moon, 2022, Multi-modal understanding and generation for medical images and text via vision-language pre-training, IEEE J. Biomed. Health Informat, 26, 6070, 10.1109/JBHI.2022.3207502 · DOI · Paper page
  62. Moor, 2023, Med-Flamingo: a multimodal medical few-shot learner, arXiv Preprint arXiv:2307.15189, 10.48550/arXiv.2307.15189 · arXiv · doi:10.48550/arxiv.2307.15189 · Paper page
  63. Nadkarni, 2011, Natural language processing: an introduction, J. Am. Med. Informat. Assoc, 18, 544, 10.1136/amiajnl-2011-000464 · DOI · Paper page
  64. Norgeot, 2020, Protected health information filter (philter): accurately and securely de-identifying free-text clinical notes, NPJ Digit. Med, 3, 258, 10.1038/s41746-020-0258-y · DOI · Paper page
  65. Papineni, 2002, “BLEU: a method for automatic evaluation of machine translation,”, Annual Meeting of the Association for Computational Linguistics, 311
  66. Pelka, 2018, Radiology objects in context (ROCO): a multimodal image dataset, Intravasc. Imag. Comput. Assist. Stent. Large-Scale Annot. Biomed. Data Expert Label Synth, 11043, 180, 10.1007/978-3-030-01364-6_20 · DOI · Paper page
  67. Pellegrini, 2023, Radialog: a large vision-language model for radiology report generation and conversational assistance, arXiv Preprint arXiv:2311.18681, 10.48550/arXiv.2311.18681 · DOI
  68. Pennington, 2014, Glove: global vectors for word representation, Empir. Methods Natur. Lang. Process, 14, 1532, 10.3115/v1/D14-1162 · DOI · Paper page
  69. Rai, 2021, Study of various methods for tokenization, Appl. Internet Things, 18, 193, 10.1007/978-981-15-6198-6_18 · DOI
  70. Ramesh, 2022, Improving radiology report generation systems by removing hallucinated references to non-existent priors, Machine Learn. Res, 193, 456, 10.48550/arXiv.2210.06340 · DOI · Paper page
  71. Ranftl, 2021, “Vision transformers for dense prediction,”, 12159
  72. Ranjit, 2023, Retrieval augmented chest X-ray report generation using openAI GPT models, arXiv Preprint arXiv:2305.03660, 10.48550/arXiv.2305.03660 · DOI
  73. Ren, 2024, Learning or self-aligning? rethinking instruction fine-tuning, arXiv Preprint arXiv:2402.18243, 10.48550/arXiv.2402.18243 · DOI
  74. Romanov, 2018, “Lessons from natural language inference in the clinical domain,”, 1586
  75. Rückert, 2022, “Overview of imageclefmedical 2022—caption prediction and concept detection,”, CEUR Workshop Proceedings, Vol. 3180, 1294
  76. Schmidt, 2019, Recurrent neural networks (RNNS): a gentle introduction and overview, arXiv Preprint arXiv:1912.05911, 10.48550/arXiv.1912.05911 · DOI
  77. Seenivasan, 2022, “Surgical-VQA: visual question answering in surgical scenes using transformer,”, Medical Image Computing and Computer Assisted Intervention—MICCAI, 33
  78. Sengupta, 2023, Automatic report generation for histopathology images using pre-trained vision transformers and BERT, arXiv Preprint arXiv:2312.01435, 10.48550/arXiv.2312.01435 · arXiv · doi:10.48550/arxiv.2312.01435
  79. Sharma, 2023, Evolution of visual data captioning methods, datasets, and evaluation metrics: a comprehensive survey, Expert Syst. Appl, 221, 119773, 10.1016/j.eswa.2023.119773 · DOI
  80. Shrestha, 2023, Medical vision language pretraining: a survey, arXiv Preprint arXiv:2312.06224, 10.48550/arXiv.2312.06224 · DOI
  81. Soviany, 2021, Curriculum learning: a survey, Int. J. Comput. Vis, 130, 1526, 10.1007/s11263-022-01611-x · DOI · Paper page
  82. Stripelis, 2021, Secure neuroimaging analysis using federated learning with homomorphic encryption, SPIE Med. Imag, 22, 1611, 10.48550/arXiv.2108.03437 · arXiv · doi:10.48550/arxiv.2108.03437
  83. Subramanian, 2020, “MediCAT: a dataset of medical images, captions, and textual references,”, 2112, 10.18653/v1/2020.findings-emnlp.191 · DOI · Paper page
  84. Sun, 2023, Aligning large multimodal models with factually augmented RLHF, arXiv Preprint arXiv:2309.14525, 10.48550/arXiv.2309.14525 · DOI
  85. Tanwani, 2022, “RepsNet: combining vision with language for automated medical reports,”, Medical Image Computing and Computer Assisted Intervention (MICCAI), 714
  86. Thawkar, 2023, XrayGPT: chest radiographs summarization using medical vision-language models, arXiv Preprint arXiv:2306.07971, 10.48550/arXiv.2306.07971 · arXiv · doi:10.48550/arxiv.2306.07971 · Paper page
  87. Ting, 2023, A survey on automatic generation of medical imaging reports based on deep learning, BioMed. Eng. OnL, 22, 1113, 10.1186/s12938-023-01113-y · DOI
  88. Tripathi, Building flexible, scalable, and machine learning-ready multimodal oncology datasets, Sensors, 24, 51634, 10.3390/s24051634 · DOI
  89. Tripathi, HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models
  90. Tyagi, 2021, “Detecting pneumonia using vision transformer and comparing with other techniques,”, International Conference on Electronics, Communication and Aerospace Technology (ICECA), 12
  91. Vaswani, 2017, Attention is all you need, Adv. Neural Inform. Process. Syst, 30, 5998, 10.48550/arXiv.1706.03762 · DOI · Paper page
  92. Verspoor, 2013, Encyclopedia of Systems Biology, Chapter Natural Language Processing, 1495, 10.1007/978-1-4419-9863-7_158 · doi:10.1007/978-1-4419-9863-7_158
  93. Wang, GIT: a generative image-to-text transformer for vision and language, arXiv Preprint arXiv:2205.14100, 10.48550/arXiv.2205.14100 · arXiv · doi:10.48550/arxiv.2205.14100 · Paper page
  94. Wang, MedCLIP: contrastive learning from unpaired medical images and text, arXiv Preprint arXiv:2210.10163, 10.48550/arXiv.2210.10163 · DOI
  95. Wang, “SimVLM: simple visual language model pretraining with weak supervision,”, International Conference on Learning Representations (ICLR)
  96. Waqas, 2023, Revolutionizing digital pathology with the power of generative artificial intelligence and foundation models, Lab. Invest, 103, 100255, 10.1016/j.labinv.2023.100255 · DOI · Paper page
  97. Waqas, Digital pathology and multimodal learning on oncology data, Artif. Intell, 1, 1, 10.1093/bjrai/ubae014 · DOI
  98. Waqas, Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes
  99. Xie, 2016, “Aggregated residual transformations for deep neural networks,”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5987
  100. Xie, 2022, “SimMIM: a simple framework for masked image modeling,”, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9643
  101. Xin, 2022, An improved transformer network for skin cancer classification, Comput. Biol. Med, 149, 105939, 10.1016/j.compbiomed.2022.105939 · DOI · Paper page
  102. Xu, 2021, “Learning domain adaptation with model calibration for surgical report generation in robotic surgery,”, 2021 IEEE International Conference on Robotics and Automation (ICRA), 12350, 10.1109/ICRA48506.2021.9561569 · DOI
  103. Yuan, 2023, RAMM: retrieval-augmented biomedical visual question answering with multi-modal pre-training, arXiv Preprint arXiv:2303.00534, 10.48550/arXiv.2303.00534 · arXiv · doi:10.48550/arxiv.2303.00534
  104. Zeng, 2020, “MedDialog: large-scale medical dialogue datasets,”, Conference on Empirical Methods in Natural Language Processing (EMNLP), 9241, 10.18653/v1/2020.emnlp-main.743 · DOI · Paper page
  105. Zhan, 2020, “Medical visual question answering via conditional reasoning,”, The 28th ACM International Conference on Multimedia, 2345
  106. Zhang, 2018, “Grounding referring expressions in images by variational context,”, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4158
  107. Zhang, Large-scale domain-specific pretraining for biomedical vision-language processing, arXiv Preprint arXiv:2303.00915, 10.48550/arXiv.2303.00915 · DOI · Paper page
  108. Zhang, Adapter learning in pretrained feature extractor for continual learning of diseases, arXiv Preprint arXiv:2304.09042, 10.48550/arXiv.2304.09042 · arXiv · doi:10.48550/arxiv.2304.09042
  109. Zhao, 2023, Retrieving multimodal information for augmented generation: a survey, arXiv Preprint arXiv:2303.10868, 10.48550/arXiv.2303.10868 · DOI
  110. Zhou, Learning without forgetting for vision-language models, arXiv Preprint arXiv:2305.19270, 10.48550/arXiv.2305.19270 · arXiv · doi:10.48550/arxiv.2305.19270

Cited by

See all 229

Metadata

Authors
Hartsock, I., & Rasool, G.
Venue
Frontiers in Artificial Intelligence
Published
2024-11-19
Volume
7
Pages
1430984-1430984
ISSN
2624-8212 (ISSN-L)
OpenAlex
W4404509535
Cited by
229
References
93
FWCI
32.35
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.