CoChat
Sign Up
Cited by 69Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationImage Retrieval and Classification Techniques

A scoping review on multimodal deep learning in biomedical images and texts

Open access
Sun, Zhaoyi, Lin, Mingquan, Zhu, Qingqing, Xie, Qianqian, Wang, Fei, Lu, Zhiyong, Peng, Yifan·2023·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2023.104482PMC PMC10591890
Open this paper in CoChat — free account

Abstract

Objective Computer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers' attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions. Methods In this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research. Result This study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation. Conclusion Our results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2023.104482

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • No review has systematically summarized the methodologies and processes specific to the application of multimodal AI in critical care.

    “However, to the best of our knowledge, no review has systematically summarized the methodologies and processes specific to the application of multimodal AI in critical care.”

    From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [24]

    The Introduction (p. 2) says earlier biomedical reviews lacked detailed discussion of implementing multimodal language models. The Discussion (p. 14) also identifies a lack of systematic methodological summary, but as a limitation of this review. Neither passage establishes that no review has summarized multimodal AI methodologies and processes specifically in critical care. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 109

  1. [4][4] Haoyue Bai, Xiaoyan Shan, Yefan Huang, and Xiaoli Wang. MVQAS: A medical visual question answering system. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , CIKM '21, pages 4675-4679, New York, NY, USA, October 2021. Association for Computing Machinery. -
  2. [5][5] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and / or Summarization , pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. -
  3. [6][6] Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Joey Liu, Dina Demner-Fushman, and Henning Muller. VQA-med: Overview of the medical visual question answering task at ImageCLEF 2019. http://ceur-ws. org/Vol-2380/paper\_272.pdf , 2019. Accessed: 2023-2-9. -
  4. [9][9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  5. [11][11] Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and Akshay Chaudhari. Roentgen: vision-language foundation model for chest x-ray generation. arXiv preprint arXiv:2211.12737 , 2022. - · arXiv
  6. [12][12] Pierre Chambon, Christian Bluethgen, Curtis P Langlotz, and Akshay Chaudhari. Adapting pretrained visionlanguage foundational models to medical imaging domains. arXiv preprint arXiv:2210.04133 , 2022. - · arXiv
  7. [14][14] Weipeng Chen, Haiwei Pan, Kejia Zhang, Xin Du, and Qianna Cui. VMEKNet: Visual memory and external knowledge based network for medical report generation. In PRICAI 2022: Trends in Artificial Intelligence , pages 188-201. Springer Nature Switzerland, 2022. -
  8. [15][15] Yueru Chen, Mozhdeh Rouhsedaghat, Suya You, Raghuveer Rao, and C-C Jay Kuo. Pixelhop ++ : A small Successive-Subspace-Learning-Based (Ssl-Based) model for image classification. In 2020 IEEE International Conference on Image Processing (ICIP) , pages 3294-3298. ieeexplore.ieee.org, October 2020. -
  9. [17][17] Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. Cross-modal memory networks for radiology report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 5904-5914, 2021. -
  10. [18][18] Zhihong Chen, Yuhao Du, Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, and Tsung-Hui Chang. Multi-modal masked autoencoders for medical Vision-and-Language pre-training. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2022 , pages 679-689. Springer Nature Switzerland, 2022. -
  11. [19][19] Zhihong Chen, Guanbin Li, and Xiang Wan. Align, reason and learn: Enhancing medical Vision-and-Language pre-training with knowledge. In Proceedings of the 30th ACM International Conference on Multimedia , MM '22, pages 5152-5161, New York, NY, USA, October 2022. Association for Computing Machinery. -
  12. [20][20] Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, Wei Xie, Gail L Rosen, Benjamin J Lengerich, Johnny Israeli, Jack Lanchantin, Stephen Woloszynek, Anne E Carpenter, Avanti Shrikumar, Jinbo Xu, Evan M Cofer, Christopher A Lavender, Srinivas C Turaga, Amr M Alexandari, Zhiyong Lu, David J Harris, Dave DeCaprio, Yanjun Qi, Anshul Kundaje, Yifan Peng, Laura K Wiley, Marwin H S Segler, Simina M Boca, S Joshua Swamidass, Austin Huang, Anthony Gitter, and Casey S Greene. Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface , 15(141), April 2018. -
  13. [21][21] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014. - · arXiv
  14. [22][22] Hannah P Cowley, Mandy Natter, Karla Gray-Roncal, Rebecca E Rhodes, Erik C Johnson, Nathan Drenkow, Timothy M Shead, Frances S Chance, Brock Wester, and William Gray-Roncal. Author correction: A framework for rigorous evaluation of human performance in human and machine learning comparison studies. Sci. Rep. , 12 (1):11559, July 2022. -
  15. [24][24] Can Cui, Haichun Yang, Yaohong Wang, Shilin Zhao, Zuhayr Asad, Lori A Coburn, Keith T Wilson, Bennett Landman, and Yuankai Huo. Deep multi-modal fusion of image and non-image data in disease diagnosis and prognosis: a review. Progress in Biomedical Engineering , 2023. -
  16. [25][25] Francesco Dalla Serra, William Clackett, Hamish MacKinnon, Chaoyang Wang, Fani Deligianni, Jeff Dalton, and Alison Q O'Neil. Multimodal generation of radiology reports using Knowledge-Grounded extraction of entities and relations. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 615-624, Online only, November 2022. Association for Computational Linguistics. -
  17. [26][26] Zachary A Daniels and Dimitris N Metaxas. Exploiting visual and Report-Based information for chest X-RAY analysis by jointly learning visual classifiers and topic models. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) , pages 1270-1274. ieeexplore.ieee.org, April 2019. -
  18. [27][27] Jean-Benoit Delbrouck, Cassie Zhang, and Daniel Rubin. QIAI at MEDIQA 2021: Multimodal radiology report summarization. In Proceedings of the 20th Workshop on Biomedical Language Processing , pages 285-290, Online, June 2021. Association for Computational Linguistics. -
  19. [28][28] Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis Langlotz. Improving the factual correctness of radiology report generation with semantic rewards. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 4348-4360, 2022. -
  20. [29][29] Jean-Benoit Delbrouck, Khaled Saab, Maya Varma, Sabri Eyuboglu, Pierre Chambon, Jared Dunnmon, Juan Zambrano, Akshay Chaudhari, and Curtis Langlotz. ViLMedic: a framework for research at the intersection of vision and language in medical AI. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations , pages 23-34, Stroudsburg, PA, USA, May 2022. Association for Computational Linguistics. -
  21. [30][30] Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. , 23(2):304-310, March 2016. -
  22. [31][31] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653 / v1 / N19-1423. URL https://aclanthology.org/N19-1423 . -
  23. [32][32] Tuong Do, Binh X Nguyen, Erman Tjiputra, Minh Tran, Quang D Tran, and Anh Nguyen. Multiple metamodel quantifying for medical visual question answering. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2021 , pages 64-74. Springer International Publishing, 2021. -
  24. [33][33] Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y Ng, and Pranav Rajpurkar. Retrieval-Based chest XRay report generation using a pre-trained contrastive Language-Image model. In Subhrajit Roy, Stephen Pfohl, Emma Rocheteau, Girmaw Abebe Tadesse, Luis Oala, Fabian Falck, Yuyin Zhou, Liyue Shen, Ghada Zamzmi, Purity Mugambi, Ayah Zirikly, Matthew B A McDermott, and Emily Alsentzer, editors, Proceedings of Machine Learning for Health , volume 158 of Proceedings of Machine Learning Research , pages 209-219. PMLR, December 2021. -
  25. [34][34] Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023 , pages 1151-1163, 2023. -
  26. [35][35] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1126-1135. PMLR, 2017. -
  27. [36][36] Jevgenij Gamper and Nasir Rajpoot. Multiple instance captioning: Learning representations from histopathology textbooks and articles. In 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16549-16559. IEEE, June 2021. -
  28. [37][37] Haifan Gong, Guanqi Chen, Sishuo Liu, Yizhou Yu, and Guanbin Li. Cross-Modal Self-Attention with MultiTask Pre-Training for medical visual question answering. In Proceedings of the 2021 International Conference on Multimedia Retrieval , ICMR '21, pages 456-460, New York, NY, USA, September 2021. Association for Computing Machinery. -
  29. [38][38] B Hao, H Zhu, and I C Paschalidis. Enhancing clinical bert embedding using a biomedical knowledge base. 28th International Conference on , 2020. -
  30. [39][39] Harishma T Haridas, Mostafa M Fouda, Zubair Md Fadlullah, Mohamed Mahmoud, Basem M ElHalawany, and Mohsen Guizani. Med-gpvs: A deep learning-based joint biomedical image classification and visual question answering system for precision e-health. In ICC 2022-IEEE International Conference on Communications , pages 3838-3843. IEEE, 2022. -
  31. [40][40] Sadid A Hasan, Yuan Ling, Oladimeji Farri, Joey Liu, Henning Muller, and Matthew Lungren. Overview of ImageCLEF 2018 medical domain visual question answering task. https://ceur-ws.org/Vol-2125/paper\_ 212.pdf , 2018. Accessed: 2023-2-9. -
  32. [41][41] Ahmad Hassan, Mehreen Sirshar, Muhammad Usman Akram, and Muhammad Umar Farooq. Analysis of multimodal representation learning across medical images and reports using multiple vision and language pre-trained models. In 2022 19th International Bhurban Conference on Applied Sciences and Technology (IBCAST) , pages 337-342. IEEE, 2022. -
  33. [43][43] Tao He, Jixiang Guo, Nan Chen, Xiuyuan Xu, Zihuai Wang, Kaiyu Fu, Lunxu Liu, and Zhang Yi. MediMLP: Using Grad-CAM to extract crucial variables for lung cancer postoperative complication prediction. IEEE J Biomed Health Inform , 24(6):1762-1771, June 2020. -
  34. [44][44] X He, Y Zhang, L Mou, E Xing, and P Xie. Pathvqa: 30000 + questions for medical visual question answering. arxiv 2020. arXiv preprint arXiv:2003.10286 , 2020. - · arXiv
  35. [45][45] Lars Heiliger, Anjany Sekuboyina, Bjoern Menze, Jan Egger, and Jens Kleesiek. Beyond medical imaging: A review of multimodal deep learning in radiology, February 2022. Accessed: 2023-1-2. -
  36. [46][46] S Hochreiter and J Schmidhuber. Long short-term memory. Neural Comput. , 9(8):1735-1780, November 1997. -
  37. [47][47] Gregory Holste, Savannah C Partridge, Habib Rahbar, Debosmita Biswas, Christoph I Lee, and Adam M Alessio. End-to-end learning of fused image and non-image features for improved breast cancer classification from MRI. In 2021 IEEE / CVF International Conference on Computer Vision Workshops (ICCVW) , pages 3294-3303. IEEE, October 2021. -
  38. [48][48] Tzu-Ming Harry Hsu, Wei-Hung Weng, Willie Boag, Matthew McDermott, and Peter Szolovits. Unsupervised multimodal representation learning across medical images and reports. arXiv preprint arXiv:1811.08615 , 2018. - · arXiv
  39. [49][49] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 , 2023. - · arXiv
  40. [51][51] Shih-Cheng Huang, Anuj Pareek, Roham Zamanian, Imon Banerjee, and Matthew P Lungren. Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Sci. Rep. , 10(1):22147, December 2020. -
  41. [53][53] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A Mong, Safwan S Halabi, Jesse K Sandberg, Ricky Jones, David B Larson, Curtis P Langlotz, Bhavik N Patel, Matthew P Lungren, and Andrew Y Ng. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. AAAI , 33(01): 590-597, July 2019. -
  42. [54][54] Grzegorz Jacenk´ ow, Alison Q O'Neil, and Sotirios A Tsaftaris. Indication as prior knowledge for multimodal disease classification in chest radiographs with transformers. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI) , pages 1-5. ieeexplore.ieee.org, March 2022. -
  43. [57][57] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 , 2019. - · arXiv
  44. [58][58] Matthias Keicher, Kamilia Mullakaeva, Tobias Czempiel, Kristina Mach, Ashkan Khakzar, and Nassir Navab. Few-shot structured radiology report generation using natural language prompts. arXiv preprint arXiv:2203.15723 , 2022. - · arXiv
  45. [59][59] Yash Khare, Viraj Bagal, Minesh Mathew, Adithi Devi, U Deva Priyakumar, and C V Jawahar. MMBERT: Multimodal BERT pretraining for improved medical VQA. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) , pages 1033-1036. ieeexplore.ieee.org, April 2021. -
  46. [60][60] Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, pages 1571-1581, Red Hook, NY, USA, 2018. Curran Associates Inc. -
  47. [63][63] Juan S Lara, Victor H Contreras O., Sebasti´ an Ot´ alora, Henning M¨ uller, and Fabio A Gonz´ alez. Multimodal latent semantic alignment for automated prostate tissue classification and retrieval. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2020 , pages 572-581. Springer International Publishing, 2020. -
  48. [65][65] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019. - · arXiv
  49. [66][66] Pengfei Li, Gang Liu, Lin Tan, Jinying Liao, and Shenjun Zhong. Self-supervised vision-language pretraining for medical visual question answering. arXiv preprint arXiv:2211.13594 , 2022. - · arXiv
  50. [67][67] Yikuan Li, Hanyin Wang, and Yuan Luo. A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports. In 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) , pages 1999-2004. ieeexplore.ieee.org, December 2020. -
  51. [68][68] Zihan Li, Yunxiang Li, Qingde Li, Puyang Wang, Dazhou Guo, Le Lu, Dakai Jin, You Zhang, and Qingqi Hong. Lvit: language meets vision transformer in medical image segmentation. IEEE Transactions on Medical Imaging , 2023. -
  52. [70][70] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. -
  53. [71][71] Mingquan Lin, Song Wang, Ying Ding, Lihui Zhao, Fei Wang, and Yifan Peng. An empirical study of using radiology reports and images to improve ICU-mortality prediction. IEEE Int Conf Healthc Inform , 2021:497498, August 2021. -
  54. [72][72] Bo Liu, Li-Ming Zhan, and Xiao-Ming Wu. Contrastive pre-training and representation distillation for medical visual question answering based on radiology images. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2021 , pages 210-220. Springer International Publishing, 2021. -
  55. [73][73] Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A Semantically-Labeled Knowledge-Enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) , pages 1650-1654. ieeexplore.ieee.org, April 2021. -
  56. [74][74] Feifan Liu, Yalei Peng, and Max P Rosen. An effective deep transfer learning and information fusion framework for medical visual question answering. In Experimental IR Meets Multilinguality, Multimodality, and Interaction , pages 238-247. Springer International Publishing, 2019. -
  57. [75][75] Fenglin Liu, Shen Ge, and Xian Wu. Competence-based multimodal curriculum learning for medical report generation. arXiv preprint arXiv:2206.14579 , 2022. - · arXiv
  58. [76][76] Guangyi Liu, Yinghong Liao, Fuyu Wang, Bin Zhang, Lu Zhang, Xiaodan Liang, Xiang Wan, Shaolin Li, Zhen Li, Shuixing Zhang, and Shuguang Cui. Medical-VLBERT: Medical visual language BERT for COVID-19 CT report generation with alternate learning. IEEE Trans Neural Netw Learn Syst , 32(9):3786-3797, September 2021. -
  59. [77][77] Shengyan Liu, Xuejie Zhang, Xiaobing Zhou, and Jian Yang. BPI-MVQA: a bi-branch model for medical visual question answering. BMC Med. Imaging , 22(1):79, April 2022. -
  60. [78][78] Danial Maleki and Hamid R Tizhoosh. Lile: Look in-depth before looking elsewhere-a dual attention network using transformers for cross-modal information retrieval in histopathology archives. In International Conference on Medical Imaging with Deep Learning , pages 879-894. PMLR, 2022. -
  61. [79][79] Jonathan Masci, Ueli Meier, Dan Cires ¸an, and J¨ urgen Schmidhuber. Stacked convolutional Auto-Encoders for hierarchical feature extraction. In Artificial Neural Networks and Machine Learning - ICANN 2011 , pages 52-59. Springer Berlin Heidelberg, 2011. -
  62. [80][80] Pablo Messina, Pablo Pino, Denis Parra, Alvaro Soto, Cecilia Besa, Sergio Uribe, Marcelo And´ ıa, Cristian Tejos, Claudia Prieto, and Daniel Capurro. A survey on deep learning and explainability for automatic report generation from medical images. ACM Comput. Surv. , 54(10s):1-40, September 2022. -
  63. [82][82] Masoud Monajatipoor, Mozhdeh Rouhsedaghat, Liunian Harold Li, C-C Jay Kuo, Aichi Chien, and Kai-Wei Chang. BERTHop: An effective Vision-and-Language model for chest x-ray disease diagnosis. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2022 , pages 725-734. Springer Nature Switzerland, 2022. -
  64. [83][83] Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young-Hak Kim, and Edward Choi. Multi-modal understanding and generation for medical images and text via Vision-Language Pre-Training. IEEE J Biomed Health Inform , PP, September 2022. -
  65. [85][85] Philip M¨ uller, Georgios Kaissis, Congyu Zou, and Daniel Rueckert. Joint learning of localized representations from medical images and reports. In Computer Vision - ECCV 2022 , pages 685-701. Springer Nature Switzerland, 2022. -
  66. [86][86] Usman Naseem, Matloob Khushi, and Jinman Kim. Vision-Language transformer for interpretable pathology visual question answering. IEEE J Biomed Health Inform , PP, March 2022. -
  67. [87][87] Binh D Nguyen, Thanh-Toan Do, Binh X Nguyen, Tuong Do, Erman Tjiputra, and Quang D Tran. Overcoming data limitation in medical visual question answering. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 , pages 522-530. Springer International Publishing, 2019. -
  68. [88][88] Jianmo Ni, Chun-Nan Hsu, Amilcare Gentili, and Julian McAuley. Learning visual-semantic embeddings for reporting abnormal findings on chest X-rays. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1954-1960, Online, November 2020. Association for Computational Linguistics. doi: 10. [18653 / v1 / 2020.findings-emnlp.176. URL https://aclanthology.org/2020.findings-emnlp.176 .](https://aclanthology.org/2020.findings-emnlp.176) -
  69. [89][89] Toru Nishino, Ryota Ozaki, Yohei Momoki, Tomoki Taniguchi, Ryuji Kano, Norihisa Nakano, Yuki Tagawa, Motoki Taniguchi, Tomoko Ohkuma, and Keigo Nakamura. Reinforcement learning with imbalanced dataset for data-to-text medical report generation. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 2223-2236, 2020. -
  70. [90][90] Esha Pahwa, Dwij Mehta, Sanjeet Kapadia, Devansh Jain, and Achleshwar Luthra. MedSkip: Medical report generation using skip connections and integrated attention. In 2021 IEEE / CVFInternational Conference on Computer Vision Workshops (ICCVW) , pages 3409-3415. IEEE, October 2021. -
  71. [91][91] Haiwei Pan, Shuning He, Kejia Zhang, Bo Qu, Chunling Chen, and Kun Shi. Muvam: A multi-view attentionbased model for medical visual question answering. arXiv preprint arXiv:2107.03216 , 2021. - · arXiv
  72. [93][93] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ACL '02, pages 311-318, USA, July 2002. Association for Computational Linguistics. doi: 10.3115 / 1073083. 1073135. -
  73. [94][94] Obioma Pelka, Sven Koitka, Johannes R¨ uckert, Felix Nensa, and Christoph M Friedrich. Radiology objects in COntext (ROCO): A multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis , pages 180-189. Springer International Publishing, 2018. -
  74. [95][95] Han Qin and Yan Song. Reinforced cross-modal alignment for radiology report generation. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 448-458, Dublin, Ireland, May 2022. Association for Computational Linguistics. -
  75. [96][96] Ziyuan Qin, Huahui Yi, Qicheng Lao, and Kang Li. Medical image understanding with pretrained vision language models: A comprehensive study. arXiv preprint arXiv:2209.15517 , 2022. - · arXiv
  76. [97][97] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 2021. -
  77. [98][98] Fuji Ren and Yangyang Zhou. CGMVQA: A new classification and generative model for medical visual question answering. IEEE Access , 8:50626-50636, 2020. -
  78. [99][99] Arpita Roy and Shimei Pan. Incorporating medical knowledge in BERT for clinical relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5357-5366, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. -
  79. [100][100] Dhruv Sharma, Sanjay Purushotham, and Chandan K Reddy. MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain. Sci. Rep. , 11(1):19826, October 2021. -
  80. [101][101] George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, Ritu R Gill, Myrna C B Godoy, Stephen Hobbs, Jean Jeudy, Archana Laroia, Palmi N Shah, Dharshan Vummidi, Kavitha Yaddanapudi, and Anouk Stein. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiol Artif Intell , 1(1):e180041, January 2019. -
  81. [102][102] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. - · arXiv
  82. [103][103] Gizem Sogancioglu, Hakime ¨ Ozt¨ urk, and Arzucan ¨ Ozg¨ ur. BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics , 33(14):i49-i58, July 2017. -
  83. [104][104] Gregor Stiglic, Primoz Kocbek, Nino Fijacko, Marinka Zitnik, Katrien Verbert, and Leona Cilar. Interpretability of machine learning-based prediction models in healthcare. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. , 10 (5):e1379, September 2020. -
  84. [105][105] Ajay K Tanwani, Joelle Barral, and Daniel Freedman. RepsNet: Combining vision with language for automated medical reports. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2022 , pages 714724. Springer Nature Switzerland, 2022. -
  85. [106][106] Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nat Biomed Eng , 6(12):13991406, December 2022. -
  86. [110][110] Tom van Sonsbeek, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational knowledge distillation for disease classification in chest X-Rays. In Information Processing in Medical Imaging , pages 334-345. Springer International Publishing, 2021. -
  87. [111][111] Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi-granularity cross-modal alignment for generalized medical visual representation learning. Advances in Neural Information Processing Systems , 35:33536-33549, 2022. -
  88. [112][112] He Wang, Haiwei Pan, Kejia Zhang, Shuning He, and Chunling Chen. M2FNet: Multi-granularity feature fusion network for medical visual question answering. In PRICAI 2022: Trends in Artificial Intelligence , pages 141-154. Springer Nature Switzerland, 2022. -
  89. [113][113] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers. TieNet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 9049-9058. IEEE, June 2018. -
  90. [114][114] Xiaosong Wang, Ziyue Xu, Leo Tam, Dong Yang, and Daguang Xu. Self-supervised image-text pre-training with mixed data in chest x-rays. arXiv preprint arXiv:2103.16022 , 2021. - · arXiv
  91. [115][115] Xuwen Wang and Jiao Li. ImageSem group at ImageCLEFmedical caption 2022 task: Generating medical image descriptions based on Vision-Language pre-training. http://ceur-ws.org/Vol-3180/paper-124.pdf , 2022. Accessed: 2023-1-5. -
  92. [116][116] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. MedCLIP: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3876-3887, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. -
  93. [117][117] Wei-Hung Weng, Yuannan Cai, Angela Lin, Fraser Tan, and Po-Hsuan Cameron Chen. Multimodal multitask representation learning for pathology biobank metadata prediction. arXiv preprint arXiv:1909.07846 , 2019. - · arXiv
  94. [118][118] Qi Wu, Peng Wang, Xin Wang, Xiaodong He, and Wenwu Zhu. Medical VQA. In Qi Wu, Peng Wang, Xin Wang, Xiaodong He, and Wenwu Zhu, editors, Visual Question Answering: From Theory to Application , pages 165-176. Springer Nature Singapore, Singapore, 2022. -
  95. [119][119] Feng Xie, Han Yuan, Yilin Ning, Marcus Eng Hock Ong, Mengling Feng, Wynne Hsu, Bibhas Chakraborty, and Nan Liu. Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies. J. Biomed. Inform. , 126:103980, February 2022. -
  96. [121][121] Ke Yan, Yifan Peng, Veit Sandfort, Mohammadhadi Bagheri, Zhiyong Lu, and Ronald M Summers. Holistic and comprehensive annotation of clinically significant findings on diverse CT images: Learning from radiology reports and label ontology. In 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8523-8532. IEEE, June 2019. -
  97. [122][122] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2129, 2016. -
  98. [123][123] Di You, Fenglin Liu, Shen Ge, Xiaoxia Xie, Jing Zhang, and Xian Wu. AlignTransformer: Hierarchical alignment of visual regions and disease tags for medical report generation. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2021 , pages 72-82. Springer International Publishing, 2021. -
  99. [124][124] Jingyi You, Dongyuan Li, Manabu Okumura, and Kenji Suzuki. JPG - jointly learn to align: Automated disease prediction and radiology report generation. In Proceedings of the 29th International Conference on Computational Linguistics , pages 5989-6001, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. -
  100. [126][126] Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages 1821-1830. IEEE, October 2017. -
  101. [127][127] Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: Generalized multimodal factorized High-Order pooling for visual question answering. IEEE Trans Neural Netw Learn Syst , 29(12):59475959, December 2018. -
  102. [128][128] Jianbo Yuan, Haofu Liao, Rui Luo, and Jiebo Luo. Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 , pages 721-729. Springer International Publishing, 2019. -
  103. [129][129] Chenlu Zhan, Peng Peng, Hongsen Wang, Tao Chen, and Hongwei Wang. Uniclam: Contrastive representation learning with adversarial masking for unified and interpretable medical vision question answering. arXiv preprint arXiv:2212.10729 , 2022. - · arXiv
  104. [130][130] Li-Ming Zhan, Bo Liu, Lu Fan, Jiaxin Chen, and Xiao-Ming Wu. Medical visual question answering via conditional reasoning. In Proceedings of the 28th ACM International Conference on Multimedia , MM '20, pages 2345-2354, New York, NY, USA, October 2020. Association for Computing Machinery. -
  105. [132][132] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference , pages 2-25. PMLR, 2022. -
  106. [133][133] Ziyuan Zhao, Jinxuan Hu, Zeng Zeng, Xulei Yang, Peisheng Qian, Bharadwaj Veeravalli, and Cuntai Guan. MMGL: Multi-Scale Multi-View Global-Local contrastive learning for Semi-Supervised cardiac image segmentation. In 2022 IEEE International Conference on Image Processing (ICIP) , pages 401-405. ieeexplore.ieee.org, October 2022. -
  107. [135][135] Hong-Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports. Nature Machine Intelligence , 4(1):32-40, 2022. -
  108. [136][136] Yi Zhou, Lei Huang, Tao Zhou, Huazhu Fu, and Ling Shao. Visual-textual attentive semantic consistency for medical report generation. In 2021 IEEE / CVF International Conference on Computer Vision (ICCV) , pages 39853994. IEEE, October 2021. -
  109. [137][137] Yuyin Zhou, Shih-Cheng Huang, Jason Alan Fries, Alaa Youssef, Timothy J Amrhein, Marcello Chang, Imon Banerjee, Daniel Rubin, Lei Xing, Nigam Shah, et al. Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detection from ct and ehr. arXiv preprint arXiv:2111.11665 , 2021. · arXiv

Cited by

Sorted by citations

Authors
Sun, Z., Lin, M., Zhu, Q., Xie, Q., Wang, F., Lu, Z., & Peng, Y.
Affiliations
  • Weill Cornell Medicine (US)
  • National Institutes of Health (US)
Corresponding author
Peng, Yifan
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2023-10
Volume
146
Pages
104482
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2023.104482
PMC
PMC10591890
OpenAlex
W4384615843
Cited by
69
References
17
Citations by year
2026: 26 · 2025: 36 · 2024: 6 · 2023: 1
FWCI
1.79
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • PubMed Central (submittedVersion)
  • arXiv (Cornell University)
Funders
  • U.S. National Library of Medicine (4R00LM013001)
  • National Science Foundation (2145640)
  • National Institutes of Health
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
multimodal deep learning, biomedical images, medical texts, computer-aided diagnosis, report generation, visual question answering, cross-modal retrieval, semantic segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review