CoChat
Sign Up
Cited by 77Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationAdvanced Image Fusion Techniques

A Review of the Application of Multi-modal Deep Learning in Medicine: Bibliometrics and Future Directions

Open access
Pei, Xiangdong, Zuo, Ke, Li, Yuan, Pang, Zhengbin·2023·International Journal of Computational Intelligence Systems
DOI 10.1007/s44196-023-00225-6
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, deep learning has been applied in the field of clinical medicine to process large-scale medical images, for large-scale data screening, and in the diagnosis and efficacy evaluation of various major diseases. Multi-modal medical data fusion based on deep learning can effectively extract and integrate characteristic information of different modes, improve clinical applicability in diagnosis and medical evaluation, and provide quantitative analysis, real-time monitoring, and treatment planning. This study investigates the performance of existing multi-modal fusion pre-training algorithms and medical multi-modal fusion methods and compares their key characteristics, such as supported medical data, diseases, target samples, and implementation performance. Additionally, we present the main challenges and goals of the latest trends in multi-modal medical convergence. To provide a clearer perspective on new trends, we also analyzed relevant papers on the Web of Science. We obtain some meaningful results based on the annual development trends, country, institution, and journal-level research, highly cited papers, and research directions. Finally, we perform co-authorship analysis, co-citation analysis, co-occurrence analysis, and bibliographic coupling analysis using the VOSviewer software.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s44196-023-00225-6

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • No review has systematically summarized the methodologies and processes specific to the application of multimodal AI in critical care.

    “However, to the best of our knowledge, no review has systematically summarized the methodologies and processes specific to the application of multimodal AI in critical care.”

    From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [22]

    The Medical Imaging and Non-imaging Models and Conclusion passages discuss multimodal deep learning in medicine, but neither addresses whether a review of multimodal AI methodologies and processes in critical care exists. Searches for critical care, intensive care, and the claimed absence of a systematic review found no passage stating that point; the available passages have no page numbers. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Atrey, P.K., et al.: Multi-modal fusion for multimedia analysis: a survey. Multimed. Syst. 16(6), 345-379 (2010)
  2. Baltrušaitis, T., et al.: Multi-modal machine learning: a survey and taxonomy. IEEE Trans. Pattern. Anal. Mach. Intell. 41(2), 423-443 (2018)
  3. Wang, Z., et al.: GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction. Bioinform. 37(18), 2963-2970 (2021)
  4. Cui, H., et al.: Co-graph attention reasoning based imaging and clinical features integration for lymph node metastasis predic- tion. In: Proc. Int. Conf. MICCAI (pp. 657-666). Springer, Cham (2021)
  5. Radford, A., et al.: Learning transferable visual models from natu- ral language supervision. In: Proc. ICML, pp. 8748-8763. PMLR (2021)
  6. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., Duerig, T. In Scaling up visual and vision- language representation learning with noisy text supervision. In: International Conference on Machine Learning, PMLR, pp. 4904-4916 (2021)
  7. Ding, M., et al.: Cogview: mastering text-to-image generation via transformers. Adv. Neural Inf. Process. Syst. 34, 19822-19835 (2021)
  8. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F. In Oscar: Object-semantics aligned pre-training for vision-language tasks. In: European Conference on Computer Vision, pp. 121-137. Springer, Berlin (2020)
  9. Chen, Y.C., et al.: UNITER: UNiversal Image-TExt Represen- tation Learning. In: Proc. ECCV, pp. 104-120. Springer, Cham (2020)
  10. Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., Gao, J.: In Vinvl: Revisiting visual representations in vision- language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588 (2021)
  11. Kim, W., et al.: ViLT: vision-and-language Transformer without convolution or region supervision. In: ICML, pp. 5583-5594. PMLR (2021)
  12. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv: 1611. 01578 (2016)
  13. Holste, G., et al.: End-to-end learning of fused image and non- image features for improved breast cancer classification from MRI. In: Proc. ICCV, pp. 3294-3303. IEEE (2021)
  14. Yan, R., et al.: Richer fusion network for breast cancer classifica- tion based on Multi-modal data. BMC Med. Inform. Decis. Mak. 21(1), 1-15 (2021)
  15. Yao, J., et al.: Deep correlational learning for survival predic- tion from multi-modality data. In: Proc. MICCAI, pp. 406-414. Springer, Cham (2017)
  16. Cheerla, A., et al.: Deep learning with multi-modal representation for pan-cancer prognosis prediction. Bioinform. 35(14), 446-454 (2019)
  17. Araújo, T., et al.: Classification of breast cancer histology images using convolutional neural networks. PLoS ONE 12(6), e0177544 (2017)
  18. Lu, M.Y., et al.: AI-based pathology predicts origins for cancers of unknown primary. Nature 594(7861), 106-110 (2021)
  19. Mobadersany, P., et al.: Predicting cancer outcomes from histol- ogy and genomics using convolutional networks. Proc. Natl. Acad. Sci. USA 115(13), E2970-E2979 (2018)
  20. Kawahara, J., et al.: Seven-point checklist and skin lesion classifi- cation using multitask multi-modal neural nets. IEEE J. Biomed. Health. Inform. 23(2), 538-546 (2018)
  21. Yoo, Y., et al.: Deep learning of brain lesion patterns and user- defined clinical and MRI features for predicting conversion to multiple sclerosis from the clinically isolated syndrome. Com- put. Methods Biomech. Biomed. Eng. Imaging Vis. 7(3), 250-259 (2019)
  22. Guan, Y., et al.: Predicting esophageal fistula risks using multi- modal self-attention network. In: Proc. Int. Conf. MICCAI, pp. 721-730. Springer, Cham (2021)
  23. Silva, L., et al.: Pan-cancer prognosis prediction using multi- modal deep learning. In: Proc. ISBI, pp. 568-571. IEEE (2020)
  24. Braman, N., et al.: Deep orthogonal fusion: Multi-modal prognos- tic biomarker discovery integrating radiology, pathology, genom- ics, and clinical data. In: Proc. MICCAI, pp. 667-677. Springer, Cham (2021)
  25. Chen, R.J., et al.: Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans. Med. Imaging 41(4), 757-770 (2022)
  26. Sandfort, V., Yan, K., Pickhardt, P.J., Summers, R.M.: Data aug- mentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Sci. Rep. 9(1), 16884 (2019)
  27. Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv: 2010. 00747 (2020)
  28. Ramesh, A., et al.: Zero-shot text-to-image generation. In: Proc. ICML, pp. 8821-8831. PMLR (2021)
  29. Li, J., et al.: BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086 (2022) · arXiv
  30. Huo, Y., et al.: WenLan: Bridging vision and language by large- scale multi-modal pre-training. arXiv preprint arXiv: 2103. 06561 (2021)
  31. Hu, R., et al.: Unit: Multi-modal multitask learning with a unified Transformer. In: Proc. ICCV, pp. 1439-1449. IEEE (2021)
  32. Zhang, Y., et al.: Modality-aware mutual learning for multi-modal medical image segmentation. In: Proc. MICCAI, pp. 589-599. Springer, Cham (2021)
  33. Li, S., et al.: A novel pathological images and genomic data fusion framework for breast cancer survival prediction. In: Proc. Int. Conf. EMBC, pp. 1384-1387. IEEE (2020)
  34. Zhou, J., et al.: Cohesive multi-modality feature learning and fusion for COVID-19 patient severity prediction. IEEE Trans. Circuits Syst. Video. Technol. (2021)
  35. Li, H., et al.: Multi-modal multi-instance learning using weakly correlated histopathological images and tabular clinical informa- tion. In: Proc. MICCAI, pp. 529-539. Springer, Cham. (2021)
  36. Zhou, T., et al.: Effective feature learning and fusion of multi- modality data using stage-wise deep neural network for dementia diagnosis. Hum. Brain. Mapp. 40(3), 1001-1016 (2019)
  37. Li, X., et al.: Self-supervised feature learning via exploiting multi- modal data for retinal disease diagnosis. IEEE Trans. Med. Imag- ing 39(12), 4023-4033 (2020)
  38. Floridi, L., Chiriatti, M.J.M.: Machines, GPT-3: its nature, scope. Lim. Conseq. 30(4), 681-694 (2020)
  39. Xu, Z., So, D., Dai, A.: MUFASA: Multi-modal fusion architec- ture search for electronic health records. Proc. AAAI Conf. Artif. Intell. 35(12), 10532-10540 (2021)
  40. Adams, J.: Information and misinformation in bibliometric time- trend analysis. J. Infometr. 12(4), 1063-1071 (2018)
  41. Garfield, E.: From the science of science to Scientometrics: visual- izing the history of science with HistCite software. J. Informetr. 3(3), 173-179 (2009)
  42. Chen, C.: CityPlace II: detecting and visualizing emerging trends and transient patterns in scientific literature. J. Assoc. Inf. Sci. Technol. 57(3), 359-377 (2006)
  43. Van, E.N.J., et al.: Software survey: VOSviewer, a computer pro- gram for bibliometric mapping. Scientometrics 84(2), 523-538 (2010)
  44. patient data. IEEE Trans. Pattern Anal. Mach. Intel. 35(8), 1930- 1943 (2012)
  45. Mahmud, M., Kaiser, M.S., Hussain, A., Vassanelli, S.: Applica- tions of deep learning and reinforcement learning to biological data. IEEE Trans. Neural. Netw. Learn. Syst. 29(6), 2063-2079 (2018)
  46. Estai, M., Bunt, S.: Best teaching practices in anatomy education: A critical review. Ann. Anat. 208, 151-157 (2016)
  47. Liu, Z., et al.: The applications of radiomics in precision diagnosis and treatment of oncology: Opportunities and challenges. Thera- nostics 9(5), 1303 (2019)
  48. Bychkov, D., et al.: Deep learning-based tissue analysis predicts outcome in colorectal cancer. Sci. Rep. 8(1), 1-11 (2018)
  49. Zeng, X., et al.: deepDR: a network-based deep learning approach to in silico drug repositioning. Bioinform. 35(24), 5191-5198 (2019)
  50. Fetit, A.E, et al. A multimodal approach to cardiovascular risk stratification in patients with type 2 diabetes incorporating retinal, genomic and clinical features. Sci. Rep. 9(1), 3591 (2019)

Cited by

Sorted by citations

Authors
Pei, X., Zuo, K., Li, Y., & Pang, Z.
Affiliations
National University of Defense Technology (CN)
Corresponding author
Pang, Zhengbin
ORCID
Venue
International Journal of Computational Intelligence Systems
Publisher
Springer Science and Business Media LLC
Published
2023-03-29
Volume
16
Issue
1
ISSN
1875-6883 (ISSN-L)
DOI
10.1007/s44196-023-00225-6
OpenAlex
W4361284377
Cited by
77
References
8
Citations by year
2026: 13 · 2025: 35 · 2024: 21 · 2023: 8
FWCI
7.00
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Computational Intelligence Systems (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
scientific research project of the Science and Technology Department of Shanxi Province (2020FP-11)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
Keywords
multimodal deep learning, medical image analysis, diagnosis, treatment planning, research trends, bibliometric analysis, co-authorship analysis, co-citation analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review