A survey of multimodal information fusion for smart healthcare: Mapping the journey from data to wisdom
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Multimodal medical data fusion has emerged as a transformative approach in smart healthcare, enabling a comprehensive understanding of patient health and personalized treatment plans. In this paper, a journey from data to information to knowledge to wisdom (DIKW) is explored through multimodal fusion for smart healthcare. We present a comprehensive review of multimodal medical data fusion focused on the integration of various data modalities. The review explores different approaches such as feature selection, rule-based systems, machine ;earning, deep learning, and natural language processing, for fusing and analyzing multimodal data. This paper also highlights the challenges associated with multimodal fusion in healthcare. By synthesizing the reviewed frameworks and theories, it proposes a generic framework for multimodal medical data fusion that aligns with the DIKW model. Moreover, it discusses future directions related to the four pillars of healthcare: Predictive, Preventive, Personalized, and Participatory approaches. The components of the comprehensive survey presented in this paper form the foundation for more successful implementation of multimodal fusion in smart healthcare. Our findings can guide researchers and practitioners in leveraging the power of multimodal fusion with the state-of-the-art approaches to revolutionize healthcare and improve patient outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Yan, S., Xiong, Y., & Lin, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12328 · Cited by 5,410 · cited 1×
“Graph-based algorithms, such as graph convolutional networks (GCNs) or graph regularized non-negative matrix factorization (GNMF), can then be applied to capture the dependencies and interactions between modalities [97].”
MethodA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“By integrating and analyzing data from various modalities, such as genomics, medical imaging, and clinical records, healthcare professionals can identify early indicators or risk factors [185].”
BackgroundWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · cited 1×
“TCIA [43] Millions of images Various data types Cancer research High”
BackgroundThe Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods
Jack, C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., L. Whitwell, J., Ward, C., et al. (2008) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.21049 · Cited by 4,347
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., Freymann, J. B., Farahani, K., & Davatzikos, C. (2017) · Scientific Data · doi:10.1038/sdata.2017.117 · Cited by 3,049
Armato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2011) · Medical Physics · doi:10.1118/1.3528204 · Cited by 2,887
Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018) · Scientific Data · doi:10.1038/sdata.2018.178 · Cited by 1,993
Industry 5.0: A survey on enabling technologies and potential applications
Maddikunta, P. K. R., Pham, Q.-V., B, P., Deepa, N., Dev, K., Gadekallu, T. R., Ruby, R., & Liyanage, M. (2022) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2021.100257 · Cited by 1,890 · cited 1×
“To address these challenges, involving clinicians and other stakeholders in the development and implementation process is crucial for both ensuring successful adoption and maximizing the impact of multimodal fusion in clinical practice [161], [162].”
BackgroundAli, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · cited 1×
“Effective interpretation is crucial for extracting meaningful insights and making actionable decisions in clinical settings [175].”
BackgroundEnabling technologies and tools for digital twin
Qi, Q., Tao, F., Hu, T., Anwer, N., Liu, A., Wei, Y., Wang, L., & Nee, A. Y. C. (2021) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2019.10.001 · Cited by 1,579 · cited 1×
“Effective visualization techniques can aid in interpreting and communicating analysis results [160].”
BackgroundBakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R. T., Berger, C., Ha, S. M., Rozycki, M., et al. (2018) · Apollo - University of Cambridge Repository · doi:10.17863/cam.38755 · Cited by 1,441
Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R. T., Berger, C., Ha, S. M., Rozycki, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.02629 · Cited by 1,441
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · cited 2×
“Integrating genomic data with other health data sources, such as EHRs and wearable devices, can provide a comprehensive overview of an individual’s health [27].”
BackgroundPreparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164
A Survey on Neural Network Interpretability
Zhang, Y., Tino, P., Leonardis, A., & Tang, K. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3100641 · Cited by 923 · cited 1×
“Various methods can be employed to enhance interpretability, such as feature importance ranking, visualization techniques, or rule extraction algorithms [71].”
MethodEmotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716 · cited 1×
“[59] emphasized its role in effective decision-making and improved patient care by identifying the most relevant features, reducing dimensionality,”
BackgroundQiu, S., Zhao, H., Jiang, N., Wang, Z., Liu, L., An, Y., Zhao, H., Miao, X., Liu, R., & Fortino, G. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.006 · Cited by 671
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606 · cited 1×
“Evaluation can involve assessing classification accuracy, regression performance, clustering quality, or other domain-specific evaluation criteria [68].”
BackgroundApplications of machine learning to diagnosis and treatment of neurodegenerative diseases
Myszczynska, M. A., Ojamies, P. N., Lacoste, A. M. B., Neil, D., Saffari, A., Mead, R., Hautbergue, G. M., Holbrook, J. D., & Ferraiuolo, L. (2020) · Nature Reviews Neurology · doi:10.1038/s41582-020-0377-8 · Cited by 595
Machine learning and natural language processing in mental health: systematic review
Le Glaz, A., Haralambous, Y., Kim-Dufor, D.-H., Lenca, P., Billot, R., Ryan, T. C., Marsh, J., DeVylder, J., Walter, M., Berrouiguet, S., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/15708 · Cited by 576 · cited 1×
“NLP techniques contribute to patient risk assessment by analyzing clinical narratives and extracting relevant information related to patient history, comorbidities, and lifestyle factors [127].”
BackgroundSarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00765-8 · Cited by 570
Multimodal data fusion for cancer biomarker discovery with deep learning
Steyaert, S., Pizurica, M., Nagaraj, D., Khandelwal, P., Hernandez-Boussard, T., Gentles, A. J., & Gevaert, O. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00633-5 · Cited by 402 · cited 2×
“Integrating multimodal fusion technologies with CDSS can enhance clinical decision-making processes by providing real-time recommendations and alerts based on the fused data [155], [165].”
BackgroundUK biobank data: come and get it
Allen, N. E., Sudlow, C., Peakman, T., Collins, R., & Biobank, O. B. O. U. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3008601 · Cited by 399
Clinical text data in machine learning: systematic review
Spasic, I., & Nenadic, G. (2020) · JMIR Medical Informatics · doi:10.2196/17984 · Cited by 358
Boehm, K. M., Aherne, E. A., Ellenson, L., Nikolovski, I., Alghamdi, M., Vázquez-García, I., Zamarin, D., Long Roche, K., Liu, Y., Patel, D., et al. (2022) · Nature Cancer · doi:10.1038/s43018-022-00388-9 · Cited by 357 · cited 2×
“In the DIKW framework, the “Participatory” component of multimodal fusion in smart healthcare focuses on empowering patients to actively participate in their own healthcare journey, fostering collaboration and shared decision-making with healthcare providers [199], [200].”
BackgroundHuman-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 1×
“Collaboration between clinicians and data scientists are crucial in developing effective data processing and analysis solutions that align with clinical needs [153], [154].”
BackgroundReinforcement learning for intelligent healthcare applications: A survey
Coronato, A., Naeem, M., De Pietro, G., & Paragliola, G. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101964 · Cited by 320
The future of sleep health: a data-driven revolution in sleep science and medicine
Perez-Pozuelo, I., Zhai, B., Palotti, J., Mall, R., Aupetit, M., Garcia-Gomez, J. M., Taheri, S., Guan, Y., & Fernandez-Luque, L. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0244-4 · Cited by 311 · cited 1×
“The integration of data from diverse sources and ensuring its quality and compatibility across different healthcare systems and modalities can be complex and time-consuming [131].”
BackgroundA comprehensive survey on multimodal medical signals fusion for smart healthcare systems
Muhammad, G., Alshehri, F., Karray, F., Saddik, A. E., Alsulaiman, M., & Falk, T. H. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.007 · Cited by 287 · cited 3×
“They provide a structured approach to decision-making and knowledge representation [72], offering a systematic framework for integrating information from various modalities.”
BackgroundMethodLong-term cancer survival prediction using multimodal deep learning
Vale-Silva, L. A., & Rohr, K. (2021) · Scientific Reports · doi:10.1038/s41598-021-92799-4 · Cited by 268
Deep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., & Najork, M. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463257 · Cited by 214
Wang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 1×
“Integration of multimodal medical data fusion into Clinical Decision Support Systems (CDSS) enhances clinical decision-making and improves patient outcomes [155], [156].”
BackgroundMulti-Modal Graph Learning for Disease Prediction
Zheng, S., Zhu, Z., Liu, Z., Guo, Z., Liu, Y., Yang, Y., & Zhao, Y. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3159264 · Cited by 196
Design and Development of a Multimodal Biomedical Information Retrieval System
Demner-Fushman, D., Antani, S., Simpson, M., & Thoma, G. R. (2012) · Journal of Computing Science and Engineering · doi:10.5626/jcse.2012.6.2.168 · Cited by 161
Soklaridis, S., Lin, E., Lalani, Y., Rodak, T., & Sockalingam, S. (2020) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2020.08.007 · Cited by 153 · cited 1×
“This data has been used to guide the development of stress management techniques, relaxation strategies, and personalised interventions to support individuals in maintaining good mental health [57].”
MethodRundo, L., Pirrone, R., Vitabile, S., Sala, E., & Gambino, O. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103479 · Cited by 113 · cited 2×
“Applying usercentered design principles and conducting usability testing can identify and address usability issues, enhancing user satisfaction and adoption rates [164, 83].”
BackgroundMultimodal data fusion for systems improvement: A review
Gaw, N., Yousefi, S., & Gahrooei, M. R. (2022) · IISE Transactions · doi:10.1080/24725854.2021.1987593 · Cited by 101 · cited 1×
“Efforts should be made to mitigate biases through rigorous data collection processes, algorithmic fairness assessments, and diverse representation in data and development teams [172].”
BackgroundMulti-Modal Neuroimaging Feature Fusion for Diagnosis of Alzheimer’s Disease
Zhang, T., & Shi, M. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2020.108795 · Cited by 91 · cited 1×
“By selecting relevant features within each modality, noise and irrelevant information can be reduced, leading to improved fusion outcomes [62].”
BackgroundIn Search of Big Medical Data Integration Solutions - A Comprehensive Survey
Dhayne, H., Haque, R., Kilany, R., & Taher, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2927491 · Cited by 84 · cited 1×
“Within the DIKW framework, the Participatory component involves leveraging multimodal fusion to provide patients with real-time feedback and insights into their health status [201].”
BackgroundPei, X., Zuo, K., Li, Y., & Pang, Z. (2023) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-023-00225-6 · Cited by 77
MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17260 · Cited by 52
MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · arXiv · doi:10.48550/arxiv.2102.02340 · Cited by 52
Cited by
Review of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93
Multimodal Deep Learning Model for Prediction of Prognosis in Central Nervous System Inflammation
Choi, B. K., Choi, Y., Jang, S., Ha, W.-S., Cho, S., Chang, K., Sohn, B., Kim, K. M., & Park, Y. R. (2025) · Brain Communications · doi:10.1093/braincomms/fcaf179 · Cited by 1
“29 The clinical variables selected for MMDL differed depending on the aetiology.”
Metadata
- Authors
- Shaik, T., Tao, X., Li, L., Xie, H., & Velásquez, J. D.
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2024-02
- Volume
- 102
- Pages
- 102040
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W4387079109
- Cited by
- 196
- References
- 52
- FWCI
- 41.11
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.