Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 501–600 of 1,036
Deep Feature Space: A Geometrical Perspective
Kansizoglou, I., Bampis, L., & Gasteratos, A. C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3094625 · Cited by 71
Introducing Representations of Facial Affect in Automated Multimodal Deception Detection
Mathur, L., & Matarić, M. J. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418864 · Cited by 72
Deep Learning for Radiotherapy Outcome Prediction Using Dose Data – A Review
Appelt, A. L., Elhaminia, B., Gooya, A., Gilbert, A., & Nix, M. G. (2021) · Clinical Oncology · doi:10.1016/j.clon.2021.12.002 · Cited by 62
Mmlatch: Bottom-Up Top-Down Fusion For Multimodal Sentiment Analysis
Paraskevopoulos, G., Georgiou, E., & Potamianos, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746418 · Cited by 76
Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features
Kakuba, S., Poulose, A., & Han, D. S. (2022) · IEEE Access · doi:10.1109/access.2022.3225684 · Cited by 66
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 69
CREST: Cross-Reference to Exchange-based Stock Trend Prediction using Long Short-Term Memory
Thakkar, A., & Chaudhari, K. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.03.328 · Cited by 63
Learning to Learn Better Unimodal Representations via Adaptive Multimodal Meta-Learning
Ya, S., Mai, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3178231 · Cited by 69
Bridged Transformer for Vision and Point Cloud 3D Object Detection
Wang, Y., Ye, T., Cao, L., Huang, W., Sun, F., He, F., & Tao, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01180 · Cited by 58
Lin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 74
Beyond Co-Occurrence: Multi-Modal Session-Based Recommendation
Zhang, X., Xu, B., Ma, F., Li, C., Yang, L., & Lin, H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3309995 · Cited by 80
Large Models for Aerial Edges: An Edge-Cloud Model Evolution and Communication Paradigm
Zhang, S., Liu, Q., Chen, K., Di, B., Zhang, H., Yang, W., Niyato, D. T., Han, Z., & Poor, H. V. (2024) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2024.3460078 · Cited by 61
Multimodal sentiment analysis with unimodal label generation and modality decomposition
Zhu, L., Zhao, H., Zhu, Z., Zhang, C., & Kong, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102787 · Cited by 73
Dress With Style: Learning Style From Joint Deep Embedding of Clothing Styles and Body Shapes
Hidayati, S. C., Goh, T. W., Chan, J.-S. G., Hsu, C.-C., See, J., Wong, L.-K., Hua, K., Tsao, Y., & Cheng, W.-H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2980195 · Cited by 68
Detecting Depression with Word-Level Multimodal Fusion
Rohanian, M., Hough, J., & Purver, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2283 · Cited by 83
Automatic Skull Stripping of Rat and Mouse Brain MRI Data Using U-Net
Hsu, L.-M., Wang, S., Ranadive, P., Ban, W., Chao, T.-H. H., Song, S., Cerri, D. H., Walton, L. R., Broadwater, M., Lee, S.-H., et al. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.568614 · Cited by 65
Qu, T., Wang, X., Fang, C., Mao, L., Li, J., Li, P., Qu, J., Li, X., Xue, H., Yu, Y., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102232 · Cited by 65
Obfuscation-Resilient Android Malware Analysis Based on Complementary Features
Gao, C., Cai, M., Yin, S., Huang, G., Li, H., Yuan, W., & Luo, X. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3302509 · Cited by 69
Multimodal vision-based human action recognition using deep learning: a review
Shafizadegan, F., Naghsh‐Nilchi, A. R., & Shabaninia, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10730-5 · Cited by 65
Contrastive Conditional Latent Diffusion for Audio-Visual Segmentation
Mao, Y., Zhang, J., Xiang, M., Lv, Y., Li, D., Zhong, Y., & Dai, Y. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3580269 · Cited by 68
A Comprehensive Review of Multimodal Deep Learning for Enhanced Medical Diagnostics
Al‐Zoghby, A. M., Ebada, A. I., Saleh, A. S., Abdelhay, M., & Awad, W. A. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.065571 · Cited by 87
Multi-Speaker Tracking From an Audio–Visual Sensing Device
Qian, X., Brutti, A., Lanz, O., Omologo, M., & Cavallaro, A. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2902489 · Cited by 72
P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene UnderstandingPreprint
Liu, Y., Yi, L., Zhang, S., Fan, Q., Funkhouser, T. A., & Dong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13089 · Cited by 66
A survey of transformer-based multimodal pre-trained modals
Han, X., Wang, Y.-T., Feng, J.-L., Deng, C., Chen, Z.-H., Huang, Y.-A., Su, H., Hu, L., & Hu, P.-W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.09.136 · Cited by 59
Yin, G., Sun, S., Yu, D., Li, D., & Zhang, K. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3490686 · Cited by 62
Leveraging Knowledge Graphs and Deep Learning for automatic art analysis
Castellano, G., Digeno, V., Sansaro, G., & Vessio, G. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108859 · Cited by 75
Excavating multimodal correlation for representation learning
Mai, S., Ya, S., Zeng, Y., & Hu, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.003 · Cited by 68
Ren, Z.-H., You, Z., Zou, Q., Yu, C.-Q., Ma, Y., Guan, Y.-J., You, H.-R., Wang, X., & Pan, J. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-03876-3 · Cited by 61
Interpretable multimodal emotion recognition using hybrid fusion of speech and image data
Kumar, P., Malik, S., & Raman, B. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16443-1 · Cited by 62
Huang, P., Li, C., He, P., Xiao, H., Ping, Y., Feng, P., Tian, S., Chen, H., Mercaldo, F., Santone, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102333 · Cited by 61
S.Rajarajeshwari, G. K. P., & Selvi, G. C. (2024) · IEEE Access · doi:10.1109/access.2024.3494840 · Cited by 68
GPT-4V(ision) as A Social Media Analysis Engine
Lyu, H., Huang, J., Zhang, D., Yu, Y., Mou, X., Pan, J., Yang, Z., Wei, Z., & Luo, J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3709005 · Cited by 66
Emotions in Code - The AI Frontier of Sentiment Analysis
Li, J. (2025) · Artificial intelligence · doi:10.5772/intechopen.1006407 · Cited by 59
Metric Learning Based Multimodal Audio-visual Emotion Recognition
Ghaleb, E., Popa, M. C., & Asteriadis, S. (2019) · IEEE Multimedia · doi:10.1109/mmul.2019.2960219 · Cited by 68
Micro-Video Popularity Prediction Via Multimodal Variational Information Bottleneck
Xie, J., Zhu, Y., & Chen, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3120537 · Cited by 75
Zuo, Q., Lei, B., Shen, Y., Liu, Y., Feng, Z., & Wang, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88010-1_40 · Cited by 78
Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition
Praveen, R. G., Granger, É., & Cardinal, P. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667055 · Cited by 73
Deep Convolution Neural Network sharing for the multi-label images classification
Coulibaly, S., Kamsu-Foguem, B., Kamissoko, D., & Traoré, D. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2022.100422 · Cited by 75
Formalizing Multimedia Recommendation through Multimodal Deep Learning
Malitesta, D., Cornacchia, G., Pomo, C., Merra, F. A., Noia, T. D., & Sciascio, E. D. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3662738 · Cited by 81
Harnessing multimodal data fusion to advance accurate identification of fish feeding intensity
Du, Z., Cui, M., Xu, X., Bai, Z., Han, J., Li, W., Yang, J., Liu, X., Wang, C., & Li, D. (2024) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2024.08.001 · Cited by 63
A machine learning approach for multimodal data fusion for survival prediction in cancer patients
Νικολάου, Ν., Salazar, D., RaviPrakash, H., Gonçalves, M., Mulla, R., Burlutskiy, N., Markuzon, N., & Jacob, E. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-00917-6 · Cited by 65
PerturbNet predicts single-cell responses to unseen chemical and genetic perturbations
Yu, H., Qian, W., Song, Y., & Welch, J. D. (2025) · Molecular Systems Biology · doi:10.1038/s44320-025-00131-3 · Cited by 57
AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos With Deep Learning
Ghose, S., & Prevost, J. J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3005033 · Cited by 57
Graph Fusion Network-Based Multimodal Learning for Freezing of Gait Detection
Hu, K., Wang, Z., Martens, K. A. E., Hagenbuchner, M., Bennamoun, M., Tsoi, A. C., & Lewis, S. J. G. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105602 · Cited by 67
UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation
Zhang, Z., Meng, X., Wang, Y., Jiang, X., Liu, Q., & Yang, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21431 · Cited by 77
Audio-Driven Talking Face Video Generation With Dynamic Convolution Kernels
Ye, Z., Xia, M., Yi, R., Zhang, J., Lai, Y., Huang, X., Zhang, G., & Liu, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142387 · Cited by 73
Explainable Multimodal Machine Learning for Engagement Analysis by Continuous Performance Test
Rahman, M. A., Brown, D. J., Shopland, N., Burton, A. M., & Mahmud, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05039-8_28 · Cited by 65
Kang, M., Zhu, R., Chen, D., Liu, X., & Yu, W. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3284666 · Cited by 57
Mapping urban functional zones with remote sensing and geospatial big data: a systematic review
Du, S., Zhang, X., Lei, Y., Huang, X., Tu, W., Liu, B., Meng, Q., & Du, S. (2024) · GIScience & Remote Sensing · doi:10.1080/15481603.2024.2404900 · Cited by 68
Multi-modal Time Series Analysis: A Tutorial and Survey
Jiang, Y., Ning, K., Pan, Z., Shen, X., Ni, J., Yu, W., Schneider, A., Chen, H., Nevmyvaka, Y., & Song, D. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736567 · Cited by 85
A Survey of Android Malware Static Detection Technology Based on Machine Learning
Wu, Q., Zhu, X., & Liu, B. (2021) · Mobile Information Systems · doi:10.1155/2021/8896013 · Cited by 78
Self-Supervised Correlation Learning for Cross-Modal Retrieval
Liu, Y., Wu, J., Qu, L., Gan, T., Yin, J., & Nie, L. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3152086 · Cited by 65
Martinez-Ríos, E. A., Montesinos, L., & Alfaro-Ponce, M. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105479 · Cited by 59
Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?
Rajan, V., Brutti, A., & Cavallaro, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746924 · Cited by 66
A review on multimodal zero‐shot learning
Cao, W., Wu, Y., Sun, Y., Zhang, H., Ren, J., Gu, D., & Wang, X. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1488 · Cited by 69
Yang, M., Wu, Y., Tao, Y., Hu, X., & Hu, B. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3322271 · Cited by 63
Kakuba, S., Poulose, A., & Han, D. S. (2023) · IEEE Access · doi:10.1109/access.2023.3325037 · Cited by 60
A Multi-Modal Vertical Federated Learning Framework Based on Homomorphic Encryption
Gong, M., Zhang, Y., Gao, Y., Qin, A. K., Wu, Y., Wang, S., & Zhang, Y. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3340994 · Cited by 70
Nguyen, C.-V. T., Mai, T., The, S., Kieu, H.-D., & Le, D.-T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.937 · Cited by 79
Vision + X: A Survey on Multimodal Learning in the Light of Data
Zhu, Y., Wu, Y., Sebe, N., & Yan, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3420239 · Cited by 80
A comprehensive review of visual–textual sentiment analysis from social media networks
Al-Tameemi, I. K. S., Feizi‐Derakhshi, M., Pashazadeh, S., & Asadpour, M. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00326-y · Cited by 67
An overview of methods and techniques in multimodal data fusion with application to healthcare
Chaabene, S., Boudaya, A., Bouaziz, B., & Chaâri, L. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00715-0 · Cited by 69
Foundations and Innovations in Data Fusion and Ensemble Learning for Effective Consensus
Du, K.-L., Zhang, R., Jiang, B., Zeng, J., & Lu, J. (2025) · Mathematics · doi:10.3390/math13040587 · Cited by 63
Bridging language and action: A survey of language-conditioned robot manipulation
Yao, X., Zhou, H., Mees, O., Meng, Y., Xiao, T. G., Bisk, Y., Oh, J., Johns, E., Shridhar, M., Shah, D., et al. (2026) · The International Journal of Robotics Research · doi:10.1177/02783649261468360 · Cited by 58
Image and Encoded Text Fusion for Multi-Modal Classification
Gallo, I., Calefati, A., Nawaz, S., & Janjua, M. K. (2018) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2018.8615789 · Cited by 67
Mann, P., Paes, A., & Matsushima, E. H. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7313 · Cited by 71
Gas Chromatographic Retention Index Prediction Using Multimodal Machine Learning
Matyushin, D. D., & Buryak, A. K. (2020) · IEEE Access · doi:10.1109/access.2020.3045047 · Cited by 56
On Removing Algorithmic Priority Inversion from Mission-critical Machine Inference Pipelines
Liu, S., Yao, S., Fu, X., Tabish, R., Yu, S., Bansal, A., Yun, H., Sha, L., & Abdelzaher, T. (2020) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss49844.2020.00037 · Cited by 69
Sentiment-influenced trading system based on multimodal deep reinforcement learning
Chen, Y., & Huang, S.-H. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107788 · Cited by 63
Mou, L., Zhou, C., Xie, P., Zhao, P., Jain, R., Gao, W., & Yin, B. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3128738 · Cited by 62
TorchGeo: Deep Learning With Geospatial Data
Stewart, A. J., Robinson, C., Corley, I., Ortiz, A., Ferres, J. L., & Banerjee, A. (2025) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3707459 · Cited by 77
Alam, M. U., & Rahmani, R. (2023) · Sensors · doi:10.3390/s23020970 · Cited by 64
Brain-Inspired Learning, Perception, and Cognition: A Comprehensive Review
Jiao, L., Ma, M., He, P., Geng, X., Liu, X., Liu, F., Ma, W., Yang, S., Hou, B., & Tang, X. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3401711 · Cited by 62
AI safety landscape for large language models: taxonomy, state-of-the-art, and future directions
Chen, C., Gong, X., Liu, Z., Jiang, W. M. W. L. H., Goh, S. Q., & Lam, K. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11590-x · Cited by 56
Tong, L., Wu, H., & Wang, M. D. (2021) · Methods · doi:10.1016/j.ymeth.2020.07.008 · Cited by 56
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
Multimodal-based weld reinforcement monitoring system for wire arc additive manufacturing
Shen, B., Lu, J., Wang, Y., Chen, D., Han, J., Zhang, Y., & Zhao, Z. (2022) · Journal of Materials Research and Technology · doi:10.1016/j.jmrt.2022.07.086 · Cited by 54
Multimodal Information Fusion for Weather Systems and Clouds Identification From Satellite Images
Bai, C., Zhao, D., Zhang, M., & Zhang, J. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3202246 · Cited by 55
Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction
Wang, Y., Sun, F., Huang, W., He, F., & Tao, D. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3211086 · Cited by 64
Learning the User’s Deeper Preferences for Multi-modal Recommendation Systems
Lei, F., Cao, Z., Yang, Y., Ding, Y., & Zhang, C. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3573010 · Cited by 59
Early-stage fusion of EEG and fNIRS improves classification of motor imagery
Li, Y., Zhang, X., & Ming, D. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1062889 · Cited by 65
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A. J. M., & Badia, S. B. I. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-022-09348-5 · Cited by 60
Nguyen, H.-S., & Vozn̆ák, M. (2024) · IEEE Access · doi:10.1109/access.2024.3363165 · Cited by 63
Contrastive Modality-Disentangled Learning for Multimodal Recommendation
Lin, X., Liu, R., Cao, Y., Zou, L., Li, Q., Wu, Y., Liu, Y., Yin, D., & Xu, G. (2025) · ACM Transactions on Information Systems · doi:10.1145/3715876 · Cited by 76
Recent Advances of Multimodal Continual Learning: A Comprehensive Survey
Yu, D., Zhang, X., Chen, Y., Liu, A., Zhang, Y., Yu, P. S., & King, I. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3658485 · Cited by 64
Cruickshank, I. J., & Carley, K. M. (2020) · Applied Network Science · doi:10.1007/s41109-020-00317-8 · Cited by 61
Computational Media Intelligence: Human-Centered Machine Analysis of Media
Somandepalli, K., Guha, T., Martínez, V. R., Kumar, N., Adam, H., & Narayanan, S. S. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3047978 · Cited by 62
Canonical Correlation Analysis (CCA) Based Multi-View Learning: An OverviewPreprint
Guo, C., & Wu, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01693 · Cited by 59
Chen, L., Li, M., Wu, M., Pedrycz, W., & Hirota, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3236320 · Cited by 53
Multimodal Representation for Neural Code Search
Gu, J. H., Chen, Z., & Monperrus, M. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00049 · Cited by 67
Multi-modal visual tracking: Review and experimental comparison
Zhang, P., Wang, D., & Lu, H. (2024) · Computational Visual Media · doi:10.1007/s41095-023-0345-5 · Cited by 61
Muroga, S., Miki, Y., & Hata, K. (2023) · Advanced Science · doi:10.1002/advs.202302508 · Cited by 57
ALIP: Adaptive Language-Image Pre-training with Synthetic Caption
Yang, K., Deng, J., An, X., Li, J., Feng, Z., Guo, J., Yang, J., & Liu, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00273 · Cited by 99
Multimodal Machine Learning for Stroke Prognosis and Diagnosis: A Systematic Review
Shurrab, S., Guerra-Manzanares, A., Magid, A., Piechowski‐Jóźwiak, B., Atashzar, S. F., & Shamout, F. E. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3448238 · Cited by 58
DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis
Verma, S., Wang, C., Zhu, L., & Liu, W. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/503 · Cited by 56
Web Table Retrieval using Multimodal Deep Learning
Shraga, R., Roitman, H., Feigenblat, G., & Canim, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401120 · Cited by 53
Factorized Multimodal Transformer for Multimodal Sequential LearningPreprint
Zadeh, A., Mao, C., Shi, K., Zhang, Y., Liang, P. P., Poria, S., & Morency, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09826 · Cited by 59
Self-citationArtificial intelligence and machine learning in cell-free-DNA-based diagnostics
Tsui, W., Ding, S. C., Jiang, P., & Lo, Y. M. D. (2025) · Genome Research · doi:10.1101/gr.278413.123 · Cited by 57
MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17260 · Cited by 52
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.