Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 301–400 of 1,036
Wannan, C., Nelson, B., Addington, J., Allott, K., Anticevic, A., Arango, C., Baker, J. T., Bearden, C. E., Billah, T., Bouix, S., et al. (2024) · Schizophrenia Bulletin · doi:10.1093/schbul/sbae011 · Cited by 102
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 192
Gkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 118
A Multimodality Fusion Deep Neural Network and Safety Test Strategy for Intelligent Vehicles
Nie, J., Yan, J., Yin, H., Ren, L., & Meng, Q. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2020.3027319 · Cited by 103
Mahajan, P., & Baths, V. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.623607 · Cited by 89
A systematic review of intermediate fusion in multimodal deep learning for biomedical applications
Guarrasi, V., Aksu, F., Caruso, C. M., Di Feola, F., Rofena, A., Ruffini, F., & Soda, P. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105509 · Cited by 95
Razi, A., Kim, S., Alsoubai, A., Stringhini, G., Solorio, T., De Choudhury, M., & Wisniewski, P. J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479609 · Cited by 95
HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm
Islam, M., & Iqbal, T. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340987 · Cited by 124
Stappen, L., Baird, A., Schumann, L., & Schuller, B. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3097002 · Cited by 91
Uncertainty-Aware Multi-View Representation Learning
Geng, Y., Han, Z., Zhang, C., & Hu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16924 · Cited by 138
Multimodal Emotion Recognition Using Deep Canonical Correlation AnalysisPreprint
Liu, W., Qiu, J., Zheng, W., & Lu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05349 · Cited by 120
Artificial intelligence for clinical decision support in neurology
Pedersen, M., Verspoor, K., Jenkinson, M., Law, M., Abbott, D. F., & Jackson, G. D. (2020) · Brain Communications · doi:10.1093/braincomms/fcaa096 · Cited by 85
Muzammel, M., Salam, H., & Othmani, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106433 · Cited by 114
Detection of Depression Severity in Social Media Text Using Transformer-Based Models
Qasim, A., Mehak, G., Hussain, N., Gelbukh, A. F., & Sidorov, G. (2025) · Information · doi:10.3390/info16020114 · Cited by 107
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93
“Additional organizational or technical challenges regarding multi-modal machine learning and AI in healthcare have previously been pointed out 20,44,52 .”
Review and Analysis of Zero, One and Few Shot Learning Approaches
Kadam, S., & Vaidya, V. G. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-16657-1_10 · Cited by 111
Affective Computing for Large-scale Heterogeneous Multimedia Data: A Survey
Zhao, S., Wang, S., Soleymani, M., Joshi, D., & Ji, Q. (2019) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3363560 · Cited by 77
Artificial Intelligence for Personalized Preventive Adolescent Healthcare
Rowe, J. P., & Lester, J. C. (2020) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2020.02.021 · Cited by 83
Deep cross-omics cycle attention model for joint analysis of single-cell multi-omics data
Zuo, C., Dai, H., & Chen, L. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab403 · Cited by 88
Deep Learning for Technical Document Classification
Jiang, S., Hu, J., Magee, C. L., & Luo, J. (2022) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2022.3152216 · Cited by 111
Multimodality in meta-learning: A comprehensive survey
Ma, Y., Zhao, S., Wang, W., Li, Y., & King, I. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108976 · Cited by 99
Li, X., Guo Zhang, A., Cui, H., Hou, S., Chen, Y., Li, Z., Li, H., & Wang, H. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.11.015 · Cited by 96
Label-efficient learning in agriculture: A comprehensive review
Li, J., Chen, D., Qi, X., Li, Z., Huang, Y., Morris, D., & Tan, X. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108412 · Cited by 107
Tang, J., Li, K., Jin, X., Cichocki, A. S., Zhao, Q., & Kong, W. (2021) · doi:10.18653/v1/2021.acl-long.412 · Cited by 92
Lithium-Ion Batteries SOH Estimation With Multimodal Multilinear Feature Fusion
Lin, M., You, Y., Meng, J., Wang, W., Wu, J., & Stroe, D.-I. (2023) · IEEE Transactions on Energy Conversion · doi:10.1109/tec.2023.3282017 · Cited by 87
Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment AnalysisPreprint
Chen, F., Luo, Z., Xu, Y., & Ke, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08138 · Cited by 108
Multi-Modal Neuroimaging Feature Fusion for Diagnosis of Alzheimer’s Disease
Zhang, T., & Shi, M. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2020.108795 · Cited by 91
Multimodal End-to-End Sparse Model for Emotion Recognition
Dai, W., Cahyawijaya, S., Liu, Z., & Fung, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.417 · Cited by 91
Tang, J., Ma, Z., Gan, K., Zhang, J., & Yin, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102129 · Cited by 96
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90
A cross-modal generative adversarial network for scenarios generation of renewable energy
Kang, M., Zhu, R., Chen, D., Li, C., Gu, W., Qian, X., & Yu, W. (2024) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2023.3277698 · Cited by 90
Zheng, L., Wang, H., Li, M., Chen, Q., Zhang, Y., & Zhang, H. (2021) · Annals of Translational Medicine · doi:10.21037/atm-21-119 · Cited by 90
Zhang, Z., Song, X., Liu, L., Yin, J., Wang, Y., & Lan, D. (2021) · Security and Communication Networks · doi:10.1155/2021/9991535 · Cited by 113
Toward Explainable Affective Computing: A Review
Cortiñas-Lorenzo, K., & Lacey, G. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270027 · Cited by 112
Zhang, L., Wilson, R., Sumner, M., & Wu, Y. (2023) · Renewable Energy · doi:10.1016/j.renene.2023.118952 · Cited by 97
Large circuit models: opportunities and challenges
Chen, L., Chen, Y., Chu, Z., Fang, W., Ho, T.-Y., Huang, R., Huang, Y., Khan, S., Li, M., Li, X., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4155-7 · Cited by 99
Mai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89
Mai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89
Holmberg, O. G., Köhler, N. D., Martins, T., Siedlecki, J., Herold, T., Keidel, L., Asani, B., Schiefelbein, J., Priglinger, S., Kortuem, K. U., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00247-1 · Cited by 88
Generative Adversarial Networks: A Literature Review
Cheng, J., Yang, Y., Tang, X., Xiong, N. N., Zhang, Y., & Lei, F. (2020) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2020.12.001 · Cited by 126
A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications
Zhou, X., Liu, H., Pourpanah, F., Zeng, T., & Wang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.119 · Cited by 114
Noise Imitation Based Adversarial Training for Robust Multimodal Sentiment Analysis
Yuan, Z., Liu, Y., Xu, H., & Gao, K. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3267882 · Cited by 119
Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion
Zhou, Y., Yang, Y., Ying, Q., Qian, Z., & Zhang, X. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592271 · Cited by 117
Advanced chemometrics toward robust spectral analysis for fruit quality evaluation
Zhang, X., & Yang, J. (2024) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2024.104612 · Cited by 89
Joint feature synthesis and embedding: Adversarial cross-modal retrieval revisited
Xu, X., Lin, K., Yang, Y., Hanjalic, A., & Shen, H. T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045530 · Cited by 88
Sadr, H., Pedram, M. M., & Teshnehlab, M. (2020) · IEEE Access · doi:10.1109/access.2020.2992063 · Cited by 94
Mashup-Oriented Web API Recommendation via Multi-Model Fusion and Multi-Task Learning
Wu, H., Duan, Y., Yue, K., & Zhang, L. (2021) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2021.3098756 · Cited by 88
A Survey on Multi-modal Summarization
Jangra, A., Mukherjee, S., Jatowt, A., Saha, S., & Hasanuzzaman, M. (2023) · ACM Computing Surveys · doi:10.1145/3584700 · Cited by 116
Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition
Zhang, Y., Chen, M., Shen, J., & Wang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20895 · Cited by 119
DIP: Dual Incongruity Perceiving Network for Sarcasm Detection
Wen, C., Jia, G., & Yang, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00250 · Cited by 88
Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 129
Has multimodal learning delivered universal intelligence in healthcare? A comprehensive survey
Lin, Q., Zhu, Y., Mei, X., Huang, L., Ma, J., He, K., Peng, Z., Cambria, E., & Feng, M. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102795 · Cited by 117
Domain Generalization in Computational Pathology: Survey and Guidelines
Jahanifar, M., Raza, M., Xu, K., Vuong, T. T. L., Jewsbury, R., Shephard, A., Tajeddin, N. Z., Kwak, J. T., Raza, S. E. A., Minhas, F., et al. (2025) · ACM Computing Surveys · doi:10.1145/3724391 · Cited by 108
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 95
Adapted Dynamic Memory Network for Emotion Recognition in Conversation
Xing, S., Mai, S., & Hu, H. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3005660 · Cited by 95
Audio- and Gaze-driven Facial Animation of Codec Avatars
Richard, A., Lea, C., Ma, S., Gall, J., la Torre, F. D., & Sheikh, Y. A. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00009 · Cited by 112
Eckardt, J., Bornhäuser, M., Wendt, K., & Middeke, J. M. (2020) · Blood Advances · doi:10.1182/bloodadvances.2020002997 · Cited by 96
Multimodal Triplet Attention Network for Brain Disease Diagnosis
Zhu, Q., Wang, H., Xu, B., Zhang, Z., Shao, W., & Zhang, D. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3199032 · Cited by 80
Machine Learning for Emergency Management: A Survey and Future Outlook
Kyrkou, C., Kolios, P., Theocharides, T., & Polycarpou, M. M. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3223186 · Cited by 100
A survey on machine learning based analysis of heterogeneous data in industrial automation
Kamm, S., Veekati, S. S., Müller, T., Jazdi, N., & Weyrich, M. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103930 · Cited by 99
Singh, R., Nisha, R., Naik, R., Upendar, K., Nickhil, C., & Deka, S. C. (2024) · Journal of Food Measurement & Characterization · doi:10.1007/s11694-024-02789-z · Cited by 87
Zhang, X., Pan, J., Shen, J. G., Din, Z. U., Li, J., Lu, D., Wu, M., & Hu, B. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981440 · Cited by 95
Multi-Modality Sensing and Data Fusion for Multi-Vehicle Detection
Roy, D., Li, Y., Jian, T., Tian, P., Chowdhury, K. R., & Ioannidis, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3145663 · Cited by 97
A guidebook of spatial transcriptomic technologies, data resources and analysis approaches
Yue, L., Liu, F., Hu, J., Yang, P., Wang, Y., Dong, J., Shu, W., Huang, X., & Wang, S. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.01.016 · Cited by 95
Visuo-haptic object perception for robots: an overview
Navarro-Guerrero, N., Toprak, S., Josifovski, J., & Jamone, L. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10091-y · Cited by 105
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85
“…performs concatenation of input data before using a model (i.e., z mm = [ x 1 , x 2 ] ) while late fusion applies suitable unimodal models to each modality to obtain their feature representations, concatenates these features, and defines a classifier to the label (i.e., z mm = [ z 1 , z 2 ] ) [10].”
BackgroundMethodInfluentialSelf-citationAligning Latent Spaces for 3D Hand Pose Estimation
Yang, L., Li, S., Lee, D., & Yao, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00242 · Cited by 98
Robust Deep Multi-modal Learning Based on Gated Information Fusion Network
Kim, J., Koh, J., Kim, Y., Choi, J., Hwang, Y., & Choi, J. W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_6 · Cited by 102
ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
Taleb, A., Kirchler, M., Monti, R., & Lippert, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02024 · Cited by 83
(2019) · neural information processing systems · Cited by 83
Visual cognition in multimodal large language models
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00963-y · Cited by 83
DeepStealth: Game-Based Learning Stealth Assessment With Deep Neural Networks
Min, W., Frankosky, M. H., Mott, B., Rowe, J., Smith, A., Wiebe, E., Boyer, K. E., & Lester, J. C. (2019) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2019.2922356 · Cited by 88
Wei, S., Luo, C., & Luo, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01919 · Cited by 136
Toward Visual Grounding: A Survey
Xiao, L., Yang, X., Lan, X., Wang, Y., & Xu, C. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3630635 · Cited by 115
Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Pham, H., Manzini, T., Liang, P. P., & Poczós, B. (2018) · Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML) · doi:10.18653/v1/w18-3308 · Cited by 76
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81
TorchGeo: deep learning with geospatial data
Stewart, A. J., Robinson, C., Corley, I., Ortiz, A., Ferres, J. L., & Banerjee, A. (2022) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3557915.3560953 · Cited by 120
Tellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 66
Vision-language navigation: a survey and taxonomy
Wu, W., Chang, T., Li, X., Yin, Q., & Hu, Y. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-09217-1 · Cited by 129
Boosting Multi-modal Model Performance with Adaptive Gradient Modulation
Li, H., Li, X., Hu, P., Lei, Y., Li, C., & Zhou, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02030 · Cited by 167
An Efficient Approach to Informative Feature Extraction from Multimodal Data
Wang, L., Wu, J., Huang, S., Zheng, L., Xu, X., Zhang, L., & Huang, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015281 · Cited by 97
Multimodal sentiment analysis based on multi-head attention mechanism
Xi, C., Lu, G., & Yan, J. (2020) · Proceedings of the 4th International Conference on Machine Learning and Soft Computing · doi:10.1145/3380688.3380693 · Cited by 72
SAR Image Change Detection Based on Multiscale Capsule Network
Gao, Y., Gao, F., Dong, J., & Li, H.-C. (2021) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2020.2977838 · Cited by 80
Wagan, S. A., Koo, J., Siddiqui, I. F., Attique, M., Shin, D. R., & Qureshi, N. M. F. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.09.005 · Cited by 87
Multimodal brain tumor detection using multimodal deep transfer learning
Razzaghi, P., Abbasi, K., Shirazi, M., & Rashidi, S. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109631 · Cited by 84
A survey of identity recognition via data fusion and feature learning
Qin, Z., Zhao, P., Zhuang, T., Deng, F., Ding, Y., & Chen, D. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.032 · Cited by 86
An image and text-based multimodal model for detecting fake news in OSN’s
Uppada, S. K., Patel, P. K., & Sivaselvan, B. (2022) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00764-y · Cited by 90
Driver Emotion Recognition With a Hybrid Attentional Multimodal Fusion Framework
Mou, L., Zhao, Y., Zhou, C., Nakisa, B., Rastgoo, M. N., Ма, Л., Huang, T., Yin, B., Jain, R., & Gao, W. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3250460 · Cited by 90
Federated Learning on Multimodal Data: A Comprehensive Survey
Lin, Y., Gao, Y., Gong, M., Zhang, S., Zhang, Y., & Li, Z. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1398-0 · Cited by 97
Opportunities and Challenges in Data-Centric AI
Kumar, S., Datta, S., Singh, V., Singh, S. K., & Sharma, R. (2024) · IEEE Access · doi:10.1109/access.2024.3369417 · Cited by 111
Including Signed Languages in Natural Language Processing
Yin, K., Moryossef, A., Hochgesang, J., Goldberg, Y., & Alikhani, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.570 · Cited by 79
Multiview Concept Learning Via Deep Matrix Factorization
Zhao, W. Z., Xu, C. X., Guan, Z., & Liu, Y. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979532 · Cited by 79
VSR: A Unified Framework for Document Layout Analysis Combining Vision, Semantics and Relations
Zhang, P., Li, C., Liang, Q., Cheng, Z., Pu, S., Niu, Y., & Wu, F. Q. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86549-8_8 · Cited by 102
Xu, Y., Su, H., Ma, G., & Liu, X. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-022-00841-3 · Cited by 79
MSAF: Multimodal Split Attention FusionPreprint
Su, L., Hu, C., Li, G., & Cao, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07175 · Cited by 91
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 78
“Multimodal learning (MML) [4, 63] is a general framework for processing, combining, and understanding information from multiple, diverse data sources.”
BackgroundBias in Multimodal AI: Testbed for Fair Automatic Recruitment
Pena, A., Serna, I., Morales, A., & Fierrez, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00022 · Cited by 64
Cross-Subject and Cross-Modal Transfer for Generalized Abnormal Gait Pattern Recognition
Gu, X., Guo, Y., Deligianni, F., Lo, B. P. L., & Yang, G. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3009448 · Cited by 83
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 91
Sentiment Analysis: Comprehensive Reviews, Recent Advances, and Open Challenges
Lu, Q., Sun, X., Long, Y., Gao, Z., Feng, J. H., & Sun, T. S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294810 · Cited by 89
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.