Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 201–300 of 1,036
Luna-Jiménez, C., Kleinlein, R., Griol, D., Callejas, Z., Montero, J. M., & Fernández-Martínez, F. (2021) · Applied Sciences · doi:10.3390/app12010327 · Cited by 149
Plant Genotype to Phenotype Prediction Using Machine Learning
Danilevicz, M. F., Gill, M., Anderson, R., Batley, J., Bennamoun, M., Bayer, P. E., & Edwards, D. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.822173 · Cited by 150
NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
Adversarial Sticker: A Stealthy Attack Method in the Physical World
Wei, X., Guo, Y., & Yu, J. (2021) · arXiv · doi:10.48550/arxiv.2104.06728 · Cited by 133
Recognizing Emotions in Video Using Multimodal DNN Feature Fusion
Williams, J., Kleinegesse, S., Comanescu, R., & Radu, O. (2018) · Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML) · doi:10.18653/v1/w18-3302 · Cited by 132
Dynamic Meta-Embeddings for Improved Sentence Representations
Kiela, D., Wang, C., & Cho, K. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1176 · Cited by 157
Chen, Z., Chen, Y., Sun, Y., Tang, L., Zhang, L., Hu, Y., He, M., Li, Z., Cheng, S., Yuan, J., et al. (2024) · Signal Transduction and Targeted Therapy · doi:10.1038/s41392-024-01932-y · Cited by 137
Deep graph regularized non-negative matrix factorization for multi-view clustering
Li, J., Zhou, G., Qiu, Y., Wang, Y., Zhang, Y., & Xie, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.12.054 · Cited by 133
GraphMFT: A graph network based multimodal fusion technique for emotion recognition in conversation
Li, J., Wang, X., Lv, G., & Zeng, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126427 · Cited by 154
XBully: Cyberbullying Detection within a Multi-Modal Context
Cheng, L., Li, J., Silva, Y. N., Hall, D. L., & Liu, H. (2019) · Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining · doi:10.1145/3289600.3291037 · Cited by 121
EEG-Based Multimodal Emotion Recognition: A Machine Learning Perspective
Liu, H., Lou, T., Zhang, Y., Wu, Y., Xiao, Y., Jensen, C. S., & Zhang, D. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3369130 · Cited by 177
A Survey of Underwater Acoustic Target Recognition Methods Based on Machine Learning
Luo, X., Chen, L., Zhou, H., & Cao, H. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11020384 · Cited by 139
Machine Learning as a Tool for Hypothesis Generation
Ludwig, J., & Mullainathan, S. (2024) · The Quarterly Journal of Economics · doi:10.1093/qje/qjad055 · Cited by 191
Ullah, I., Adhikari, D., Su, X., Palmieri, F., Wu, C., & Choi, C. (2024) · Digital Communications and Networks · doi:10.1016/j.dcan.2024.02.007 · Cited by 147
Multimodal Federated Learning: A Survey
Che, L., Wang, J., Zhou, Y., & Ma, F. (2023) · doi:10.20944/preprints202307.1420.v1 · Cited by 127
Deep learning for cross-domain data fusion in urban computing: Taxonomy, advances, and outlook
Zou, X., Yan, Y., Hao, X., Hu, Y., Wen, H., Liu, E., Zhang, J., Li, Y., Li, T., Zheng, Y., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102606 · Cited by 178
A bird's-eye view of deep learning in bioimage analysis
Meijering, E. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.003 · Cited by 126
Multimodal Federated Learning on IoT Data
Zhao, Y., Barnaghi, P. M., & Haddadi, H. (2022) · IEEE/ACM International Conference on Internet-of-Things Design and Implementation (IoTDI) · doi:10.1109/iotdi54339.2022.00011 · Cited by 170
Narotamo, H., Dias, M., Santos, R., Carreiro, A. V., Gamboa, H. F. S., & Silveira, M. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106141 · Cited by 126
Kabir, H., Tham, M., & Chang, Y. C. (2023) · Digital Communications and Networks · doi:10.1016/j.dcan.2023.05.006 · Cited by 143
A unified framework for multi-modal federated learning
Xiong, B., Yang, X., Qi, F., & Xu, C. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.063 · Cited by 158
Barua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 152
The Role of AI in Drug Discovery
Abbas, M. K. G., Rassam, A., Karamshahi, F., Abunora, R., & Abouseada, M. (2024) · ChemBioChem · doi:10.1002/cbic.202300816 · Cited by 141
Markus, B., Gruber Christian, C., Andreas, K., Kummer, A., Stefan, L., Oberdorfer, G., Elina, S., & Šnajdrová, R. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c03417 · Cited by 128
Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects
Warner, E., Lee, J., Hsu, W., Syeda-Mahmood, T., Kahn, C. E., Gevaert, O., & Rao, A. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02032-8 · Cited by 91
Machine learning on neutron and x-ray scattering and spectroscopies
Chen, Z., Andrejevic, N., Drucker, N. C., Nguyen, T., Xian, R. P., Smidt, T., Wang, Y., Ernstorfer, R., Tennant, D. M., Chan, M. K. Y., et al. (2021) · Chemical Physics Reviews · doi:10.1063/5.0049111 · Cited by 133
Ye, J., Tian, J., Yan, M., Yang, X., Wang, X., Zhang, J., He, L., & Lin, X. A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01506 · Cited by 154
From single- to multi-modal remote sensing imagery interpretation: a survey and taxonomy
Sun, X., Tian, Y., Lu, W., Wang, P., Niu, R., Yu, H., & Fu, K. (2023) · Science China Information Sciences · doi:10.1007/s11432-022-3588-0 · Cited by 126
Smart Wearables for the Detection of Cardiovascular Diseases: A Systematic Literature Review
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Sensors · doi:10.3390/s23020828 · Cited by 120
Zhang, Y., Li, Q., Tu, W., Mai, K., Yao, Y., & Chen, Y. (2019) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2019.101374 · Cited by 122
Deep learning for wildfire risk prediction: Integrating remote sensing and environmental data
Xu, Z., Li, J., Cheng, S. L., Xue, R., Zhao, Y., He, H., Guan, H., Sharma, A., Erxleben, M., Chang, R., et al. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.06.002 · Cited by 130
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 150
Modality-Invariant Asymmetric Networks for Cross-Modal Hashing
Zhang, Z., Luo, H., Zhu, L., Lu, G., & Shen, H. T. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3144352 · Cited by 150
Peng, L., Jian, S., Kan, Z., Qiao, L., & Li, D. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103564 · Cited by 140
Exploring Research Trends of Emerging Technologies in Health Metaverse: A Bibliometric AnalysisPreprint
Chen, D., & Zhang, R. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.3998068 · Cited by 129
A survey and taxonomy of adversarial neural networks for text‐to‐image synthesis
Agnese, J., Herrera, J., Tao, H., & Zhu, X. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1345 · Cited by 111
Xiao, M., & Yi, H. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22235 · Cited by 92
Multi-Modal Domain Adaptation Variational Autoencoder for EEG-Based Emotion Recognition
Wang, Z., Qiu, S., Li, D., Du, C., Lu, B., & He, H. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105515 · Cited by 126
A Review of Optical and SAR Image Deep Feature Fusion in Semantic Segmentation
Liu, C., Sun, Y., Xu, Y., Sun, Z., Zhang, X., Lei, L., & Kuang, G. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3424831 · Cited by 131
Transformers in speech processing: Overcoming challenges and paving the future
Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 126
Machine Learning for Human Learners: Opportunities, Issues, Tensions and Threats
Webb, M. E., Fluck, A., Magenheim, J., Malyn-Smith, J., Waters, J., Deschênes, M., & Zagami, J. (2021) · Educational Technology Research and Development · doi:10.1007/s11423-020-09858-2 · Cited by 116
Dentamaro, V., Impedovo, D., Musti, L., Pirlo, G., & Taurisano, P. (2024) · Scientific Reports · doi:10.1038/s41598-024-70165-4 · Cited by 116
Multiview learning for understanding functional multiomics
Nguyen, N. D., & Wang, D. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007677 · Cited by 107
MS²-GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection
Chen, T., Hong, R., Guo, Y., Hao, S., & Hu, B. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3197127 · Cited by 106
Recent Advances and Trends in Multimodal Deep Learning: A ReviewPreprint
Jabeen, S., Li, X., Amin, M. S., Li, S., & Abdul, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11087 · Cited by 141
Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial Learning
Zheng, C., Wu, Z., Wang, T., Cai, Y., & Li, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3013398 · Cited by 122
A Multimodal Misinformation Detector for COVID-19 Short Videos on TikTok
Shang, L., Kou, Z., Zhang, Y., & Wang, D. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671928 · Cited by 176
The Missing Ingredient in Zero-Shot Neural Machine TranslationPreprint
Arivazhagan, N., Bapna, A., Fırat, O., Aharoni, R., Johnson, M., & Macherey, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.07091 · Cited by 145
Turay, T., & Vladimirova, T. (2022) · IEEE Access · doi:10.1109/access.2022.3147495 · Cited by 136
Multimodal federated learning: Concept, methods, applications and future directions
Huang, W. X., Wang, D., Ouyang, X., Wan, J., Liu, J., & Li, T. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102576 · Cited by 132
Self-Supervised Multi-Modal Hybrid Fusion Network for Brain Tumor Segmentation
Fang, F., Yao, Y., Zhou, T., Xie, G.-S., & Lu, J. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3109301 · Cited by 127
Text categorization: past and present
Dhar, A., Mukherjee, H., Dash, N. S., & Roy, K. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09919-1 · Cited by 122
Deep learning model for classifying endometrial lesions
Zhang, Y.-Z., Wang, Z., Zhang, J., Wang, C., Wang, Y., Chen, H., Shan, L., Huo, J., Gu, J., & Ma, X. (2021) · Journal of Translational Medicine · doi:10.1186/s12967-020-02660-x · Cited by 111
Shankar, V., & Parsana, S. (2022) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-022-00840-3 · Cited by 117
Semi-Supervised Multiview Feature Selection With Adaptive Graph Learning
Jiang, B., Wu, X., Zhou, X., Liu, Y., Cohn, A. G., Sheng, W., & Chen, H. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3194957 · Cited by 125
Credit risk prediction of SMEs in supply chain finance by fusing demographic and behavioral data
Zhang, W., Yan, S., Li, J., Tian, X., & Yoshida, T. (2022) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2022.102611 · Cited by 125
Xiao, L., Wu, X., Xu, J., Li, W., Jin, C., & He, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102304 · Cited by 131
A Survey of Multi-modal Knowledge Graphs: Technologies and Trends
Liang, W., Meo, P. D., Tang, Y. Z., & Zhu, J. (2024) · ACM Computing Surveys · doi:10.1145/3656579 · Cited by 137
A survey of multimodal deep generative models
Suzuki, M., & Matsuo, Y. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2035253 · Cited by 109
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
BackgroundMethodInfluentialCorti, C., Cobanaj, M., Dee, E. C., Criscitiello, C., Tolaney, S. M., Celi, L. A., & Curigliano, G. (2022) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2022.102498 · Cited by 121
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Zadeh, A., Chan, M., Liang, P. P., Tong, E., & Morency, L.-P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00901 · Cited by 91
Self-citationShi, Y., Zhao, Y., Liu, X., Zheng, F., Ou, W., You, X., & Peng, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3172716 · Cited by 113
Sun, D., Li, Y., Liu, Z., Jia, S., & Noman, K. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102394 · Cited by 110
Investigating Audio, Video, and Text Fusion Methods for End-to-End Automatic Personality Prediction
Kampman, O. P., Barezi, E. J., Bertero, D., & Fung, P. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2096 · Cited by 127
Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition
Ghaleb, E., Popa, M., & Asteriadis, S. (2019) · 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2019.8925444 · Cited by 93
Multi-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 121
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 130
Rodis, N., Sardianos, C., Grammatikis, P. I. R., Sarigiannidis, P. G., Varlamis, I., & Papadopoulos, G. T. (2024) · IEEE Access · doi:10.1109/access.2024.3467062 · Cited by 123
Federated Self-Supervised Learning of Multisensor Representations for Embedded Intelligence
Saeed, A., Salim, F. D., Ozcelebi, T., & Lukkien, J. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3009358 · Cited by 106
Multi-View Information-Bottleneck Representation Learning
Wan, Z., Zhang, C., Zhu, P., & Hu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17210 · Cited by 131
Wang, Y., Feng, Y., Zhang, L., Zhou, J. T., Liu, Y., Goh, R. S. M., & Zhen, L. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102535 · Cited by 114
Ma, T., & Zhang, A. (2019) · BMC Genomics · doi:10.1186/s12864-019-6285-x · Cited by 105
Zanfardino, M., Franzese, M., Pane, K., Cavaliere, C., Monti, S., Esposito, G., Salvatore, M., & Aiello, M. (2019) · Journal of Translational Medicine · doi:10.1186/s12967-019-2073-2 · Cited by 96
Multimodal machine translation through visuals and speech
Sulubacak, U., Caglayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2019) · Machine Translation · doi:10.1007/s10590-020-09250-0 · Cited by 105
Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis
Xiao, L., Wu, X., Yang, S., Xu, J., Zhou, J., & He, L. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103508 · Cited by 131
Enhancing Dyadic Relations with Homogeneous Graphs for Multimodal Recommendation
Zhou, H., Zhou, X., Zhang, L., & Shen, Z. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230631 · Cited by 138
Sound Active Attention Framework for Remote Sensing Image Captioning
Lu, X., Wang, B., & Zheng, X. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2951636 · Cited by 108
Liu, W., Zheng, W., Li, Z., Wu, S., Gan, L., & Lu, B. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac5c8d · Cited by 121
Artificial intelligence-assisted dermatology diagnosis: From unimodal to multimodal
Luo, N., Zhong, X., Su, L., Cheng, Z., Ma, W., & Hao, P. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107413 · Cited by 115
M3Care: Learning with Missing Modalities in Multimodal Healthcare Data
Zhang, C., Chu, X., Ma, L., Zhu, Y., Wang, Y., Wang, J., & Zhao, J. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539388 · Cited by 102
The Next Grand Challenges: Integrating the Internet of Things and Data Science
Ranjan, R. K., RANA, O. F., Nepal, ., Yousif, M. S., James, P., Wen, Z., Barr, S. L., Watson, P., Jayaraman, P. P., Georgakopoulos, D., et al. (2018) · IEEE Cloud Computing · doi:10.1109/mcc.2018.032591612 · Cited by 102
M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
Wang, X., He, J., Jin, Z., Yang, M., Wang, Y., & Qu, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114794 · Cited by 91
AF: An Association-Based Fusion Method for Multi-Modal Classification
Liang, X., Qian, Y., Guo, Q. J., Cheng, H., & Liang, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3125995 · Cited by 118
Machine Learning: A Review of Learning TypesPreprint
Sah, S. (2020) · Preprints.org · doi:10.20944/preprints202007.0230.v1 · Cited by 128
Predicting MCI Status From Multimodal Language Data Using Cascaded Classifiers
Fraser, K. C., Lundholm Fors, K., Eckerström, M., Öhman, F., & Kokkinakis, D. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00205 · Cited by 97
Zhou, D., Xu, Z., Li, W., Xie, X., & Peng, S. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab473 · Cited by 105
Data Harmonization for Heterogeneous Datasets: A Systematic Literature Review
Kumar, G., Basri, S. B., Imam, A. A., Khowaja, S. A., Capretz, L. F., & Balogun, A. O. (2021) · Applied Sciences · doi:10.3390/app11178275 · Cited by 120
MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00628 · Cited by 137
Ensemble Large Language Models: A Survey
Mienye, I. D., & Swart, T. G. (2025) · Information · doi:10.3390/info16080688 · Cited by 111
Li, B., Fei, H., Liao, L., Zhao, Y., Teng, C., Chua, T., Ji, D., & Li, F. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612053 · Cited by 158
Human Action Recognition from Various Data Modalities: A ReviewPreprint
Rahmani, H., Bennamoun, M., & Ke, Q. (2021) · doi:10.36227/techrxiv.13708270 · Cited by 105
Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification
Lee, S., Han, D. K., & Ko, H. (2021) · IEEE Access · doi:10.1109/access.2021.3092735 · Cited by 89
Buyl, M., Cociancig, C., Frattone, C., & Roekens, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533169 · Cited by 119
SKEAFN: Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis
Zhu, C., Chen, M., Zhang, S., Sun, C., Liang, H., Liu, Y., & Chen, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101958 · Cited by 109
Gemini-the most powerful LLM: Myth or Truth
Islam, R., & Ahmed, I. (2024) · Information Communication Technologies Conference (ICTC) · doi:10.1109/ictc61510.2024.10602253 · Cited by 118
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
Tsai, Y.-H. H., Ma, M., Yang, M., Salakhutdinov, R., & Morency, L.-P. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.143 · Cited by 98
Self-citationA survey on visual navigation for artificial agents with deep reinforcement learning
Zeng, F., Wang, C., & Ge, S. S. (2020) · IEEE Access · doi:10.1109/access.2020.3011438 · Cited by 98
Comparative analysis on cross-modal information retrieval: A review
Kaur, P., Pannu, H. S., & Malhi, A. K. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100336 · Cited by 125
Multimodal Artificial Intelligence in Medical Diagnostics
Jandoubi, B., & Akhloufi, M. A. (2025) · Information · doi:10.3390/info16070591 · Cited by 115
Islam, M. M., & Iqbal, T. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3059624 · Cited by 92
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.