Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 401–500 of 1,036
Cell2Sentence: Teaching Large Language Models the Language of BiologyPreprint
Levine, D. M., Rizvi, S. A., Lévy, S., Pallikkavaliyaveetil, N., Zhang, D., Chen, X., Ghadermarzi, S., Wu, R., Zheng, Z., Vrkic, I., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.09.11.557287 · Cited by 122
Pei, X., Zuo, K., Li, Y., & Pang, Z. (2023) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-023-00225-6 · Cited by 77
Li, J., & Wang, Q. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.018 · Cited by 77
Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition
Sterpu, G., Saam, C., & Harte, N. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243014 · Cited by 90
TCGA-TCIA Impact on Radiogenomics Cancer Research: A Systematic Review
Zanfardino, M., Pane, K., Mirabelli, P., Salvatore, M., & Franzese, M. (2019) · International Journal of Molecular Sciences · doi:10.3390/ijms20236033 · Cited by 74
Deep learning for bioimage analysis in developmental biology
Hallou, A., Yevick, H. G., Dumitrascu, B., & Uhlmann, V. (2021) · Development · doi:10.1242/dev.199616 · Cited by 84
Deep learning-based multi-task prediction system for plant disease and species detection
Keçeli, A. S., Kaya, A., Catal, C., & Tekinerdoğan, B. (2022) · Ecological Informatics · doi:10.1016/j.ecoinf.2022.101679 · Cited by 77
Reviewing Multimodal Machine Learning and Its Use in Cardiovascular Diseases Detection
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Electronics · doi:10.3390/electronics12071558 · Cited by 85
Harmony: Heterogeneous Multi-Modal Federated Learning through Disentangled Model Training
Ouyang, X., Xie, Z., Fu, H., Cheng, S., Pan, L., Ling, N., Xing, G., Zhou, J., & Huang, J. (2023) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3581791.3596844 · Cited by 110
Zhao, B., Richardson, R. E., & You, F. (2024) · Nexus · doi:10.1016/j.ynexs.2024.100043 · Cited by 89
Multi-Modal Sequence Fusion via Recursive Attention for Emotion Recognition
Beard, R., Das, R., Ng, R. W. M., Gopalakrishnan, P. G. K., Eerens, L., Świętojański, P., & Mikšík, O. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1025 · Cited by 87
Data-Driven Modeling of Pregnancy-Related Complications
Espinosa, C. A., Becker, M., Marić, I., Wong, R. J., Shaw, G. M., Gaudilliere, B. L., Aghaeepour, N., Stevenson, D. K., Stelzer, I. A., Peterson, L. S., et al. (2021) · Trends in Molecular Medicine · doi:10.1016/j.molmed.2021.01.007 · Cited by 83
Beyond word embeddings: A survey
Incitti, F., Urli, F., & Snidaro, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.08.024 · Cited by 97
Deep Multi-Modal Discriminative and Interpretability Network for Alzheimer’s Disease Diagnosis
Zhu, Q., Xu, B., Huang, J., Wang, H., Xu, R., Shao, W., & Zhang, D. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3230750 · Cited by 76
Applications of deep learning in understanding gene regulation
Li, Z., Gao, E., Zhou, J., Han, W., Xu, X., & Gao, X. (2023) · Cell Reports Methods · doi:10.1016/j.crmeth.2022.100384 · Cited by 92
Multi-Task Momentum Distillation for Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3282410 · Cited by 95
Fusing pairwise modalities for emotion recognition in conversations
Fan, C., Lin, J., Mao, R., & Cambria, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102306 · Cited by 86
Deep transfer network for rotating machine fault analysis
Qian, W., Li, S., & Jiang, X. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.106993 · Cited by 78
Danilevicz, M. F., Bayer, P. E., Boussaid, F., Bennamoun, M., & Edwards, D. (2021) · Remote Sensing · doi:10.3390/rs13193976 · Cited by 75
Multi-source Multimodal Data and Deep Learning for Disaster Response: A Systematic Review
Algiriyage, N., Prasanna, R., Stock, K. M., Doyle, E. E. H., & Johnston, D. M. (2021) · SN Computer Science · doi:10.1007/s42979-021-00971-4 · Cited by 88
MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System
Qin, L., Huang, S., Chen, Q., Cai, C., Zhang, Y., Liang, B., Che, W., & Xu, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.689 · Cited by 140
Research Progress of EEG-Based Emotion Recognition: A Survey
Wang, Y., Zhang, B., & Di, L. (2024) · ACM Computing Surveys · doi:10.1145/3666002 · Cited by 88
Zhang, Y., He, K., Guo, Y., Liu, X., Yang, Q., Zhang, C., Xie, Y., Mu, S., Guo, Y., Fu, Y., et al. (2020) · Frontiers in Oncology · doi:10.3389/fonc.2020.00457 · Cited by 77
Disaster Image Classification by Fusing Multimodal Social Media Data
Zou, Z., Gan, H., Huang, Q., Cai, T., & Cao, K. (2021) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi10100636 · Cited by 84
Wang, K., Yang, X., Miao, J., Li, L., Yao, J., Zhou, P., Xue, W., Zhou, G., Zhuang, X., & Ni, D. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102362 · Cited by 81
Lei, B., Li, Y., Fu, W., Yang, P., Chen, S., Wang, T., Xiao, X., Niu, T., Fu, Y., Wang, S., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103213 · Cited by 73
Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention
Duan, B., Tang, H., Wang, W., Zong, Z., Yang, G., & Yan, Y. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00406 · Cited by 73
A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition
Roche, J., Silva, V. D., Hook, J., Moencks, M., & KONDOZ, A. M. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3085489 · Cited by 81
Lu, Q., Sun, X., Gao, Z., Long, Y., Feng, J., & Zhang, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103538 · Cited by 80
Qu, X., Liu, Z., Wu, C. Q., Hou, A., Yin, X., & Chen, Z. (2024) · Sensors · doi:10.3390/s24020637 · Cited by 85
C 2 KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation
Huo, F., Xu, W., Guo, J., Wang, H., & Guo, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01515 · Cited by 116
Multimodal Agent AI: A Survey of Recent Advances and Future Directions
Sun, Y., Sun, H., Ma, J., Zhang, P., & Huang, X. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-4802-8 · Cited by 79
Sentiment analysis of social images via hierarchical deep fusion of content and links
Xu, J. J., Huang, F., Zhang, X., Wang, S., Li, C., Li, Z., & He, Y. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.04.010 · Cited by 76
Modality-DTA: Multimodality Fusion Strategy for Drug–Target Affinity Prediction
Yang, X., Niu, Z., Liu, Y., Song, B., Lü, W., Zeng, L., & Zeng, X. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3205282 · Cited by 75
Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-Based Recruitment
Peña, A., Serna, I., Morales, A., Fiérrez, J., Ortega, A., Sánchez, A. H., Saéz, M. A., & Ortega-García, J. (2023) · SN Computer Science · doi:10.1007/s42979-023-01733-0 · Cited by 98
Learn from View Correlation: An Anchor Enhancement Strategy for Multi-View Clustering
Liu, S., Liang, K., Dong, Z., Wang, S., Yang, X., Zhou, S., Zhu, E., & Liu, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02471 · Cited by 72
Pseudo Labeling Methods for Semi-Supervised Semantic Segmentation: A Review and Future Perspectives
Ran, L., Li, Y., Liang, G., & Zhang, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3508768 · Cited by 92
Multimodality Fusion Aspects of Medical Diagnosis: A Comprehensive Review
Kumar, S., Rani, S., Sharma, S., & Min, H. (2024) · Bioengineering · doi:10.3390/bioengineering11121233 · Cited by 86
Multimodal Music Information Processing and Retrieval: Survey and Future Challenges
Simonetta, F., Ntalampiras, S., & Avanzini, F. (2019) · International Workshop on Multilayer Music Representation and Processing (MMRP) · doi:10.1109/mmrp.2019.00012 · Cited by 96
Multiˆ2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT
Ro, Y., Lee, Y., & Kang, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.99 · Cited by 93
Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Ma, F., Li, Y., Ni, S., Huang, S., & Zhang, L. (2022) · Applied Sciences · doi:10.3390/app12010527 · Cited by 81
Simultaneously Optimizing Perturbations and Positions for Black-Box Adversarial Patch Attacks
Wei, X., Guo, Y., Yu, J., & Zhang, B. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3231886 · Cited by 98
Predictive Beamforming for Vehicles With Complex Behaviors in ISAC Systems: A Deep Learning Approach
Zhang, X., Yuan, W., Liu, C., Wu, J., & Ng, D. W. K. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3405856 · Cited by 81
Fake news detection for epidemic emergencies via deep correlations between text and images
Zeng, J., Zhang⋆, Y., & Ma, X. (2020) · Sustainable Cities and Society · doi:10.1016/j.scs.2020.102652 · Cited by 78
Multimodal Phased Transformer for Sentiment Analysis
Cheng, J., Fostiropoulos, I., BOEHM, B. W., & Soleymani, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.189 · Cited by 93
Analysis of multimodal data fusion from an information theory perspective
Dai, Y., Yan, Z., Cheng, J., Duan, X., & Wang, G. (2022) · Information Sciences · doi:10.1016/j.ins.2022.12.014 · Cited by 78
Robots for the people, by the people: Personalizing human-machine interaction
Clabaugh, C., & Matarić, M. (2018) · Science Robotics · doi:10.1126/scirobotics.aat7451 · Cited by 65
Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis
Qiu, J., Liu, W., & Lu, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-04221-9_20 · Cited by 78
Hu, D., Zhang, H., Wu, Z., Wang, F., Wang, L., Smith, J. K., Lin, W., Li, G., & Shen, D. G. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3013825 · Cited by 78
A Comparative Analysis of Early and Late Fusion for the Multimodal Two-Class Problem
Pereira, L. M., Salazar, A., & Vergara, L. (2023) · IEEE Access · doi:10.1109/access.2023.3296098 · Cited by 69
Joint discriminative feature learning for multimodal finger recognition
Li, S., Zhang, B., Fei, L., & Zhao, S. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107704 · Cited by 69
Li, R., Xiang, J., Sun, F., Yuan, Y., Yuan, L., & Gou, S. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3272471 · Cited by 80
Prediction of mechanical properties of composite materials using multimodal fusion learning
Song, L., Wang, D., Liu, X., Yin, A., & Long, Z. (2023) · Sensors and Actuators A Physical · doi:10.1016/j.sna.2023.114433 · Cited by 72
LSTM-based real-time action detection and prediction in human motion streams
Carrara, F., Eliáš, P., Sedmidubský, J., & Zezula, P. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-07827-3 · Cited by 69
Multimodal Fusion via Teacher-Student Network for Indoor Action Recognition
Yu, B. X. B., Liu, Y., & Chan, K. C. C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16430 · Cited by 82
Multimodal temporal machine learning for Bipolar Disorder and Depression Recognition
Ceccarelli, F., & Mahmoud, M. (2021) · Pattern Analysis and Applications · doi:10.1007/s10044-021-01001-y · Cited by 77
DeepVecFont: synthesizing high-quality vector fonts via dual-modality learning
Wang, Y., & Lian, Z. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480488 · Cited by 112
MEDT: Using Multimodal Encoding-Decoding Network as in Transformer for Multimodal Sentiment Analysis
Qi, Q., Lin, L., Zhang, R., & Xue, C. (2022) · IEEE Access · doi:10.1109/access.2022.3157712 · Cited by 74
A Survey of Multimodal Fake News Detection: A Cross-Modal Interaction Perspective
Li, X., Qiao, J., Yin, S., Wu, L., Gao, C., Wang, Z., & Li, X. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3543389 · Cited by 86
AI-Enabled Blockchain: An Outlier-Aware Consensus Protocol for Blockchain-Based IoT Networks
Salimitari, M., Joneidi, M., & Chatterjee, M. (2019) · Globecom · doi:10.1109/globecom38437.2019.9013824 · Cited by 73
On the Benefits of Early Fusion in Multimodal Representation LearningPreprint
Barnum, G., Talukder, S., & Yue, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07191 · Cited by 85
Emotion Based Hate Speech Detection using Multimodal LearningPreprint
Rana, A., & Jha, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06218 · Cited by 89
Fusion of tactile and visual information in deep learning models for object recognition
Babadian, R. P., Faez, K., Amiri, M., & Falotico, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.11.032 · Cited by 67
Ahuja, C., Ma, S., Morency, L., & Sheikh, Y. A. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353725 · Cited by 75
Self-citationLearning to Solve 3-D Bin Packing Problem via Deep Reinforcement Learning and Constraint Programming
Jiang, Y., Cao, Z., & Zhang, J. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3121542 · Cited by 81
Towards Optimal Multi-Modal Federated Learning on Non-IID Data with Hierarchical Gradient Blending
Chen, S., & Li, B. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796724 · Cited by 85
He, Q., Li, X., Kim, D. W. N., Jia, X., Gu, X., Zhen, X., & Zhou, L. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.09.001 · Cited by 74
Adversarial Multimodal Representation Learning for Click-Through Rate Prediction
Li, X., Wang, C., Tan, J., Zeng, X., Ou, D., & Zheng, B. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380163 · Cited by 75
Liu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66
Deep learning approaches and interventions for futuristic engineering in agriculture
Chakraborty, S. K., Chandel, N. S., Jat, D., Tiwari, M. K., Rajwade, Y. A., & Subeesh, A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07744-x · Cited by 73
Audio–Visual Fusion for Emotion Recognition in the Valence–Arousal Space Using Joint Cross-Attention
Praveen, R. G., Cardinal, P., & Granger, É. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2022.3233083 · Cited by 83
MMGRec: Multimodal Generative Recommendation with Transformer Model
Liu, H., Wei, Y., Song, X., Guan, W., Li, Y.-F., & Nie, L. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3838594 · Cited by 72
Shao, J., Ma, J., Yu, Y., Zhang, S., Wang, W., Li, W., & Wang, C. (2024) · The Innovation · doi:10.1016/j.xinn.2024.100648 · Cited by 78
Polarity and Intensity: the Two Aspects of Sentiment Analysis
Tian, L., Lai, C., & Moore, J. D. (2018) · doi:10.18653/v1/w18-3306 · Cited by 77
Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension
Mai, S., Hu, H., Xu, J., & Xing, S. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3000510 · Cited by 68
Lessons from infant learning for unsupervised machine learning
Zaadnoordijk, L., Besold, T. R., & Cusack, R. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00488-2 · Cited by 65
Holste, G., Partridge, S. C., Rahbar, H., Biswas, D., Lee, C. I., & Alessio, A. M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00368 · Cited by 58
Gong, X., Chen, C. L. P., Hu, B., & Zhang, T. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3392791 · Cited by 73
Big data and deep learning for RNA biology
Hwang, H., Jeon, H., Yeo, N., & Baek, D. (2024) · Experimental & Molecular Medicine · doi:10.1038/s12276-024-01243-w · Cited by 73
A Multimodal Fusion Network for Student Emotion Recognition Based on Transformer and Tensor Product
Ao, X., Qi, Z., Wang, H., Yang, Q., & Ma, D. (2024) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece61636.2024.10729485 · Cited by 82
HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data
Hemker, K., Simidjievski, N., & Jamnik, M. (2024) · neural information processing systems · doi:10.52202/079017-2057 · Cited by 104
A Unified Framework for Multimodal Domain Adaptation
Qi, F., Yang, X., & Xu, C. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240633 · Cited by 64
RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey
Gao, M., Jiang, J., Zou, G., John, V., & Liu, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2907071 · Cited by 60
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64
Xu, M., Ouyang, L., Han, L., Sun, K., Yu, T., Li, Q., Tian, H., Safarnejad, L., Zhang, H., Gao, Y., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/25535 · Cited by 65
A survey of fish behaviour quantification indexes and methods in aquaculture
An, D., Huang, J., & Wei, Y. (2021) · Reviews in Aquaculture · doi:10.1111/raq.12564 · Cited by 66
Deep Mutual Information Maximin for Cross-Modal Clustering
Mao, Y., Yan, X., Guo, Q., & Ye, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17076 · Cited by 76
Worldwide COVID-19 Vaccines Sentiment Analysis Through Twitter Content
Ansari, M. T. J., & Khan, N. A. (2021) · Electronic Journal of General Medicine · doi:10.29333/ejgm/11316 · Cited by 64
Marcinkevičs, R., Wolfertstetter, P. R., Klimiene, U., Chin-Cheong, K., Paschke, A., Zerres, J., Denzinger, M., Niederberger, D., Wellmann, S. M., Özkan, E., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103042 · Cited by 78
Robust Visual Question Answering: Datasets, Methods, and Future Challenges
Ma, J., Wang, P., Kong, D., Wang, Z., Liu, J., Pei, H., & Zhao, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3366154 · Cited by 87
Lu, X., Xie, L., Xu, L., Mao, R., Xu, X., & Chang, S. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.04.030 · Cited by 75
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, D., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 63
Quantum-inspired Neural Network for Conversational Emotion Recognition
Li, Q., Gkoumas, D., Sordoni, A., Nie, J., & Melucci, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17567 · Cited by 90
Resources for image-based high-throughput phenotyping in crops and data sharing challenges
Danilevicz, M. F., Bayer, P. E., Nestor, B. J., Bennamoun, M., & Edwards, D. (2021) · PLANT PHYSIOLOGY · doi:10.1093/plphys/kiab301 · Cited by 70
Deep multimodal learning for lymph node metastasis prediction of primary thyroid cancer
Wu, X., Li, M., Cui, X., & Xu, G. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac4c47 · Cited by 63
Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3542927 · Cited by 68
TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis
Wang, F., Tian, S., Yu, L., Liu, J., Wang, J., Li, K., & Wang, Y. (2022) · Cognitive Computation · doi:10.1007/s12559-022-10073-9 · Cited by 72
Unlocking the Future of Drug Development: Generative AI, Digital Twins, and Beyond
Mariam, Z., Niazi, S. K., & Magoola, M. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020079 · Cited by 71
Wang, S., He, X., Jian, Z., Li, J., Xu, C., Chen, Y., Liu, Y., Chen, H., Huang, C., Hu, J., et al. (2024) · Eye and Vision · doi:10.1186/s40662-024-00405-1 · Cited by 84
LLM4CAD: Multimodal Large Language Models for Three-Dimensional Computer-Aided Design Generation
Li, X., Sun, Y., & Sha, Z. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4067085 · Cited by 90
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.