Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 701–800 of 1,036
Collective Affinity Learning for Partial Cross-Modal Hashing
Guo, J. F., & Zhu, W. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2941858 · Cited by 48
Aloshban, N., Esposito, A. A., & Vinciarelli, A. (2021) · Cognitive Computation · doi:10.1007/s12559-020-09808-3 · Cited by 53
Preoperative prediction of perineural invasion with multi-modality radiomics in rectal cancer
Guo, Y. L., Wang, Q., Guo, Y., Zhang, Y., Fu, Y., & Zhang, H. (2021) · Scientific Reports · doi:10.1038/s41598-021-88831-2 · Cited by 46
Core Challenges in Embodied Vision-Language Planning
Francis, J., Kitamura, N., Labelle, F., Lu, X., Navarro, I., & Oh, J. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13646 · Cited by 62
A Multimodal Deep Learning-Based Fault Detection Model for a Plastic Injection Molding Process
Kim, G., Choi, J. G., Ku, M., Cho, H., & Lim, S. (2021) · IEEE Access · doi:10.1109/access.2021.3115665 · Cited by 44
Causality-Structured Deep Learning for Soil Moisture Predictions
Li, L., Dai, Y., Shangguan, W., Wei, Z., Wei, N., & Li, Q. (2022) · Journal of Hydrometeorology · doi:10.1175/jhm-d-21-0206.1 · Cited by 50
Xie, X., Li, Z., Tang, Z., Yao, D., & Ma, H. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103154 · Cited by 46
Dual-Key Multimodal Backdoors for Visual Question Answering
Walmer, M., Sikka, K., Sur, I., Shrivastava, A., & Jha, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01494 · Cited by 78
A Theory-Driven Deep Learning Method for Voice Chat–Based Customer Response Prediction
Chen, G., Xiao, S., Zhang, C., & Zhao, H. (2023) · Information Systems Research · doi:10.1287/isre.2022.1196 · Cited by 49
Ali, S., Razi, A., Kim, S., Alsoubai, A., Ling, C., Choudhury, M. D., Wiśniewski, P., & Stringhini, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579608 · Cited by 51
Learning from the global view: Supervised contrastive learning of multimodal representation
Mai, S., Zeng, Y., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101920 · Cited by 57
Incorporating stock prices and text for stock movement prediction based on information fusion
Zhang, Q., Zhang, Y., Bao, F., Liu, Y., Zhang, C., & Лю, П. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107377 · Cited by 50
A novel multimodal depression diagnosis approach utilizing a new hybrid fusion method
Zhang, X., Li, B., & Qi, G. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106552 · Cited by 50
Diagnosing and Re-learning for Balanced Multimodal Learning
Wei, Y., Li, S., Feng, R., & Hu, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73039-9_5 · Cited by 102
Personalized uncertainty quantification in artificial intelligence
Chakraborti, T., Banerji, C. R. S., Marandon, A., Hellon, V., Mitra, R., Lehmann, B., Bräuninger, L., McGough, S. F., Turkay, Ç., Frangi, A. F., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01024-8 · Cited by 62
Artificial intelligence in dermatology: A practical introduction to a paradigm shift
Eapen, B. (2020) · Indian Dermatology Online Journal · doi:10.4103/idoj.idoj_388_20 · Cited by 40
Citrus Huanglongbing Detection Based on Multi-Modal Feature Fusion Learning
Yang, D., Wang, F., Hu, Y., Lan, Y., & Deng, X. (2021) · Frontiers in Plant Science · doi:10.3389/fpls.2021.809506 · Cited by 48
Kuntz, D., & Wilson, A. K. (2022) · Pure and Applied Chemistry · doi:10.1515/pac-2022-0202 · Cited by 52
Automated mental stress recognition through mobile thermal imaging
Cho, Y. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273662 · Cited by 46
Li, B., & Sano, A. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9175463 · Cited by 44
Learning Disentangled Representation for Multimodal Cross-Domain Sentiment Analysis
Zhang, Y., Zhang, Y., Guo, W., Cai, X., & Yuan, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3147546 · Cited by 52
High-Fidelity Generalized Emotional Talking Face Generation with Multi-Modal Emotion Space Learning
Xu, C., Zhu, J., Zhang, J., Han, Y., Chu, W., Tai, Y. Y., Wang, C., Xie, Z., & Liu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00639 · Cited by 65
Cong, P., Xiao, Y., Wan, X., Deng, M., Li, J., & Zhang, X. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3321656 · Cited by 65
VLP2MSA: Expanding vision-language pre-training to multimodal sentiment analysis
Yi, G., Fan, C., Zhu, K., Lv, Z., Liang, S., Wen, Z., Pei, G., Li, T., & Tao, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111136 · Cited by 51
DCCNMF: Deep Complementary and Consensus Non-negative Matrix Factorization for multi-view clustering
Gunawardena, S. D. L., Luong, K., Balasubramaniam, T., & Nayak, R. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111330 · Cited by 43
D’Souza, N. S., Wang, H., Giovannini, A. A., Foncubierta–Rodríguez, A., Beck, K. L., Boyko, O. B., & Syeda-Mahmood, T. F. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103064 · Cited by 47
Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment
Ibrahimi, S., Sun, X., Wang, P., Garg, A., Sanan, A., & Omar, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01107 · Cited by 72
Semantic2Graph: graph-based multi-modal feature fusion for action segmentation in videos
Zhang, J., Tsai, P.-H., & Tsai, M. (2024) · Applied Intelligence · doi:10.1007/s10489-023-05259-z · Cited by 54
Multimodal Colearning Meets Remote Sensing: Taxonomy, State of the Art, and Future Works
Kieu, N., Nguyen, K., Nazib, A., Fernando, T., Fookes, C., & Sridharan, S. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3378348 · Cited by 52
Multimodal missing data in healthcare: A comprehensive review and future directions
Le, L. P., Nguyen, T., Riegler, M., Halvorsen, P., & Nguyen, B. T. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2024.100720 · Cited by 55
Adam, M., Albaseer, A. M., Baroudi, U. A., & Abdallah, M. M. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3554537 · Cited by 63
Automatic multimodal assessment of soft skills in social interactions: a review
Rasipuram, S., & Jayagopi, D. B. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08561-6 · Cited by 52
Flower classification with modified multimodal convolutional neural networks
Bae, K. I., Park, J., Lee, J., Lee, Y.-S., & Lim, C. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113455 · Cited by 49
Attention and Memory-Augmented Networks for Dual-View Sequential Learning
He, Y., Wang, C., Li, N., & Zeng, Z. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403055 · Cited by 50
Shin, H., Kwon, H.-Y., & Ryu, S. (2020) · Electronics · doi:10.3390/electronics9091527 · Cited by 45
On Modality Bias in the TVQA Dataset
Winterbottom, T., Xiao, S., McLean, A., & Moubayed, N. A. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.122 · Cited by 53
Consistent cross-modal identification of cortical neurons with coupled autoencoders
Gala, R., Budzillo, A., Baftizadeh, F., Miller, J., Gouwens, N., Arkhipov, A., Murphy, G., Tasic, B., Zeng, H., Hawrylycz, M., et al. (2021) · Nature Computational Science · doi:10.1038/s43588-021-00030-1 · Cited by 39
Integrating Multimodal Information in Large Pretrained TransformersPreprint
Rahman, W., Hasan, M. K., Lee, S., Zadeh, A., Mao, C., Morency, L., & Hoque, M. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05787 · Cited by 56
Self-citationMachine Learning-based Weather Support for the 2022 Winter Olympics
Xia, J., Li, H., Kang, Y., Yu, C., Ji, L., Wu, L., Lou, X., Zhu, G., Wang, Z., Yan, Z., et al. (2020) · Advances in Atmospheric Sciences · doi:10.1007/s00376-020-0043-5 · Cited by 43
Fine-Grained Visual Categorization: A Spatial–Frequency Feature Fusion Perspective
Wang, M., Zhao, P., Lu, X., Min, F., & Wang, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3227737 · Cited by 42
Clinical applications of graph neural networks in computational histopathology: A review
Meng, X., & Zou, T. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107201 · Cited by 47
Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction
Hu, X., Chen, J., Liu, A., Meng, S., Wen, L., & Yu, P. S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611899 · Cited by 60
Peng, Y., Lock, I., & Salah, A. A. (2023) · Communication Methods and Measures · doi:10.1080/19312458.2023.2277956 · Cited by 50
Cross-Domain Transfer Hashing for Efficient Cross-Modal Retrieval
Li, F., Wang, B., Zhu, L., Li, J., Zhang, Z., & Chang, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3374791 · Cited by 50
Integrated Heterogeneous Graph Attention Network for Incomplete Multi-modal Clustering
Wang, Y., Yao, X., Zhu, P., Li, W., Cao, M., & Hu, Q. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02066-y · Cited by 46
A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social Environments
Duncan, J. A., Alambeigi, F., & Pryor, M. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3657030 · Cited by 54
Moorthy, S., & Moon, Y.-K. (2025) · Mathematics · doi:10.3390/math13071100 · Cited by 45
Song, Q., Hu, J., Xiao, L., Sun, B., Gao, X., & Li, S. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3583509 · Cited by 50
Utilizing multimodal artificial intelligence to advance cardiovascular diseases
Yang, X.-Y., Li, Y., Wang, J., Jia, Y., Zhang, Y., & Chen, M. (2025) · Precision Clinical Medicine · doi:10.1093/pcmedi/pbaf016 · Cited by 51
Hybrid deep neural networks for reservoir production prediction
Yuan, Z., Huang, H., Jiang, Y., & Li, J. (2020) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2020.108111 · Cited by 42
Breaking Isolation: Multimodal Graph Fusion for Multimedia Recommendation by Edge-wise Modulation
Chen, F., Wang, J., Wei, Y., Zheng, H.-T., & Shao, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548399 · Cited by 60
Multimodal deep learning methods enhance genomic prediction of wheat breeding
Montesinos-López, A., Rivera, C., Pinto, F., Piñera, F., Gonzalez, D., Reynolds, M., Pérez-Rodríguez, P., Li, H., Montesinos-López, O. A., & Crossa, J. (2023) · G3: Genes, Genomes, Genetics · doi:10.1093/g3journal/jkad045 · Cited by 41
Sutter, T. M., Daunhawer, I., & Vogt, J. E. (2021) · arXiv · doi:10.48550/arxiv.2105.02470 · Cited by 41
Wu, C., Zhang, R., Ramamohanarao, K., & Bouvry, P. (2023) · ACM Computing Surveys · doi:10.1145/3571807 · Cited by 51
100 years of anthropogenic impact causes changes in freshwater functional biodiversity
Eastwood, N., Zhou, J., Derelle, R., Abdallah, M. A., Stubbings, W. A., Jia, Y., Crawford, S. E., Davidson, T. A., Colbourne, J. K., Creer, S., et al. (2023) · eLife · doi:10.7554/elife.86576 · Cited by 47
Huang, G., Li, R., Bai, Q., & Alty, J. (2023) · Health Information Science and Systems · doi:10.1007/s13755-023-00231-0 · Cited by 46
Zheng, X., Wang, M., Huang, K., & Zhu, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102077 · Cited by 46
Pre-gating and contextual attention gate — A new fusion method for multi-modal data tasks
Zhang, D., Nayak, R., & Bashar, M. A. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106553 · Cited by 48
Wang, K., Shan, S., Dou, W., Wei, H., & Zhang, K. (2024) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2024.3494266 · Cited by 49
A Survey on Knowledge Editing of Neural Networks
Mazzia, V., Pedrani, A., Caciolai, A., Rottmann, K., & Bernardi, D. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3498935 · Cited by 52
Yang, W., Chen, Z., Zhao, H., Chen, S., & Shi, C. (2025) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2024.106351 · Cited by 44
Multimodal subspace support vector data description
Sohrab, F., Raitoharju, J., Iosifidis, A., & Gabbouj, M. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107648 · Cited by 44
Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks
Mitri, D. D., Schneider, J., Specht, M. M., & Drachsler, H. (2019) · Sensors · doi:10.3390/s19143099 · Cited by 47
Huang, J., Habib, A.-R., Mendis, D., Chong, J., Smith, M., Duvnjak, M., Chiu, C., Singh, N. P., & Wong, E. H. (2019) · The Journal of Laryngology & Otology · doi:10.1017/s0022215119002536 · Cited by 41
Document image classification: Progress over two decades
Liu, L., Wang, Z., Qiu, T., Chen, Q., Lu, Y., & Suen, C. Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.114 · Cited by 49
Multi-View Speech Emotion Recognition Via Collective Relation Construction
Hou, M.-X., Zhang, Z., Cao, Q. Z., Zhang, D. D., & Lu, G. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3133196 · Cited by 43
Heterogeneous data integration methods for patient similarity networks
Gliozzo, J., Mesiti, M., Notaro, M., Petrini, A., Patak, A., Gallardo, A. P., Paccanaro, A., Valentini, G., & Casiraghi, E. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac207 · Cited by 47
Multimodal Conversational AI: A Survey of Datasets and Approaches
Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 56
Yang, S., Li, M., & Wang, J. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3221417 · Cited by 41
Foundations of Multimodal Co-learning
Zadeh, A., Liang, P. P., & Morency, L.-P. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.001 · Cited by 40
Self-citationJoint modelling of audio-visual cues using attention mechanisms for emotion recognition
Ghaleb, E., Niehues, J., & Asteriadis, S. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13557-w · Cited by 45
AlignFusionNet: Efficient Cross-Modal Alignment and Fusion for 3D Semantic Occupancy Prediction
Xu, Z., Qi, L., Du, H.-E., Yang, J., & Chen, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3589858 · Cited by 47
Multimodal perception-driven decision-making for human-robot interaction: a survey
Zhao, W., Gangaraju, K., & Yuan, F. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2025.1604472 · Cited by 54
Improving rare disease classification using imperfect knowledge graph
Li, X., Wang, Y., Wang, D., Yuan, W., Peng, D., & Mei, Q. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0938-1 · Cited by 40
Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data
Fayek, H. M., & Kumar, A. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/78 · Cited by 29
Automatic Mobile App Identification From Encrypted Traffic With Hybrid Neural Networks
Wang, X., Chen, S., & Su, J. (2020) · IEEE Access · doi:10.1109/access.2020.3029190 · Cited by 43
Predicting TCR-Epitope Binding Specificity Using Deep Metric Learning and Multimodal Learning
Luu, A., Leistico, J., Miller, T., Kim, S., & Song, J. (2021) · Genes · doi:10.3390/genes12040572 · Cited by 38
Real-Time Human Recognition at Night via Integrated Face and Gait Recognition Technologies
Manssor, S. A. F., Sun, S., & Elhassan, M. A. M. (2021) · Sensors · doi:10.3390/s21134323 · Cited by 44
Employing multimodal co-learning to evaluate the robustness of sensor fusion for industry 5.0 tasks
Rahate, A., Mandaokar, S., Chandel, P., Walambe, R. A., Ramanna, S., & Kotecha, K. V. (2022) · Soft Computing · doi:10.1007/s00500-022-06802-9 · Cited by 40
Imbalanced Malware Family Classification Using Multimodal Fusion and Weight Self-Learning
Li, S., Li, Y., Wu, X., Otaibi, S. A., & Tian, Z. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3208891 · Cited by 40
Zeng, Z., Sun, S., & Li, Q. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103378 · Cited by 41
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Bugliarello, E., Liu, F., Pfeiffer, J., Reddy, S., Elliott, D., Ponti, E. M., & Vulić, I. (2022) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.2201.11732 · Cited by 89
VIUNet: Deep Visual–Inertial–UWB Fusion for Indoor UAV Localization
Kao, P.-Y., Chang, H.-J., Tseng, K.-W., Chen, T., Luo, H.-L., & Hung, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3279292 · Cited by 39
Multi-view cost-sensitive kernel learning for imbalanced classification problem
Tang, J., Hou, Z., Yu, X., Fu, S., & Tian, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126562 · Cited by 42
Yang, S., Zhuang, L., Zhang, J., Lan, J., & Li, B. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3358403 · Cited by 39
MDF-FND: A dynamic fusion model for multimodal fake news detection
Lv, H., Yang, W., Yin, Y., Wei, F., Peng, J., & Geng, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113417 · Cited by 48
Modality-based Factorization for Multimodal Fusion
Barezi, E. J., & Fung, P. (2019) · Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) · doi:10.18653/v1/w19-4331 · Cited by 32
Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning
Dolmans, T. C., Poel, M., Klooster, J. W. J. R. V. '., & Veldkamp, B. P. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.609096 · Cited by 39
Classification of Arabic Tweets: A Review
Alruily, M. (2021) · Electronics · doi:10.3390/electronics10101143 · Cited by 41
Ensemble learning based multi-modal intra-hour irradiance forecasting
Shan, S., Li, C., Ding, Z., Wang, Y., Zhang, K., & Wei, H. (2022) · Energy Conversion and Management · doi:10.1016/j.enconman.2022.116206 · Cited by 39
Universal Multimodal Representation for Language Understanding
Zhang, Z., Chen, K., Wang, R., Utiyama, M., Sumita, E., Li, Z., & Zhao, H. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234170 · Cited by 53
Multimodal Finger Recognition Based on Asymmetric Networks With Fused Similarity
Huang, Y., Ma, H., & Wang, M. (2023) · IEEE Access · doi:10.1109/access.2023.3242984 · Cited by 39
Designing Large Foundation Models for Efficient Training and Inference: A SurveyPreprint
Liu, D., Yu, Y., Wang, Y., Wu, J., Wan, Z., Alinejad, S., Lengerich, B. J., & Wu, Y. N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.01990 · Cited by 43
Hou, J., Zhang, R., Xie, Y., Li, C., & Qin, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02257-y · Cited by 38
Dynamic alignment and fusion of multimodal physiological patterns for stress recognition
Zhang, X., Wei, X., Zhou, Z., Zhao, Q., Zhang, S., Yang, Y., Li, R., & Hu, B. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3290177 · Cited by 38
Efficient Driver Anomaly Detection via Conditional Temporal Proposal and Classification Network
Su, L., Sun, C., Cao, D., & Khajepour, A. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3158480 · Cited by 38
Real-time multimodal interaction in virtual reality - a case study with a large virtual interface
Cao, L., Zhang, H., Peng, C. Y., & Hansberger, J. T. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14381-6 · Cited by 41
Explainable AI in Healthcare Application
Sindiramutty, S. R., Tee, W. J., Balakrishnan, S., Kaur, S., Thangaveloo, R., Jazri, H., Khan, N. A., Gharib, A. H., & Manchuri, A. R. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-6361-1.ch005 · Cited by 45
Forecasting backdraft with multimodal method: Fusion of fire image and sensor data
Zhang, T., Ding, F., Wang, Z., Xiao, F., Lu, C. X., & Huang, X. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.107939 · Cited by 41
Multimodal 1D CNN for delamination prediction in CFRP drilling process with industrial robots
Choi, J. G., Kim, D. C., Chung, M., Lim, S., & Park, H. W. (2024) · Computers & Industrial Engineering · doi:10.1016/j.cie.2024.110074 · Cited by 41
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.