Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 101–200 of 1,036
Multimodal learning analytics for game‐based learning
Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.12992 · Cited by 148
Exploiting BERT for Multimodal Target Sentiment Classification through Input Space Translation
Khan, Z., & Fu, Y. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475692 · Cited by 193
Kurunathan, H., Huang, H., Li, K., Ni, W., & Hossain, E. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3312221 · Cited by 192
Deep Learning Techniques for Future Intelligent Cross-Media Retrieval
Rehman, S. U., Waqas, M. M., Tu, S., Koubâa, A., Rehman, O. U., Ahmad, J. E., Hanif, M. I., & Han, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01191 · Cited by 190
Shaban, A., & Yousefi, S. (2024) · Springer Optimization and Its Applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 189
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 249
Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification
Han, Z., Yang, F., Huang, J., Zhang, C., & Yao, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02005 · Cited by 188
Gao, Y., Li, W., Zhang, M., Wang, J., Sun, W., Tao, R., & Du, Q. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3097093 · Cited by 188
Data‐Driven Materials Innovation and Applications
Wang, Z., Sun, Z., Yin, H., Liu, X., Wang, J., Zhao, H., Pang, C. H., Wu, T., Li, S., Yin, Z., et al. (2022) · Advanced Materials · doi:10.1002/adma.202104113 · Cited by 187
Disturbance observers and extended state observers for marine vehicles: A survey
Gu, N., Wang, D., Peng, Z., Wang, J., & Han, Q.-L. (2022) · Control Engineering Practice · doi:10.1016/j.conengprac.2022.105158 · Cited by 187
Collaboration between clinicians and vision–language models in radiology report generation
Tanno, R., Barrett, D. G. T., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Lau, C., Tu, T., Azizi, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03302-1 · Cited by 186
A Survey of Textual Emotion Recognition and Its Challenges
Deng, J., & Ren, F. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3053275 · Cited by 186
Reasoning with Heterogeneous Graph Alignment for Video Question Answering
Jiang, P., & Han, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6767 · Cited by 186
Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review
Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2939399 · Cited by 184
Rahman, M. H., Sejan, M. A. S., Aziz, M. A., Tabassum, R., Baik, J.-I., & Song, H. (2024) · Remote Sensing · doi:10.3390/rs16050879 · Cited by 205
Multi-View Graph Convolutional Network for Multimedia Recommendation
Yu, P., Tan, Z., Lu, G., & Bao, B.-K. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3613915 · Cited by 183
Mai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182
Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching
Liu, C., Mao, Z., Liu, A.-A., Zhang, T., Wang, B., & Zhang, Y. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350869 · Cited by 182
From Robotic Process Automation to Intelligent Process Automation: Emerging Trends
Chakraborti, T., Isahagian, V., Khalaf, R. Y., Khazaeni, Y., Muthusamy, V., Rizk, Y., & Unuvar, M. (2020) · Lecture notes in business information processing · doi:10.1007/978-3-030-58779-6_15 · Cited by 181
Effective Techniques for Multimodal Data Fusion: A Comparative Analysis
Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S. (2023) · Sensors · doi:10.3390/s23052381 · Cited by 180
A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data
Ma, X., Zhang, X., & Pun, M.-O. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3165005 · Cited by 179
Current and Emerging Trends in Medical Image Segmentation With Deep Learning
Conze, P.-H., Andrade-Miranda, G. X., Singh, V. K., Jaouen, V., & Visvikis, D. (2023) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2023.3265863 · Cited by 194
Multimodal fusion on low-quality data: A comprehensive survey
Zhang, Q., Wei, Y., Han, Z., Fu, H., Peng, X., Hu, Q., Deng, C., Xu, C., Wen, J., Hu, D., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104437 · Cited by 222
A systematic survey on multimodal emotion recognition using learning algorithms
Ahmed, N., Aghbari, Z. A., & Girija, S. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200171 · Cited by 178
Lin, M., Guo, J., Gu, Z., Tang, W., Tao, H., You, S., Jia, D., Sun, Y., & Jia, P. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-06425-2 · Cited by 191
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., & Hoque, M. (. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1211 · Cited by 174
Self-citationTowards Robust Pattern Recognition: A Review
Zhang, X.-Y., Liu, C., & Suen, C. Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2989782 · Cited by 202
Cukurova, M., Kent, C., & Luckin, R. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12829 · Cited by 173
ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis
Yang, J., Yu, Y., Niu, D., Guo, W., & Xu, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.421 · Cited by 173
Semi-Supervised Multi-View Deep Discriminant Representation Learning
Jia, X., Jing, X.-Y., Zhu, X., Chen, S., Du, B., Cai, Z., He, Z., & Yue, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2973634 · Cited by 173
Multimodal Fake News Detection via CLIP-Guided Learning
Zhou, Y., Yang, Y., Ying, Q., Qian, Z., & Zhang, X. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00480 · Cited by 229
Deep Collaborative Multi-View Hashing for Large-Scale Image Search
Zhu, L., Lu, X., Cheng, Z., Li, J., & Zhang, H. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2974065 · Cited by 170
The promise and challenges of multimodal learning analytics
Cukurova, M., Giannakos, M., & Martinez‐Maldonado, R. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.13015 · Cited by 169
Synthesis of Compositional Animations from Textual Descriptions
Ghosh, A., Cheema, N., Oguz, C., Theobalt, C., & Slusallek, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00143 · Cited by 169
MAMO: Memory-Augmented Meta-Optimization for Cold-start Recommendation
Dong, M., Yuan, F., Yao, L., Xu, X., & Zhu, L. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403113 · Cited by 167
Wu, Z., Zheng, C., Cai, Y., Chen, J., Leung, H.-F., & Li, Q. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413650 · Cited by 211
Zhou, H., Zhou, X., Zeng, Z., Zhang, L., & Miao, C. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3710585 · Cited by 193
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 229
Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
Wang, T., Li, F., Zhu, L., Li, J., Zhang, Z., & Shen, H. T. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3525147 · Cited by 247
CentralNet: A Multilayer Approach for Multimodal Fusion
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11024-6_44 · Cited by 166
Multimodal learning with graphs
Ektefaie, Y., Dasoulas, G., Noori, A., Farhat, M., & Zitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00624-6 · Cited by 166
End-to-End Adversarial-Attention Network for Multi-Modal Clustering
Zhou, R., & Shen, Y.-D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01463 · Cited by 166
Fang, S., Li, K., & Li, Z. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3121028 · Cited by 166
Xue, Z., & Marculescu, R. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00256 · Cited by 258
Multimodal data integration for oncology in the era of deep neural networks: a review
Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1408843 · Cited by 165
Latent Structure Mining With Contrastive Modality Fusion for Multimedia Recommendation
Zhang, J., Zhu, Y., Liu, Q., Zhang, M., Wu, S., & Wang, L. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3221949 · Cited by 165
Gehrig, D., Ruegg, M., Gehrig, M., Hidalgo-Carrio, J., & Scaramuzza, D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3060707 · Cited by 165
Emotion Recognition From Multiple Modalities: Fundamentals and methodologies
Zhao, S., Jia, G., Yang, J., Ding, G., & Keutzer, K. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3106895 · Cited by 165
Ahmed, A., Xi, R., Hou, M., Shah, S. A., & Hameed, S. (2023) · IEEE Access · doi:10.1109/access.2023.3323574 · Cited by 199
Towards Open Vocabulary Learning: A Survey
Wu, J., Li, X., Xu, S., Yuan, H., Ding, H., Yang, Y., Li, X., Zhang, J., Tong, Y., Jiang, X., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3361862 · Cited by 164
Next-generation deep learning based on simulators and synthetic data
de Melo, C. M., Torralba, A., Guibas, L., DiCarlo, J., Chellappa, R., & Hodgins, J. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.11.008 · Cited by 164
Li, J., Wang, X., Lv, G., & Zeng, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3260635 · Cited by 184
MMRec: Simplifying Multimodal Recommendation
Zhou, X. (2023) · doi:10.1145/3611380.3628561 · Cited by 163
Li, Y., Zhou, Y., Zhang, Y., Zhong, L., Wang, J., & Chen, J. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.02.013 · Cited by 162
Cheng, J., Deng, C., Su, Y., An, Z., & Wang, Q. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.03.012 · Cited by 177
Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning
Luna-Jiménez, C., Griol, D., Callejas, Z., Kleinlein, R., Montero, J. M., & Fernández-Martínez, F. (2021) · Sensors · doi:10.3390/s21227665 · Cited by 161
Multi-modal Attention Network Learning for Semantic Source Code Retrieval
Wan, Y., Shu, J., Sui, Y., Xu, G., Zhao, Z., Wu, J., & Yu, P. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00012 · Cited by 161
Goshisht, M. K. (2024) · ACS Omega · doi:10.1021/acsomega.3c05913 · Cited by 161
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01589 · Cited by 160
Pan, X., Lu, Y., Zhao, K., Huang, H., Wang, M., & Chen, H. (2021) · Geophysical Research Letters · doi:10.1029/2021gl095302 · Cited by 161
A Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 159
Supervised Multimodal Bitransformers for Classifying Images and Text
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., & Testuggine, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02950 · Cited by 159
Bayoudh, K. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102217 · Cited by 159
Actor and Observer: Joint Modeling of First and Third-Person Videos
Sigurdsson, G. A., Gupta, A., Schmid, C., Farhadi, A., & Alahari, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00772 · Cited by 139
Multimodal Relation Extraction with Efficient Graph Alignment
Zheng, C., Feng, J., Fu, Z., Cai, Y., Li, Q., & Wang, T. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3476968 · Cited by 209
A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications
Jiao, T., Guo, C., Feng, X., Chen, Y., & Song, J. (2024) · Computers, Materials & Continua · doi:10.32604/cmc.2024.053204 · Cited by 158
Artificial Intelligence and Fraud Detection
Bao, Y., Hilary, G., & Ke, B. (2022) · Springer Series in Supply Chain Management · doi:10.1007/978-3-030-75729-8_8 · Cited by 158
AVEC 2018 Workshop and Challenge: Bipolar Disorder and Cross-Cultural Affect Recognition
Ringeval, F., Schuller, B., Valstar, M., Cowie, R., Kaya, H., Schmitt, M., Amiriparian, S., Cummins, N., Lalanne, D., Michaud, A., et al. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266316 · Cited by 148
MMNet: A Model-Based Multimodal Network for Human Action Recognition in RGB-D Videos
Yu, B. X. B., Liu, Y., Zhang, X., Zhong, S.-H., & Chan, K. C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3177813 · Cited by 206
Image-text Retrieval: A Survey on Recent Research and Development
Cao, M., Li, S., Li, J., Nie, L., & Zhang, M. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/759 · Cited by 189
Adversarial Sticker: A Stealthy Attack Method in the Physical World
Wei, X., Guo, Y., & Yu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3176760 · Cited by 216
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
Vassilev, A. (2025) · doi:10.6028/nist.ai.100-2e2025 · Cited by 194
Multimodal Video Sentiment Analysis Using Deep Learning Approaches, a Survey
Abdu, S. A., Yousef, A. H., & Salem, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.003 · Cited by 165
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153
Nakach, F.-Z., Idri, A., & Göçeri, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10984-z · Cited by 171
Multimodal Self-supervised Learning for Medical Image Analysis
Taleb, A., Lippert, C., Klein, T., & Nabi, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78191-0_51 · Cited by 199
A Review of Key Technologies for Emotion Analysis Using Multimodal Information
Zhu, X., Guo, C., Feng, H., Huang, Y., Feng, Y., Wang, X., & Wang, R. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10287-z · Cited by 197
Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective
Wang, M., Wang, S., Yang, H., Zhang, Z., Chen, X., & Qi, G. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475470 · Cited by 197
A Comprehensive Survey of Artificial Intelligence Techniques for Talent Analytics
Qin, C., Zhang, L., Cheng, Y., Zha, R., Shen, D., Zhang, Q., Chen, X., Sun, Y., Zhu, C., Zhu, H., et al. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3572744 · Cited by 174
Novel research and future prospects of artificial intelligence in cancer diagnosis and treatment
Zhang, C., Xu, J., Tang, R., Yang, J., Wang, W., Yu, X., & Shi, S. (2023) · Journal of Hematology & Oncology · doi:10.1186/s13045-023-01514-5 · Cited by 148
MetaLoc: Learning to Learn Wireless Localization
Gao, J., Wu, D., Yin, F., Kong, Q., Xu, L., & Cui, S. R. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3322766 · Cited by 165
Multi-modal learning from unpaired images: application to multi-organ segmentation in CT and MRI
Valindria, V. V., Pawlowski, N., Rajchl, M., Lavdas, I., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00066 · Cited by 146
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 47
Multimodal Federated Learning: A Survey
Che, L., Wang, J., Zhou, Y., & Ma, F. (2023) · Sensors · doi:10.3390/s23156986 · Cited by 185
Blockchain Systems, Technologies, and Applications: A Methodology Perspective
Cao, B., Wang, Z., Zhang, L., Feng, D., Peng, M., Zhang, L., & Han, Z. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3204702 · Cited by 159
Interpretability-Based Multimodal Convolutional Neural Networks for Skin Lesion Diagnosis
Wang, S., Yin, Y., Wang, D., Wang, Y., & Jin, Y. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3069920 · Cited by 135
Deepfakes Detection Techniques Using Deep Learning: A Survey
Almars, A. M. (2021) · Journal of Computer and Communications · doi:10.4236/jcc.2021.95003 · Cited by 143
Quantum-inspired multimodal fusion for video sentiment analysis
Li, Q., Gkoumas, D., Lioma, C., & Melucci, M. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.08.006 · Cited by 143
Multimodal Classification: Current Landscape, Taxonomy and Future Directions
Sleeman, W. C., Kapoor, R., & Ghosh, P. (2023) · ACM Computing Surveys · doi:10.1145/3543848 · Cited by 124
Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
Ao, T., Gao, Q., Lou, Y., Chen, B., & Liu, L. (2022) · ACM Transactions on Graphics · doi:10.1145/3550454.3555435 · Cited by 190
CM-BERT: Cross-Modal BERT for Text-Audio Sentiment Analysis
Yang, K., Xu, H., & Gao, K. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413690 · Cited by 129
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 169
Artificial intelligence and blockchain: A review
Hussain, A. A., & Al-Turjman, F. M. (2021) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.4268 · Cited by 164
D-vlog: Multimodal Vlog Dataset for Depression Detection
Yoon, J., Kang, C., Kim, S., & Han, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21483 · Cited by 116
Work Together: Correlation-Identity Reconstruction Hashing for Unsupervised Cross-Modal Retrieval
Zhu, L., Wu, X., Li, J., Zhang, Z., Guan, W., & Shen, H. T. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3218656 · Cited by 157
Threat of Adversarial Attacks on DL-Based IoT Device Identification
Bao, Z., Lin, Y., Zhang, S., Li, Z., & Mao, S. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3120197 · Cited by 148
A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445472 · Cited by 137
Lifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 173
Wang, M., Tan, K., Jia, X., Wang, X., & Chen, Y. (2020) · Remote Sensing · doi:10.3390/rs12020205 · Cited by 141
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.