Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 601–700 of 1,036
MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · arXiv · doi:10.48550/arxiv.2102.02340 · Cited by 52
CAMM: Cross-Attention Multimodal Classification of Disaster-Related Tweets
Khattar, A., & Quadri, S. M. K. (2022) · IEEE Access · doi:10.1109/access.2022.3202976 · Cited by 52
Donovan, S. M., Aghaeepour, N., Andres, A., Azad, M. B., Becker, M., Carlson, S. E., Järvinen, K. M., Lin, W., Lönnerdal, B. L., Slupsky, C. M., et al. (2023) · American Journal of Clinical Nutrition · doi:10.1016/j.ajcnut.2022.12.021 · Cited by 56
Li, Z., Guo, Q., Pan, Y., Ding, W., Yu, J., Zhang, Y., Liu, W., Chen, H., Wang, H., & Xie, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101891 · Cited by 58
Deep learning methods improve genomic prediction of wheat breeding
Montesinos‐López, A., Crespo‐Herrera, L., Dreisigacker, S., Gerard, G., Vitale, P., Pierre, C. S., Govindan, V., Tadesse, Z., Flores, M. C., Pérez‐Rodríguez, P., et al. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1324090 · Cited by 52
GCCNet: A Novel Network Leveraging Gated Cross-Correlation for Multi-View Classification
Zeng, Y., Zhang, R., Zhang, H., Qiao, S., Huang, F., Tian, Q., & Peng, Y. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521733 · Cited by 57
Multi-modal Active Learning From Human Data: A Deep Reinforcement Learning Approach
Rudovic, O. O., Zhang, M., Schuller, B. W., & Picard, R. W. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353742 · Cited by 62
Self-Learning Optimal Cruise Control Based on Individual Car-Following Style
Chu, H., Guo, L., Yan, Y., Gao, B., & Chen, H. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2981493 · Cited by 52
Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition
Wu, J., Harrison, C., Bigham, J. P., & Laput, G. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376875 · Cited by 57
mEBAL: A Multimodal Database for Eye Blink Detection and Attention Level Estimation
Daza, R., Morales, A., Fiérrez, J., & Tolosana, R. (2020) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3395035.3425257 · Cited by 52
Multimodal Deep Learning for Social Media Popularity Prediction With Attention Mechanism
Xu, K., Lin, Z., Zhao, J., Shi, P., Deng, W. G., & Wang, H. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3416274 · Cited by 52
Aymerich, E., Sias, G., Pisano, F., Cannas, B., Carcangiu, S., Sozzi, C., Stuart, C., Carvalho, P. J., Fanni, A., & Contributors, J. (2022) · Nuclear Fusion · doi:10.1088/1741-4326/ac525e · Cited by 52
Two-Stage Fuzzy Fusion Based-Convolution Neural Network for Dynamic Emotion Recognition
Wu, M., Su, W., Chen, L., Pedrycz, W., & Hirota, K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2966440 · Cited by 55
Haptic Signal Reconstruction for Cross-Modal Communications
Wei, X., Shi, Y., & Zhou, L. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3119860 · Cited by 56
Breast Tumor Classification Based on MRI-US Images by Disentangling Modality Features
Qiao, M., Liu, C., Li, Z., Zhou, J., Xiao, Q., Zhou, S., Chang, C., Gu, Y., Guo, Y., & Wang, Y. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3140236 · Cited by 51
Xiao, L., Wu, X., Wu, W., Yang, J., & He, L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747542 · Cited by 63
Vortmann, L.-M., Ceh, S. M., & Putze, F. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.780580 · Cited by 57
Liu, L., Wan, X., Li, J., Wang, W., & Gao, Z. (2022) · Sensors · doi:10.3390/s22176391 · Cited by 53
CAMR: cross-aligned multimodal representation learning for cancer survival prediction
Wu, X., Shi, Y., Wang, M., & Li, A. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad025 · Cited by 49
Nam, W. K., & Jang, B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121168 · Cited by 63
Instruction-ViT: Multi-modal prompts for instruction learning in vision transformer
Xiao, Z., Chen, Y., Yao, J., Zhang, L., Liu, Z., Wu, Z., Yu, X., Pan, Y., Zhao, L., Ma, C., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102204 · Cited by 58
A survey on membership inference attacks and defenses in machine learning
Niu, J., Liu, P., Zhu, X., Shen, K., Wang, Y., Chi, H., Shen, Y., Jiang, X., Ma, J., & Zhang, Y. (2024) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2024.02.001 · Cited by 81
Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer
Wang, Y., Liu, W., Li, G., Ding, J., Hu, D., & Li, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28378 · Cited by 79
Versatile Audio-Visual Learning for Emotion Recognition
Goncalves, L., Leem, S.-G., Lin, W.-C., Şişman, B., & Busso, C. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3433386 · Cited by 58
Antognoli, V., Presutti, L., Bovo, M., Torreggiani, D., & Tassinari, P. (2025) · Animals · doi:10.3390/ani15172508 · Cited by 57
Xiao, Y., Li, Y., Cui, G., Zhang, H., & Zhang, W. (2025) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2025.105277 · Cited by 52
Understanding Multimodal Popularity Prediction of Social Media Videos With Self-Attention
Bielski, A., & Trzcinski, T. P. (2018) · IEEE Access · doi:10.1109/access.2018.2884831 · Cited by 51
Semi-supervised cross-modal image generation with generative adversarial networks
Li, D., Du, C., & He, H. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.107085 · Cited by 54
Multimodal Document Image Classification
Jain, R., & Wigington, C. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00021 · Cited by 58
Learning to Translate for Cross-Source Remote Sensing Image Retrieval
Xiong, W., Lv, Y., Zhang, X., & Cui, Y. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2968096 · Cited by 51
Meningiomas: Preoperative predictive histopathological grading based on radiomics of MRI
Han, Y., Wang, T., Wu, P. S., Zhang, H., Chen, H., & Yang, C. (2020) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2020.11.009 · Cited by 53
Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrieval
Hu, F., Chen, A., Wang, Z., Zhou, F., Dong, J., & Li, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19781-9_26 · Cited by 69
AI-based diabetes care: risk prediction models and implementation concerns
Wang, S. C. Y., Nickel, G., Venkatesh, K. P., Raza, M. M., & Kvedar, J. C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01034-7 · Cited by 64
All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment
Zhang, C., Xin, S., Yang, Y., Liu, L., Liu, Q., Zhou, X., & Wang, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611803 · Cited by 78
Merkt, O. (2019) · Annals of Computer Science and Information Systems · doi:10.15439/2019f101 · Cited by 50
On the Effectiveness of Transfer Learning for Code Search
Salza, P., Schwizer, C., Gu, J., & Gall, H. C. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3192755 · Cited by 43
RadioPathomics: Multimodal learning in non-small cell lung cancer for adaptive radiotherapy
Tortora, M., Cordelli, E., Sicilia, R., Nibid, L., Ippolito, E., Perrone, G., Ramella, S., & Soda, P. (2023) · IEEE Access · doi:10.1109/access.2023.3275126 · Cited by 49
Parameter Efficient Multimodal Transformers for Video Representation Learning
Lee, S., Yu, Y., Kim, G., Breuel, T., Kautz, J., & Song, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.04124 · Cited by 49
Chen, C., Song, M., Song, W., Guo, L., & Jian, M. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3203421 · Cited by 56
Hand-based multimodal biometric fusion: A review
Li, S., Fei, L., Zhang, B., Ning, X., & Wu, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102418 · Cited by 49
Automated Ensemble Multimodal Machine Learning for Healthcare
Imrie, F., Denner, S., Brunschwig, L. S., Maier-Hein, K. H., & van der Schaar, M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3530156 · Cited by 57
Cuoco, E., Cavaglià, M., Heng, I. S., Keitel, D., & Messenger, C. (2025) · Living Reviews in Relativity · doi:10.1007/s41114-024-00055-8 · Cited by 53
Multimodal fusion techniques: Review, data representation, information fusion, and application areas
Hangloo, S., & Arora, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130827 · Cited by 60
Multimodal Integration in Health Care: Development With Applications in Disease Management
Hao, Y., Cheng, C., Li, J., Li, H., Di, X., Zeng, X., Jin, S., Han, X., Liu, C., Wang, Q., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/76557 · Cited by 64
Audio-Visual Speaker Recognition with a Cross-Modal Discriminative Network
Tao, R., Das, R. K., & Li, H. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1814 · Cited by 61
Modality Matches Modality: Pretraining Modality-Disentangled Item Representations for Recommendation
Han, T., Wang, P., Niu, S., & Li, C. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512079 · Cited by 49
Zhou, X., Lin, Q., Gui, Y., Wang, Z., Liu, M., & Lu, H. J. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.710133 · Cited by 55
Detection of Sleep Apnea from Electrocardiogram and Pulse Oximetry Signals Using Random Forest
Zhu, J., Zhou, A., Gong, Q., Zhou, Y., Huang, J., & Chen, Z. (2022) · Applied Sciences · doi:10.3390/app12094218 · Cited by 53
Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 56
Robust multi-view learning via adaptive regression
Jiang, B., Xiang, J., Wu, X., Wang, Y., Chen, H., Cao, W., & Sheng, W. (2022) · Information Sciences · doi:10.1016/j.ins.2022.08.017 · Cited by 53
MIA-Net: Multi-Modal Interactive Attention Network for Multi-Modal Affective Analysis
Li, S., Zhang, T., Chen, B., & Chen, C. L. P. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3259010 · Cited by 54
Rate-Adaptive Coding Mechanism for Semantic Communications With Multi-Modal Data
He, Y., Yu, G., & Cai, Y. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3335977 · Cited by 63
Grounding 3D Object Affordance from 2D Interactions in Images
Yang, Y., Zhai, W., Luo, H., Cao, Y. H., Luo, J., & Zha, Z.-J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01001 · Cited by 99
Slaughter, I., Charla, J. L., Siderius, M., & Lipor, J. J. (2024) · Journal of Ocean Engineering and Science · doi:10.1016/j.joes.2024.01.002 · Cited by 57
Continual Learning for Smart City: A Survey
Yang, L., Luo, Z., Zhang, S., Teng, F., & Li, T. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3447123 · Cited by 57
Federated Edge Learning for 6G: Foundations, Methodologies, and Applications
Tao, M., Zhou, Y., Shi, Y., Lu, J., Cui, S. R., Lü, J., & Letaief, K. B. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3509739 · Cited by 59
ECG-LM: Understanding Electrocardiogram with a Large Language Model
Yang, K., Hong, M., Zhang, J., Luo, Y., Zhao, S., Zhang, O., Yu, X., Zhou, J., Yang, L., Zhang, P., et al. (2025) · Health Data Science · doi:10.34133/hds.0221 · Cited by 67
Research on WebAssembly Runtimes: A Survey
Zhang, Y., Liu, M., Wang, H., Ma, Y., Huang, G., & Liu, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714465 · Cited by 61
Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications
Cajas, S. A., Samanta, J., Suárez‐Cetrulo, A. L., & Carbajo, R. S. (2025) · Future Internet · doi:10.3390/fi17090417 · Cited by 57
A Linguistically-Informed Fusion Approach for Multimodal Depression Detection
Morales, M. R., Scherer, S. J., & Levitan, R. (2018) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/w18-0602 · Cited by 55
Conditioned Source Separation for Musical Instrument Performances
Slizovskaia, O., Haro, G., & Gomez, E. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3082331 · Cited by 48
Drug-Drug interaction extraction using a position and similarity fusion-based attention mechanism
Fatehifar, M., & Karshenas, H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103707 · Cited by 51
Evolutionary Deep Fusion Method and its Application in Chemical Structure Recognition
Liang, X., Guo, Q. J., Qian, Y., Ding, W., & Zhang, Q. (2021) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2021.3064943 · Cited by 58
Seo, S., Kim, Y., Han, H., Son, W., Hong, Z.-Y., Sohn, I., Shim, J., & Hwang, C. (2021) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.670670 · Cited by 52
Multi-Level Confidence Learning for Trustworthy Multimodal Classification
Zheng, X., Tang, C., Wan, Z., Hu, C., & Zhang, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26346 · Cited by 70
Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models
Zhang, W., Han, J., Xu, Z., Ni, H., Lyu, T., Liu, H., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3821528 · Cited by 49
Towards verifiable cancer digital twins: tissue level modeling protocol for precision medicine
Kemkar, S., Tao, M., Ghosh, A., STAMATAKOS, G. S., Graf, N. M., Poorey, K., Balakrishnan, U., Trask, N., & Radhakrishnan, R. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1473125 · Cited by 50
Generative technology for human emotion recognition: A scoping review
Ma, F., Yuan, Y., Xie, Y., Ren, H., Liu, I., He, Y., Ren, F., Yu, F. R., & Ni, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102753 · Cited by 61
Review of deep learning models for crypto price prediction: Implementation and evaluation
Wu, J., Zhang, X., Huang, F., Zhou, H., & Chandra, R. (2026) · Next research. · doi:10.1016/j.nexres.2026.101513 · Cited by 52
Dense Multimodal Fusion for Hierarchically Joint Representation
Hu, D., Wang, C., Nie, F., & Li, X. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683898 · Cited by 57
Escaping Oz: Autonomy in Socially Assistive Robotics
Clabaugh, C., & Matarić, M. J. (2019) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-060117-104911 · Cited by 55
Zhang, L., Xu, G., Qin, Z. T., Wang, H., Wang, X., Zhang, Y., Liang, J., Wu, G., Song, X., & Wang, Y. (2022) · Sustainability · doi:10.3390/su141912397 · Cited by 53
A Multimodal Memes Classification: A Survey and Open Research Issues
Afridi, T. H., Alam, A., Khan, M. N., Khan, J., & Lee, Y.-K. (2020) · Innovations in Smart Cities Applications Volume 4 · doi:10.1007/978-3-030-66840-2_109 · Cited by 45
Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition
Hu, D., Li, X., Mou, L., Jin, P., Chen, D., Jing, L., Zhu, X. X., & Dou, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_5 · Cited by 54
Mix-VAEs: A novel multisensor information fusion model for intelligent fault diagnosis
Wang, C., Xin, C., Xu, Z., Qin, M., & He, M. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.044 · Cited by 47
AIA-Net: Adaptive Interactive Attention Network for Text–Audio Emotion Recognition
Zhang, T., Li, S., Chen, B., Yuan, H., & Chen, C. L. P. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3195739 · Cited by 52
The Latent Space: Foundation, Evolution, Mechanism, Ability, and OutlookPreprint
Yu, X., Chen, Z., He, Y., Fu, T., Dong, G., Yang, C., Xu, C., Ma, Y., Hu, X., Cao, Z., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.02029 · Cited by 52
Multimodal Sensor Fusion with Differentiable Filters
Lee, M. A., Yi, B., Martin-Martin, R., Savarese, S., & Bohg, J. (2020) · 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341579 · Cited by 46
“and multimodal state estimation for decades [2, 7, 21].”
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Abootorabi, M. M., Zobeiri, A., Dehghani, M., Mohammadkhani, M., Mohammadi, B., Ghahroodi, O., Baghshah, M. S., & Asgari, E. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.861 · Cited by 76
YoTube: Searching Action Proposal Via Recurrent and Static Regression Networks
Zhu, H., Vial, R., Lu, S., Peng, X., Fu, H., Tian, Y., & Cao, X. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2806279 · Cited by 47
New Ideas and Trends in Deep Multimodal Content Understanding: A Review
Chen, W., Wang, W., Liu, L., & Lew, M. S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.042 · Cited by 50
Unsupervised Multimodal Anomaly Detection With Missing Sources for Liquid Rocket Engine
Feng, Y., Liu, Z., Chen, J., Lv, H., Wang, J., & Zhang, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3162949 · Cited by 53
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
Lyu, Y., Liang, P. P., Deng, Z., Salakhutdinov, R., & Morency, L. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534148 · Cited by 68
Self-citationHuang, Y., Huang, S., Wang, Y., Li, Y., Gui, Y., & Huang, C. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.937546 · Cited by 54
Giering, S. L. C., Culverhouse, P. F., Johns, D. G., McQuatters‐Gollop, A., & Pitois, S. G. (2022) · Frontiers in Marine Science · doi:10.3389/fmars.2022.986206 · Cited by 47
Multimodal Deep Learning Crime Prediction Using Tweets
Sakirin, T., & Tanrıöver, Ö. Ö. (2023) · IEEE Access · doi:10.1109/access.2023.3308967 · Cited by 58
Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation
Liu, H., Wei, Y., Liu, F., Wang, W., Nie, L., & Chua, T. (2023) · ACM Transactions on Information Systems · doi:10.1145/3617827 · Cited by 69
Co-space Representation Interaction Network for multimodal sentiment analysis
Shi, H., Pu, Y., Zhao, Z., Huang, J., Zhou, D., Xu, D., & Cao, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111149 · Cited by 53
IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-Training
Liu, C., Cheng, S. L., Shi, M., Shah, A., Bai, W., & Arcucci, R. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3449690 · Cited by 60
A survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 59
Leveraging Knowledge of Modality Experts for Incomplete Multimodal Learning
Xu, W., Jiang, H., & Liang, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681683 · Cited by 102
Multimodal Attention-Mechanism For Temporal Emotion Recognition
Ghaleb, E., Niehues, J., & Asteriadis, S. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191019 · Cited by 57
Li, H., Yang, F., Xing, X., Zhao, Y., Zhang, J., Liu, Y., Han, M., Huang, J., Wang, L., & Yao, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87237-3_51 · Cited by 64
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive SurveyPreprint
Chen, L., Wang, Z., Ren, S., Li, L., Zhao, H., Li, Y., Cai, Z., Guo, H., Zhang, L., Xiong, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.18619 · Cited by 57
Xiao, S., Chen, G., Zhang, C., & Li, X. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118138 · Cited by 47
Cui, C., Liu, H., Liu, Q., Deng, R., Asad, Z., Wang, Y., Zhao, S., Yang, H., Landman, B. A., & Huo, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_60 · Cited by 65
Geometric Correspondence-Based Multimodal Learning for Ophthalmic Image Analysis
Wang, Y., Zhen, L., Tan, T., Fu, H., Feng, Y., Wang, Z., Xu, X., Goh, R. S. M., Ng, Y. X., Calhoun, C. T., et al. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3352602 · Cited by 49
Shareef, A. M. A., Seçkiner, S. U., Eid, B. M., & Abumeteir, H. A. (2024) · Frontiers in Energy Research · doi:10.3389/fenrg.2024.1377950 · Cited by 54
Duy, P. T., Khoa, N. H., Quyen, N. H., Trinh, L. C., Kien, V. T., Hoang, T. M., & Pham, V.-H. (2024) · International Journal of Information Security · doi:10.1007/s10207-024-00965-2 · Cited by 51
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.