Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 801–900 of 1,036
Machine learning-based in-process monitoring for laser deep penetration welding: A survey
Lu, R., Lou, M., Xia, Y.-J., Huang, S., Li, Z., Lyu, T., Wu, Y., & Li, Y. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109059 · Cited by 41
Visual Question Answering: A Survey of Methods, Datasets, Evaluation, and Challenges
Kim, B. S., Kim, J., Lee, D., & Jang, B. (2025) · ACM Computing Surveys · doi:10.1145/3728635 · Cited by 79
PRIME 2.0: Proposed Requirements for Cardiovascular Imaging-Related Multimodal-AI Evaluation
Kagiyama, N., Tokodi, M., Hathaway, Q. A., Arnaout, R., Davies, R. H., Dey, D., Duchateau, N., Fraser, A. G., Goto, S., Jamthikar, A. D., et al. (2025) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2025.08.004 · Cited by 44
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384452 · Cited by 35
Somani, S. S., Honarvar, H., Narula, S., Landi, I., Lee, S. Y., Khachatoorian, Y., Rehmani, A., Kim, A., Freitas, J. K. D., Teng, S., et al. (2021) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztab101 · Cited by 38
Hazmy, A. I., Hawbani, A., Wang, X., Al‐Dubai, A., Ghannami, A., Yahya, A. A., Zhao, L., & Alsamhi, S. H. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3343428 · Cited by 42
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets
Hessel, J., Mimno, D., & Lee, L. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1199 · Cited by 38
Multimodal Language Analysis with Recurrent Multistage FusionPreprint
Liang, P. P., Ziyin, L., Zadeh, A., & Morency, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.03920 · Cited by 56
Self-citationMultimodal heterogeneous graph attention network
Jia, X., Jiang, M., Dong, Y., Zhu, F., Lin, H., Xin, Y., & Chen, H. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07862-6 · Cited by 51
Non-contact multimodal indoor human monitoring systems: A survey
Nguyen, L. N., Susarla, P., Mukherjee, A., Cañellas, M. L., Casado, C. Á., Wu, X., Sílven, O., Jayagopi, D. B., & López, M. B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102457 · Cited by 52
Ensemble Pretrained Models for Multimodal Sentiment Analysis using Textual and Video Data Fusion
Liu, Z., Braytee, A., Anaissi, A., Zhang, G., Qin, L., & Akram, J. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651971 · Cited by 46
Zeng, L., Ma, P., Li, Z., Liang, S., Wu, C., Hong, C., Li, Y., Cui, H., Li, R., Wang, J., et al. (2024) · Advanced Science · doi:10.1002/advs.202404047 · Cited by 37
Multimodal music datasets? Challenges and future goals in music processing
Christodoulou, A.-M., Lartillot, O., & Jensenius, A. R. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00344-6 · Cited by 43
A review and critical analysis of multimodal datasets for emotional AI
Al-Azani, S., & El-Alfy, E.-S. M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11271-1 · Cited by 40
Retro-Remote Sensing: Generating Images From Ancient Texts
Bejiga, M. B., Melgani, F., & Vascotto, A. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2895693 · Cited by 39
How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
Sterpu, G., Saam, C., & Harte, N. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2980436 · Cited by 41
Nikzad-Khasmakhi, N., Balafar, M. A., Feizi‐Derakhshi, M., & Motamed, C. (2021) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2021.111260 · Cited by 42
Hierarchical Federated Learning for Hybrid Data Partitioning Across Multitype Sensors
Su, L., & Lau, V. K. N. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3051382 · Cited by 42
Pretraining model for biological sequence data
Song, B., Li, Z., Lin, X., Wang, J., Wang, T., & Fu, X. (2021) · Briefings in Functional Genomics · doi:10.1093/bfgp/elab025 · Cited by 39
AOMD: An analogy-aware approach to offensive meme detection on social media
Shang, L., Zhang, Y., Zha, Y., Chen, Y., Youn, C., & Wang, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102664 · Cited by 48
Improving Multimodal fusion via Mutual Dependency Maximisation
Colombo, P., Chapuis, É., Labeau, M., & Clavel, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.21 · Cited by 39
An Adaptive Graph Morlet Wavelet Transform for Railway Wayside Acoustic Detection
Zhang, D., Xie, M., Hamadache, M., Entezami, M., & Stewart, E. (2022) · Journal of Sound and Vibration · doi:10.1016/j.jsv.2022.116965 · Cited by 37
Xu, X., Zhu, Q., Ying, H., Li, J., Cai, X., Li, S., Liu, X., & Yu, Y. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3220788 · Cited by 38
Multimodal Arabic Rumors Detection
Albalawi, R. M., Jamal, A. T., Khadidos, A. O., & Alhothali, A. (2023) · IEEE Access · doi:10.1109/access.2023.3240373 · Cited by 48
Multimodal deep learning for short-term stock volatility predictionPreprint
Nascimento, M. S., & Manandhar, S. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1812.10479 · Cited by 42
Multimodal Generative Learning Utilizing Jensen-Shannon DivergencePreprint
Sutter, T. M., Daunhawer, I., & Vogt, J. E. (2020) · arXiv (Cornell University) · doi:10.3929/ethz-b-000456059 · Cited by 50
Dar, S. S., Rehman, M. Z. U., Bais, K., Haseeb, M. A., & Kumar, N. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125337 · Cited by 36
Robust Camera Lidar Sensor Fusion Via Deep Gated Information Fusion Network
Kim, J., Choi, J., Kim, Y., Koh, J., Chung, C. C., & Choi, J. W. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500711 · Cited by 36
Multiple color representation and fusion for diabetes mellitus diagnosis based on back tongue images
Zhang, N., Zhang, D. D., Li, J., & Zhang, D. D. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106652 · Cited by 40
Wei, S., Luo, Y., Ma, X., Ren, P., & Luo, C. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3265650 · Cited by 53
Multimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 54
An Investigation of Applying Large Language Models to Spoken Language Learning
Gao, Y., Nuchged, B., Li, Y., & Peng, L. (2023) · Applied Sciences · doi:10.3390/app14010224 · Cited by 41
Temporal cues enhanced multimodal learning for action recognition in RGB-D videos
Liu, D., Meng, F., Xia, Q., Ma, Z., Mi, J., Gan, Y., Ye, M., & Zhang, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127882 · Cited by 38
Ganhör, C., Moscati, M., Hausberger, A., Nawaz, S., & Schedl, M. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688138 · Cited by 54
Graph-based multimodal fusion with metric learning for multimodal classification
Angelou, M., Solachidis, V., Vretos, N., & Daras, P. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.06.013 · Cited by 41
Vector score alpha integration for classifier late fusion
Safont, G., Salazar, A., & Vergara, L. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.05.014 · Cited by 38
On Robustness of Multi-Modal Fusion—Robotics Perspective
Bednarek, M., Kicki, P., & Walas, K. (2020) · Electronics · doi:10.3390/electronics9071152 · Cited by 43
GoG: Relation-aware Graph-over-Graph Network for Visual Dialog
Chen, F., Chen, X., Meng, F., Li, P., & Zhou, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.20 · Cited by 42
Bayesian metamodeling of complex biological systems across varying representations
Raveh, B., Sun, L., White, K. L., Sanyal, T., Tempkin, J. O. B., Zheng, D., Bharath, K., Singla, J., Wang, C., Zhao, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2104559118 · Cited by 37
Exploiting Cross-Modal Prediction and Relation Consistency for Semisupervised Image Captioning
Yang, Y., Wei, H., Zhu, H., Yu, D., Xiong, H., & Yang, J. J. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3156367 · Cited by 49
Kang, P., Li, J., Jiang, S., & Shull, P. B. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3232159 · Cited by 43
Correlated Attention Networks for Multimodal Emotion Recognition
Qiu, J., Li, X., & Hu, K. (2018) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2018.8621129 · Cited by 39
COM: Contrastive Masked-attention model for incomplete multimodal learning
Qian, S., & Wang, C. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.03.003 · Cited by 45
Li, L., Zhao, W., Wang, C., Fotouhi, A., & Liu, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121458 · Cited by 35
Hou, K., Zhang, X., Yang, Y., Zhao, Q., Yuan, W., Zhou, Z., Zhang, S., Li, C., Shen, J. G., & Hu, B. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3320107 · Cited by 38
Multimodal Automated Fact-Checking: A Survey
Akhtar, M., Schlichtkrull, M., Guo, Z., Cocarascu, O., Simperl, E. P. B., & Vlachos, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.361 · Cited by 99
Adapt and explore: Multimodal mixup for representation learning
Lin, R., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102216 · Cited by 52
Cao, Y., Tang, Q., Yang, F. Y., Su, X. Q., You, S., Lu, X., & Xu, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02147 · Cited by 48
Prediction of ultimate tensile strength of Al‐Si alloys based on multimodal fusion learning
Zhu, L., Luo, Q., Chen, Q., Zhang, Y. X., Zhang, L., Hu, B., Han, Y., & Li, Q. (2024) · Materials Genome Engineering Advances · doi:10.1002/mgea.26 · Cited by 46
A survey of methods for addressing the challenges of referring image segmentation
Ji, L., Du, Y., Dang, Y., Gao, W., & Zhang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127599 · Cited by 42
Chen, X., Singh, M. M., & Geyer, P. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111774 · Cited by 38
Bi-Level Graph Structure Learning for Next POI Recommendation
Wang, L., Wu, S., Liu, Q., Zhu, Y., Tao, X., Zhang, M., & Wang, L. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3397683 · Cited by 39
GUME: Graphs and User Modalities Enhancement for Long-Tail Multimodal Recommendation
Lin, G., Zhen, M., Wang, D., Long, Q., Zhou, Y., & Xiao, M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679620 · Cited by 58
GLoMo: Global-Local Modal Fusion for Multimodal Sentiment Analysis
Yan, Z., Zhang, Y., Hu, Z., Zhang, X., Deng, J., & Ren, F. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681527 · Cited by 54
Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System Design
Hu, Y. '., Tang, J., Gong, X., Zhou, Z., Zhang, S., Elvitigala, D. S., Mueller, F., Hu, W. S., & Quigley, A. J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714161 · Cited by 42
Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
Ma, F., Zhang, W., Li, Y., Huang, S., & Zhang, L. (2020) · Applied Sciences · doi:10.3390/app10207239 · Cited by 38
Review of AI-based methods for chatter detection in machining based on bibliometric analysis
Kounta, C. A. K. A., Arnaud, L., Kamsu-Foguem, B., & Tangara, F. (2022) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-022-10059-9 · Cited by 36
Multimodality information fusion for automated machine translation
Li, L., Tayir, T., Han, Y., Tao, X., & Velásquez, J. D. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.018 · Cited by 35
Satellite MEC with Federated Learning: Architectures, Technologies and Challenges
Yi, J., Wang, J., Jiang, C., & Zhan, Y. (2022) · IEEE Network · doi:10.1109/mnet.001.2200202 · Cited by 35
Kikuchi, T., Sakita, K., Nishiyama, S., & Takahashi, K. (2023) · Natural Hazards · doi:10.1007/s11069-023-05862-w · Cited by 36
AcFormer: An Aligned and Compact Transformer for Multimodal Sentiment Analysis
Zong, D., Ding, C., Li, B., Li, J., Zheng, K., & Zhou, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611974 · Cited by 47
CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities
Kontras, K., Chatzichristos, C., Phan, H., Suykens, J. A. K., & Vos, M. D. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2024.3354388 · Cited by 40
Dual-view multi-modal contrastive learning for graph-based recommender systems
Guo, F., Wang, Z., Wang, X., Lu, Q., & Ji, S. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109213 · Cited by 35
Perspectives on Advancing Multimodal Learning in Environmental Science and Engineering Studies
Liu, W., Chen, J., Wang, H., Fu, Z., Peijnenburg, W. J. G. M., & Hong, H. (2024) · Environmental Science & Technology · doi:10.1021/acs.est.4c03088 · Cited by 54
A novel multi-modal fusion method based on uncertainty-guided meta-learning
Zhang, D., Bashar, M. A., & Nayak, R. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110993 · Cited by 36
Liang, K., Meng, L., Liu, Y., Liu, M., Wei, W., Liu, S., Tu, W., Wang, S., Zhou, S., & Liu, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681112 · Cited by 56
Sun, D., Li, Y., Jia, S., Gao, S., Noman, K., & Eliker, K. (2024) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2024.112222 · Cited by 34
A Joint Label Space for Generalized Zero-Shot Classification
Li, J., Lan, X., Long, Y., Liu, Y., Chen, X., Shao, L., & Zheng, N. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2986892 · Cited by 37
Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
Dai, W., Liu, Z., Yu, T., & Fung, P. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.30 · Cited by 47
Digital Phenotyping Using Multimodal Data
Cohen, A. S., Cox, C. R., Masucci, M. D., Le, T. P., Cowan, T., Coghill, L. M., Holmlund, T. B., & Elvevåg, B. (2020) · Current Behavioral Neuroscience Reports · doi:10.1007/s40473-020-00215-4 · Cited by 35
Auto-weighting for Breast Cancer Classification in Multimodal Ultrasound
Wang, J., Miao, J., Yang, X., Li, R., Zhou, G., Huang, Y., Lin, Z., Xue, W., Jia, X., Zhou, J., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59725-2_19 · Cited by 41
A Comprehensive Survey on Multimodal Data Representation and Information Fusion Algorithms
Gaonkar, A., Chukkapalli, Y., Raman, P. J., Srikanth, S., & Gurugopinath, S. (2021) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit51480.2021.9498415 · Cited by 42
Multimodal Sentiment Analysis based on Recurrent Neural Network and Multimodal Attention
Cai, C., He, Y. L., Sun, L., Lian, Z., Liu, B., Tao, J., Xu, M., & Wang, K. (2021) · doi:10.1145/3475957.3484454 · Cited by 38
DCAE: A dual conditional autoencoder framework for the reconstruction from EEG into image
Zeng, H., Xia, N., Tao, M., Pan, D., Zheng, H., Wang, C., Xu, F., Zakaria, W., & Dai, G. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.104440 · Cited by 36
UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning
Zou, H., Shen, M., Chen, C., Hu, Y., Rajan, D., & Chng, E. S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.41 · Cited by 44
Improving Affect Detection in Game-Based Learning with Multimodal Data Fusion
Henderson, N., Rowe, J., Paquette, L., Baker, R. S., & Lester, J. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52237-7_19 · Cited by 34
Wang, L., Wang, C., Li, C., Murai, T., Bai, Y., Song, Z., Zhang, S., Zhang, Q., Huang, Y., Bi, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01933-3 · Cited by 33
“However, each of these public datasets, as well as the private datasets used in included studies, contains no more than 280 subjects (with sample sizes ranging from 24 to 275), which hampers the ability to identify clinical biomarkers universally and reliably, especially with DL approaches 132,133 .”
Understanding image-text relations and news values for multimodal news analysis
Cheema, G. S., Hakimov, S., Müller-Budack, E., Otto, C., Bateman, J. A., & Ewerth, R. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1125533 · Cited by 37
MoCG: Modality Characteristics-Guided Semantic Segmentation in Multimodal Remote Sensing Images
Xiao, S., Wang, P., Diao, W., Rong, X., Li, X., Fu, K., & Sun, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3334471 · Cited by 33
Comparison of dimensionality reduction techniques for multi-variable spatiotemporal flow fields
Wang, Z., Zhang, G., Xing, X., Xu, X., & Sun, T. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.116421 · Cited by 36
Tian, F., Zhang, L., Zhu, L., Zhao, M., Liu, J., Dong, Q., & Zhao, Q. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3420445 · Cited by 33
Rethinking prediction-based video anomaly detection from local–global normality perspective
Zhao, M., Zeng, X., Liu, Y., Liu, J., & Pang, C. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125581 · Cited by 37
Text–image multimodal fusion model for enhanced fake news detection
Lin, S., Chen, Y.-C., Chang, Y.-H., Lo, S.-H., & Chao, K. (2024) · Science Progress · doi:10.1177/00368504241292685 · Cited by 41
Speech-Based Depression Assessment: A Comprehensive Survey
Leal, S. S., Ntalampiras, S., & Sassi, R. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3521327 · Cited by 41
Advancements in Medical Radiology Through Multimodal Machine Learning: A Comprehensive Overview
Haq, I. U., Mhamed, M., Alharbi, M., Osman, H., Hamd, Z. Y., & Liu, Z. (2025) · Bioengineering · doi:10.3390/bioengineering12050477 · Cited by 40
Designing accurate emulators for scientific processes using calibration-driven deep models
Thiagarajan, J. J., Venkatesh, B., Anirudh, R., Bremer, P., Gaffney, J., Anderson, G. J., & Spears, B. K. (2020) · Nature Communications · doi:10.1038/s41467-020-19448-8 · Cited by 38
Integrating information theory and adversarial learning for cross-modal retrieval
Chen, W., Liu, Y., Bakker, E. M., & Lew, M. S. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107983 · Cited by 38
Efficient Environmental Context Prediction for Lower Limb Prostheses
Zhong, B., da Silva, R. L., Tran, M., Huang, H., & Lobaton, E. J. (2021) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2021.3084036 · Cited by 33
Alam, M. U., & Rahmani, R. (2021) · Sensors · doi:10.3390/s21155025 · Cited by 36
Pareto optimization of deep networks for COVID-19 diagnosis from chest X-rays
Guarrasi, V., D’Amico, N. C., Sicilia, R., Cordelli, E., & Soda, P. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108242 · Cited by 33
SAR image classification based on multi‐feature fusion decision convolutional neural network
Guo, L. (2021) · IET Image Processing · doi:10.1049/ipr2.12323 · Cited by 33
Deep Multimodal Complementarity Learning
Wang, D., Zhao, T., Yu, W., Chawla, N. V., & Jiang, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3165180 · Cited by 40
KIEGLFN: A unified acne grading framework on face images
Yi, L., Jiang, J., Ma, Z., Chen, D., Guan, Y., You, H., Cheng, X., Liu, B., & Luo, G. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106911 · Cited by 37
Abtahi, M., Le, D., Lim, J. I., & Yao, X. (2022) · Biomedical Optics Express · doi:10.1364/boe.468483 · Cited by 36
Multimodal Sentiment Recognition With Multi-Task Learning
Zhang, S., Yin, C., & Yin, Z. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3224929 · Cited by 36
Hu, F., Hu, Y., Zhang, W., Huang, H., Pan, Y. F., & Yin, P. (2023) · Advanced Science · doi:10.1002/advs.202301223 · Cited by 40
Deep Hierarchical Fusion with Application in Sentiment Analysis
Georgiou, E., Papaioannou, C., & Potamianos, A. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3243 · Cited by 41
Unveiling Shadows: Harnessing Artificial Intelligence for Insider Threat Detection
Yilmaz, E., & Can, O. (2024) · Engineering, Technology & Applied Science Research · doi:10.48084/etasr.6911 · Cited by 32
Prospects for multi-agent collaboration and gaming: challenge, technology, and application
Liu, Y., Li, Z., Jiang, Z., & He, Y. (2022) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2200055 · Cited by 34
Multimodal fusion recognition for digital twin
Zhou, T., Zhang, X., Kang, B., & Chen, M. (2022) · Digital Communications and Networks · doi:10.1016/j.dcan.2022.10.009 · Cited by 38
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.