Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Items 901–1,000 of 1,036
A Multimodal Social Signal Processing Approach to Team Interactions
Lehmann‐Willenbrock, N., & Hung, H. (2023) · Organizational Research Methods · doi:10.1177/10944281231202741 · Cited by 40
Intra- and Inter-Modal Curriculum for Multimodal Learning
Zhou, Y., Wang, X. Y., Chen, H., Duan, X., & Zhu, W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612468 · Cited by 48
Confidence-aware multi-modality learning for eye disease screening
Zou, K., Lin, T., Han, Z., Wang, M., Yuan, X., Chen, H., Zhang, C., Shen, X., & Fu, H. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103214 · Cited by 42
Liang, H., Zhang, J., Li, Y., Wang, B., & Huang, J. (2024) · IEEE Access · doi:10.1109/access.2024.3408843 · Cited by 37
S2MVTC: A Simple Yet Efficient Scalable Multi-View Tensor Clustering
Long, Z., Wang, Q., Ren, Y., Liu, Y., & Zhu, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02274 · Cited by 43
From data analysis to intelligent maintenance: a survey on visual defect detection in aero-engines
Wu, P., Li, H., Luo, X., Hu, L., Yang, R., & Zeng, N. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/add6c8 · Cited by 47
MARIA: A multimodal transformer model for incomplete healthcare data
Caruso, C. M., Soda, P., & Guarrasi, V. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110843 · Cited by 32
Understanding, Categorizing and Predicting Semantic Image-Text Relations
Otto, C., Springstein, M., Anand, A., & Ewerth, R. (2019) · International Conference on Multimedia Retrieval · doi:10.1145/3323873.3325049 · Cited by 32
Multi-view Factorization AutoEncoder with Network Constraints for Multi-omic Integrative Analysis
Ma, T., & Zhang, A. (2018) · 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2018.8621379 · Cited by 30
Multifaceted radiomics for distant metastasis prediction in head & neck cancer
Zhou, Z., Wang, K., Folkert, M. R., Liu, H., Jiang, S. B., Sher, D. J., & Wang, J. (2020) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ab8956 · Cited by 32
Multi-view self-attention for interpretable drug–target interaction prediction
Agyemang, B., Wu, W., Kpiebaareh, M. Y., Lei, Z., Nanor, E., & Chen, L. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103547 · Cited by 35
AiRound and CV-BrCT: Novel Multiview Datasets for Scene Classification
Machado, G. L. S., Ferreira, E., Nogueira, K., Oliveira, H., Brito, M., Gama, P. H. T., & dos Santos, J. A. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3033424 · Cited by 32
EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset
Northcutt, C. G., Zha, S., Lovegrove, S. J., & Newcombe, R. A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3025105 · Cited by 43
Stroke constrained attention network for online handwritten mathematical expression recognition
Wang, J., Du, J., Zhang, J., Wang, B., & Ren, B. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108047 · Cited by 32
Self-Supervised Multimodal Opinion Summarization
Im, J., Kim, M., Lee, H., Cho, H., & Chung, S. (2021) · doi:10.18653/v1/2021.acl-long.33 · Cited by 44
Real-Time Task Scheduling for Machine Perception in In Intelligent Cyber-Physical Systems
Liu, S., Yao, S., Fu, X., Shao, H., Tabish, R., Yu, S., Bansal, A., Yun, H., Sha, L., & Abdelzaher, T. (2021) · IEEE Transactions on Computers · doi:10.1109/tc.2021.3106496 · Cited by 46
Multi-Modal Contrastive Pre-training for Recommendation
Liu, Z., Ma, Y., Schubert, M., Ouyang, Y., & Zhang, X. (2022) · International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531378 · Cited by 48
Bai, N., Nourian, P., Luo, R., & Roders, A. P. (2022) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi11090469 · Cited by 37
What Is a Multi-Modal Knowledge Graph: A Survey
Peng, J., Hu, X., Huang, W., & Yang, J. K. (2023) · Big Data Research · doi:10.1016/j.bdr.2023.100380 · Cited by 42
Detecting hate speech in memes: a review
Hermida, P., & Santos, E. M. D. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10459-7 · Cited by 53
Razzaq, M. A., Hussain, J., Bang, J., Hua, C.-H., Satti, F. A., Rehman, U. U., Bilal, H. S. M., Kim, S. T., & Lee, S. (2023) · Sensors · doi:10.3390/s23094373 · Cited by 36
Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis
Mao, H., Zhang, B., Xu, H., Yuan, Z., & Liu, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.27078 · Cited by 40
Artificial intelligence accelerates multi-modal biomedical process: A Survey
Li, J., Han, X., Qin, Y., Tan, F., Chen, Y., Wang, Z., Song, H., Zhou, X., Zhang, Y., Hu, L., et al. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126720 · Cited by 35
Regularized Vector Quantization for Tokenized Image Synthesis
Zhang, J., Zhan, F., Theobalt, C., & Lu, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01771 · Cited by 83
Wang, Y., Lei, Y., Li, N., Gao, X., Liu, X., Ren, Q., & Jiang, J. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110025 · Cited by 32
Semantic Drift in Multilingual Representations
Beinborn, L., & Choenni, R. (2020) · Computational Linguistics · doi:10.1162/coli_a_00382 · Cited by 48
Harmonized Multimodal Learning with Gaussian Process Latent Variable Models
Song, G., Wang, S., Huang, Q., & Tian, Q. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2942028 · Cited by 35
COBRA: Contrastive Bi-Modal Representation AlgorithmPreprint
Udandarao, V., Maiti, A., Srivatsav, D., Vyalla, S. R., Yin, Y., & Shah, R. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03687 · Cited by 36
Visual question answering: from early developments to recent advances -- a surveyPreprint
Huynh, N. D., Bouadjenek, M. R., Aryal, S., Razzak, I., & Hacid, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.03939 · Cited by 50
Multimodal explainability via latent shift applied to COVID-19 stratification
Guarrasi, V., Tronchin, L., Albano, D., Faiella, E., Fazzini, D., Santucci, D., & Soda, P. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110825 · Cited by 31
Assessing learners’ English public speaking anxiety with multimodal deep learning technologies
Zheng, C., Zhang, T., Chen, X., Zhang, H., Wan, J., & Wu, B. (2024) · Computer Assisted Language Learning · doi:10.1080/09588221.2024.2351129 · Cited by 35
Zhang, J., Shen, D., Chen, D., Ming, D., Ren, D., & Diao, Z. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109089 · Cited by 31
Faria, D. R., Godkin, A. L., & Ayrosa, P. P. S. (2025) · Sensors · doi:10.3390/s25041161 · Cited by 38
Chen, F., Li, Y., Chen, Y., Bian, Z., Duo, L., Zhou, Q., Zhang, L., & ADVANCED Working Group (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70075 · Cited by 34
Artificial Intelligence in Endurance Sports: Metabolic, Recovery, and Nutritional Perspectives
Grivas, G. V., & Safari, K. (2025) · Nutrients · doi:10.3390/nu17203209 · Cited by 34
Liu, J., Meng, S., Gao, Y., Mao, S., Cai, P., Yan, G., Chen, Y., Bian, Z., Wang, D., & Shi, B. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00099 · Cited by 54
Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy
Pattnayak, P., Patel, H. L., Kumar, B., Agarwal, A., Banerjee, I., Panda, S., & Kumar, T. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14041-8_4 · Cited by 33
Novel Framework Based on HOSVD for Ski Goggles Defect Detection and Classification
Le, N. T., Wang, J.-W., Wang, C.-C., & Nguyen, T. N. (2019) · Sensors · doi:10.3390/s19245538 · Cited by 32
Low-latency speculative inference on distributed multi-modal data streams
Li, T., Huang, J., Risinger, E., & Ganesan, D. (2021) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3458864.3467884 · Cited by 35
Li, S., Xie, Y., Wang, G., Zhang, L., & Zhou, W. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3161466 · Cited by 36
Zhu, J., Yang, C., Xie, X., Wei, S., Li, Y., Li, X., & Hu, B. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3171782 · Cited by 35
Automatic Chart Understanding: A Review
Farahani, A. M., Adibi, P., Ehsani, M. S., Hutter, H.-P., & Darvishy, A. (2023) · IEEE Access · doi:10.1109/access.2023.3298050 · Cited by 38
Xiao, J., Li, J., & Gao, H. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3301874 · Cited by 33
Mulsemedia communication research challenges for metaverse in 6G wireless systems
Akyildiz, I. F., Guo, H., Dai, R., & Gerstacker, W. (2023) · ITU Journal on Future and Evolving Technologies · doi:10.52953/fpky2771 · Cited by 31
Topic prevalence and trends of metaverse in healthcare: a bibliometric analysis
Wu, P., Chen, D., & Zhang, R. (2023) · Data Science and Management · doi:10.1016/j.dsm.2023.12.003 · Cited by 35
Zhou, P., Gao, B., Zhao, C., & Chai, T. (2023) · Control Engineering Practice · doi:10.1016/j.conengprac.2023.105479 · Cited by 31
EXIF as Language: Learning Cross-Modal Associations between Images and Camera Metadata
Zheng, C., Shrivastava, A., & Owens, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00671 · Cited by 37
WFNet: A hierarchical convolutional neural network for wildfire spread prediction
Jiang, W., Qiao, Y., Su, G., Li, X., Meng, Q., Wu, H., Quan, W., Wang, J., & Wang, F. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105841 · Cited by 35
Crossmodal Translation Based Meta Weight Adaption for Robust Image-Text Sentiment Analysis
Zhang, B., Yuan, Z., Xu, H., & Gao, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3405662 · Cited by 33
Artificial intelligence for the study of human ageing: a systematic literature review
Bernal, M. C., Batista, E., Martínez-Ballesté, A., & Solanas, A. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05817-z · Cited by 32
Multi-modal learning-based algae phyla identification using image and particle modalities
Kwon, D. H., Lee, M. J., Jeong, H., Park, S., & Cho, K. H. (2025) · Water Research · doi:10.1016/j.watres.2025.123172 · Cited by 31
UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs
He, Y., Sui, Y., He, X., Liu, Y., Sun, Y., & Hooi, B. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714818 · Cited by 75
Pang, H., Zhang, C., Wang, F., Liu, J., & Sun, L. (2019) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2019.8737395 · Cited by 30
Strong and Simple Baselines for Multimodal Utterance Embeddings
Liang, P. P., Lim, Y. C., Tsai, Y.-H. H., Salakhutdinov, R., & Morency, L.-P. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1267 · Cited by 27
Self-citationAdversarial Learning-Based Semantic Correlation Representation for Cross-Modal Retrieval
Zhu, L., Song, J., Zhu, X., Zhang, C., Zhang, S., & Yuan, X. (2020) · IEEE Multimedia · doi:10.1109/mmul.2020.3015764 · Cited by 31
Adaptive Multimodal Fusion for Facial Action Units Recognition
Yang, H., Wang, T., & Yin, L. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413538 · Cited by 36
Li, Y., Wang, B., Shao, G., & Shao, S. (2020) · IEEE Access · doi:10.1109/access.2020.3046124 · Cited by 31
Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection
Mathur, L., & Matarić, M. J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413550 · Cited by 30
Multimodal CNN Fusion Architecture with Multi-Features for Heart Sound Classification
Ranipa, K., Zhu, W., & Swamy, M. N. S. (2021) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas51556.2021.9401551 · Cited by 30
Multi-view Teacher–Student Network
Tian, Y., Sun, S., & Tang, J. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.11.002 · Cited by 33
Deep Multimodal Guidance for Medical Image Classification
Mallya, M., & Hamarneh, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_29 · Cited by 43
Geometric Multimodal Deep Learning With Multiscaled Graph Wavelet Convolutional Network
Behmanesh, M., Adibi, P., Ehsani, M. S., & Chanussot, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3213589 · Cited by 32
Automated testing of image captioning systems
Yu, B., Zhong, Z., Qin, X., Yao, J., Wang, Y., & He, P. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534389 · Cited by 38
Predicting the evolution of COVID-19 mortality risk: A Recurrent Neural Network approach
Villegas, M., González-Agirre, A., Gutiérrez-Fandiño, A., Armengol-Estapé, J., Carrino, C. P., Pérez‐Fernández, D., Soares, F., Balazote, P. S., Pedrera‐Jiménez, M., Barrio, N. G., et al. (2022) · Computer Methods and Programs in Biomedicine Update · doi:10.1016/j.cmpbup.2022.100089 · Cited by 31
Guarrasi, V., & Soda, P. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106625 · Cited by 28
Xi, J., Sun, D., Chang, C., Zhou, S., & Huang, Q. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106672 · Cited by 30
Category-Aware Multimodal Attention Network for Fashion Compatibility Modeling
Jing, P., Cui, K., Guan, W., Nie, L., & Su, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3246796 · Cited by 30
Biswas, S. D., Kosta, A. K., Liyanagedera, C., Apolinario, M. P. E., & Roy, K. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00586 · Cited by 46
Modeling Both Intra- and Inter-Modality Uncertainty for Multimodal Fake News Detection
Wei, L., Hu, D., Zhou, W. J., & Hu, S. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3229966 · Cited by 34
A survey of multimodal federated learning: background, applications, and perspectives
Pan, H., Zhao, X., He, L., Shi, Y., & Lin, X. S. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01422-9 · Cited by 33
Innovating beyond electrophysiology through multimodal neural interfaces
Ramezani, M., Ren, Y., Cubukcu, E., & Kuzum, D. (2024) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-024-00121-x · Cited by 32
Surveying the deep: A review of computer vision in the benthos
Trotter, C., Griffiths, H. J., & Whittle, R. J. (2025) · Ecological Informatics · doi:10.1016/j.ecoinf.2024.102989 · Cited by 31
Multimodal destination image and user engagement: a sequential research design
Tan, J., Cheng, M., Chen, J., Zhu, J., Yu, Q., & Chen, S. (2025) · Tourism Management · doi:10.1016/j.tourman.2025.105209 · Cited by 34
Multimodal integration strategies for clinical application in oncology
Zhang, B., Wan, Z., Luo, Y., Zhao, X., Samayoa, J. A., Zhao, W., & Wu, S. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1609079 · Cited by 36
Ghost Imputation: Accurately Reconstructing Missing Data of the Off Period
Rawassizadeh, R., Keshavarz, H., & Pazzani, M. J. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2914653 · Cited by 30
Haputhanthri, D., Silva, D. D., Sierla, S., Alahakoon, D., Nawaratne, R., Jennings, A., & Vyatkin, V. V. (2021) · Frontiers in Energy Research · doi:10.3389/fenrg.2021.722212 · Cited by 29
Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics
Cwiek, A. P., Rajtmajer, S. M., Wyble, B., Honavar, V., Grossner, E. C., & Hillary, F. G. (2022) · Network Neuroscience · doi:10.1162/netn_a_00212 · Cited by 30
Applications of deep learning for mobile malware detection: A systematic literature review
Catal, C., Giray, G., & Tekinerdoğan, B. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06597-0 · Cited by 32
Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation
Katada, S., Okada, S., & Komatani, K. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3155604 · Cited by 40
Water chemical oxygen demand prediction model based on the CNN and ultraviolet-visible spectroscopy
Ye, B., Cao, X., Liu, H., Wang, Y., Tang, B., Chen, C., & Chen, Q. (2022) · Frontiers in Environmental Science · doi:10.3389/fenvs.2022.1027693 · Cited by 29
Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss
Franceschini, R., Fini, E., Beyan, C., Conti, A., Arrigoni, F., & Ricci, E. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956589 · Cited by 34
WiFE: WiFi and Vision Based Unobtrusive Emotion Recognition via Gesture and Facial Expression
Gu, Y., Zhang, X., Yan, H., Huang, J., Liu, Z., Dong, M., & Ren, F. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3285777 · Cited by 37
Chen, D., & Zhang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3291503 · Cited by 33
Kwon, D., Woo, S. Y., Lee, K.-H., Hwang, J., Kim, H., Park, S., Shin, W., Bae, J., Kim, J., & Lee, J. (2023) · Science Advances · doi:10.1126/sciadv.adg9123 · Cited by 29
Multimodal analysis methods in predictive biomedicine
Qoku, A., Katsaouni, N., Flinner, N., Buettner, F., & Schulz, M. H. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.11.011 · Cited by 29
sEMG-Based Gesture Recognition With Embedded Virtual Hand Poses and Adversarial Learning
Hu, Y., Wong, Y., Dai, Q., Kankanhalli, M., Geng, W., & Li, X. (2019) · IEEE Access · doi:10.1109/access.2019.2930005 · Cited by 31
Multi-Modal Diagnosis of Infectious Diseases in the Developing World
Tadesse, G. A., Javed, H., Thanh, N. L. N., Thi, H. D. H., Tan, L. V., Thwaites, L., Clifton, D. A., & Zhu, T. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2959839 · Cited by 29
Characterization and classification of semantic image-text relations
Otto, C., Springstein, M., Anand, A., & Ewerth, R. (2020) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-019-00187-6 · Cited by 29
Multimodal Deep Learning for Mental Disorders Prediction from Audio Speech SamplesPreprint
Naderi, H., Soleimani, B. H., & Matwin, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01067 · Cited by 35
What is Multimodality?Preprint
Pârcălăbescu, L., Trost, N., & Frank, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06304 · Cited by 34
DravidianMultiModality: A Dataset for Multi-modal Sentiment Analysis in Tamil and MalayalamPreprint
Chakravarthi, B. R., K, J. P. P., Premjith, B., Soman, K. P., Ponnusamy, R., Kumaresan, P. K., Thamburaj, K. P., & McCrae, J. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04853 · Cited by 35
Learning Multimodal Data Augmentation in Feature SpacePreprint
Liu, Z., Tang, Z., Shi, X., Zhang, A., Li, M., Shrivastava, A., & Wilson, A. G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.14453 · Cited by 34
Deep learning on multi-view sequential data: a survey
Xie, Z., Yang, Y. Z., Zhang, Y., Wang, J., & Du, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10332-z · Cited by 30
MFFNet: multimodal feature fusion network for point cloud semantic segmentation
Ren, D., Li, J., Wu, Z., Guo, J. J., Wei, M., & Guo, Y. (2023) · The Visual Computer · doi:10.1007/s00371-023-02907-w · Cited by 33
Karimanzira, D., Weis, J., Wünsch, A., Ritzau, L., Liesch, T., & Ohmer, M. (2023) · Frontiers in Water · doi:10.3389/frwa.2023.1193142 · Cited by 29
Online multi-hypergraph fusion learning for cross-subject emotion recognition
Pan, T., Ye, Y., Zhang, Y., Xiao, K., & Cai, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102338 · Cited by 32
Kim, J., Im, J., Shin, W., Lee, S., Oh, S., Kwon, D., Jung, G., Choi, W. Y., & Lee, J. (2024) · Advanced Science · doi:10.1002/advs.202308460 · Cited by 28
MEMF: Multi-entity multimodal fusion framework for sales prediction in live streaming commerce
Xu, G., Ren, M., Wang, Z., & Li, G. (2024) · Decision Support Systems · doi:10.1016/j.dss.2024.114277 · Cited by 38
Mai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26
Collaborative Multi-Robot Systems for Search and Rescue: Coordination and Perception
Queralta, J. P., Taipalmaa, J., Pullinen, B. C., Sarker, V. K., Gia, T. N., Tenhunen, H., Gabbouj, M., Raitoharju, J., & Westerlund, T. (2020) · arXiv · doi:10.48550/arxiv.2008.12610 · Cited by 27
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.