Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities
The DOI matches its Crossref record; 88 of the 98 references it cites were found in a registry or an index.
Abstract
Multimodal emotion recognition (MER) aims to understand the way that humans express their emotions by exploring complementary information across modalities. However, it is hard to guarantee that full-modality data is always available in real-world scenarios. To deal with missing modalities, researchers focused on meaningful joint multimodal representation learning during cross-modal missing modality imagination. However, the cross-modal imagination mechanism is highly susceptible to errors due to the “modality gap” issue, which affects the imagination accuracy, thus, the final recognition performance. To this end, we introduce the concept of a modality-invariant feature into the missing modality imagination network, which contains two key modules: 1) a novel contrastive learning-based module to extract modality-invariant features under full modalities; 2) a robust imagination module based on imagined invariant features to reconstruct missing information under missing conditions. Finally, we incorporate imagined and available modalities for emotion recognition. Experimental results on benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art strategies. Compared with our previous work, our extended version is more effective on multimodal emotion recognition with missing modalities. The code is released athttps://github.com/ZhuoYulang/CIF-MMIN.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
(2008) · Journal of Machine Learning Research · Cited by 33,051
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Research works printed but not linked: 2
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Maas, Rectifier nonlinearities improve neural network acoustic models
Cited by
Xiang, J., Zhu, X., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103809 · Cited by 58
A systematic literature review on incomplete multimodal learning: techniques and challenges
Zhan, Y., Yang, R., You, J., Huang, M., Liu, W., & Liu, X. (2025) · Systems Science & Control Engineering · doi:10.1080/21642583.2025.2467083 · Cited by 73
“In sentiment analysis, multimodal learning can combine text, audio, and video information to assess the user’s emotional state, remarkably improving the accuracy of the analysis (Z. Liu et al., 2024; Y. Sun et al., 2024).”
InfluentialGenerative technology for human emotion recognition: A scoping review
Ma, F., Yuan, Y., Xie, Y., Ren, H., Liu, I., He, Y., Ren, F., Yu, F. R., & Ni, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102753 · Cited by 61
Multimodal fusion in speech emotion recognition: A comprehensive review of methods and technologies
Nguyen, N. M., Nguyen, T. T., Tran, P.-N., Lim, C. P., Pham, N. T., & Dang, D. N. M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112624 · Cited by 18
Multisensory Neuromorphic Devices: From Physics to Integration
Gui, A., Mu, H., Yang, R., Zhang, G., & Lin, S. (2026) · Nano-Micro Letters · doi:10.1007/s40820-025-01940-9 · Cited by 20
Conv-Enhanced Transformer and Robust Optimization Network for robust multimodal sentiment analysis
Sun, B., Jia, L., Cui, Y., Wang, N., & Jiang, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129842 · Cited by 17
Liu, Y., Liu, C., Wen, J., Shen, L., Zhang, B., & Xu, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3665813 · Cited by 40
“For the latter, the core idea is to extract view-consistent information by minimizing the distance between anchors and positive samples in the feature space while maximizing the distance between anchors and negative samples [27].”
Fan, Q., Li, Y., Xin, Y., Cheng, X., Gao, G., & Ma, M. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689412 · Cited by 14
“In the context of the MER2024-SEMI challenge, participants must contend with a discrete set of six emotions: neutral, anger, happiness, sadness, worry, and surprise.”
Li, C., Xie, L., Wang, X., Pan, H., & Wang, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125822 · Cited by 11
A missing multimodal imputation diffusion model for 2D X-ray and 3D CT in COVID-19 diagnosis
Gao, Z., Chen, D., & Shen, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127367 · Cited by 11
Retrieval-Augmented Dialogue Knowledge Aggregation for expressive conversational speech synthesis
Liu, R., Jia, Z., Bao, F., & Li, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102948 · Cited by 20
Self-citationSun, H., Zhao, S., Li, S., Kong, X., Wang, X., Zhou, J., Kong, A., Chen, Y., Zeng, W., & Qin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889485 · Cited by 24
“Many works focus on the missing modality problem, and we select several representative ones for comparison [16], [17], [32]–[34], with the results shown in Table I.”
Listening and seeing again: Generative error correction for audio-visual speech recognition
Liu, R., Yuan, H., Gao, G., & Li, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103077 · Cited by 18
Self-citationWang, Y., Lei, J., Zhu, X., & Zhang, T. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103278 · Cited by 7
Sun, Q., Zuo, H., Liu, R., & Li, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3490694 · Cited by 14
“2) It is helpful to set up two separate text encoders for the Sent-CLAP and Sent-CLIP pre-training stages, rather than the same encoder: As shown in the results of “w/ same text encoder” row of Table III, we find that the overall performance witness a significant drop after replacing the two separate…”
Self-citationRao, Y., Cao, M., & Ye, M. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3568519 · Cited by 9
“Similarly, Liu et al. [52] introduced contrastive learning to extract modality-invariant features under full modality, addressing the challenge that cross-modal imag-ination mechanisms are prone to errors due to modality gaps.”
A cross-modal attention model with contextual enhancements for speech emotion recognition
Qi, R., Zhao, C., Guo, X., Li, Z., Li, S., Chen, H., & Sun, Y. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103285 · Cited by 8
Liu, R., Zuo, H., Lian, Z., Yuan, H., & Fan, Q. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755605 · Cited by 18
“Liu et al. [23] further improved the reconstruction process using modality-invariant features to strengthen model robustness under incomplete inputs.”
InfluentialSelf-citationEnhancing multimodal emotion recognition with dynamic fuzzy membership and attention fusion
Nguyen, N. M., Nguyen, T. M., Nguyen, T. T., Tran, P.-N., Pham, N. T., Le, L., Othmani, A., Saddik, A. E., & Dang, D. N. M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.113396 · Cited by 9
Chen, Y., Hu, C., Kimura, T., Li, Q., Liu, S., Wu, F., & Chen, G. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699779 · Cited by 9
“Overcoming the application boundary: With similar modality constitutions as FOCAL [25] that both use acoustic and seismic signals to detect nearby physical activities, we have made an initial attempt to combine our self-collected Inoffice dataset with the MOD dataset proposed by [25].”
MASKSER: A Robust Emotion Recognition Model Based on Voice Data and Noisy Transcripts
Yoon, Y. C., & Kim, S. K. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3617234 · Cited by 7
“Among them, TDFNet [1] and CIF-MMIN [42] achieved WA/UA scores of 76.3/77.0 and 74.3/75.7, respectively.”
InfluentialLi, Z., Liu, Y., Yang, C., Zhou, Y., & Hu, S. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590929 · Cited by 11
“Translation-basedmeth-ods[1],[2],[3],[38],[39]generatemissingmodalitiesfrom availablemodalities.”
InfluentialAffection-Guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing
Lin, R., He, Q., Zeng, Y., Mai, S., Huang, L., & Hu, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3632063 · Cited by 10
“Alignment-based methods use techniques such as contrastive learning [7], [11], [12], canonical correlation analysis [13], [14], [15], and data augmentation like mixup or noisy input [6], [10], [16] to align the representations of incomplete modalities with the ones of complete multimodal inputs.”
DCCL: Question-guided dual-channel contrastive learning framework for emotion-cause pair extraction
Wang, H., Du, Y.-J., Liu, J., Li, X.-Y., Chen, X.-L., Lee, Y.-L., Qi, Q., & Zhang, W. (2026) · Expert systems with applications · doi:10.1016/j.eswa.2026.131357 · Cited by 2
TTSlow: Slow Down Text-to-Speech With Efficiency Robustness Evaluations
Gao, X., Chen, Y., Yue, X., Tsao, Y., & Chen, N. F. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533357 · Cited by 14
“These developments have led to diverse applications in virtual assistants, audiobooks, emotion understanding, voice-over narration, and navigation systems [34], [35], [36], [37], [38].”
Prompt-based hybrid supervised contrastive learning for emotion recognition in conversation
Cai, C., Zhang, K., Hu, Z., Lin, X., & Pan, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130453 · Cited by 7
Fan, Q., Zuo, H., Liu, R., Lian, Z., & Gao, G. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689411 · Cited by 14
“…a noise schedule across time steps represented by β s . β s = ( β 1 , . . . , β T ) It initializes a sequence of values starting from a small positive number β start (denoting a low noise level) and progressively increasing towards a larger number β end (denoting a high noise level) after T steps.”
BackgroundSelf-citationHierarchical Control of Emotion Rendering in Speech Synthesis
Inoue, S., Zhou, K., Wang, S., & Li, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3582715 · Cited by 9
Self-citationEmotional boundaries and intensity aware model for incomplete multimodal sentiment analysis
Zhang, Y., Xie, D., Luo, D., & Sun, B. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105023 · Cited by 6
Feature Evaluation and Joint Interaction for Audio-Visual Emotion Recognition
Li, S., Lu, C., Zong, Y., Lian, H., & Zheng, W. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3619205 · Cited by 4
“Given that di ff erent modal features simultaneously reflect the emotional information of the same sample, they exhibit a degree of consistency, which renders the features of di ff erent modalities within the same frame time series highly interchangeable [51], [52].”
Expert-level financial sentiment analysis via internal and external financial knowledge fusion
Shi, L., Jia, Z., & Liu, R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103884 · Cited by 3
Self-citationYang, W., Li, H., Wang, Y.-K., & Wang, Y.-Q. (2026) · Expert systems with applications · doi:10.1016/j.eswa.2025.131083 · Cited by 2
Self-citationHypergraph multi-modal learning for EEG-based emotion recognition in conversation
Kang, Z., Li, Y., Gong, S., Zeng, W., Yan, H., Bian, L., Zhang, Z., Siok, W. T., & Wang, N. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109057 · Cited by 16
Enhancing Multimodal Sentiment Analysis with Dynamic Weight Adjustment and Adversarial Training
Zhu, Y., & Liu, X. (2024) · China Automation Congress (CAC) · doi:10.1109/cac63892.2024.10865061 · Cited by 3
Li, C., Xie, L., Wang, X., Pan, H., & Wang, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01758-w · Cited by 2
“The non-attention-based fusion methods include TFN [14], MISA [30], IF-MMIN [26], and CIF-MMIN [25].”
Modality emotion semantic correlation analysis for multimodal emotion recognition
Zhang, Y., Xie, D., Luo, D., & Sun, B. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110467 · Cited by 4
Xu, G.-X., Zuo, X., & Li, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04984-1_33 · Cited by 2
Zhu, Q., Mao, Q., Dong, W., Shao, X., Huang, X., & Zheng, W. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3618148 · Cited by 7
Asymmetric simulation-enhanced flow reconstruction for incomplete multimodal learning
Yao, J., Zhang, J., Wang, Y., & Li, Z. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112413 · Cited by 3
External Financial Knowledge Fusion for Accurate Financial Sentiment Analysis
Shi, L., Jia, Z., & Liu, R. (2025) · International Conference on Asian Language Processing · doi:10.1109/ialp68296.2024.11156561 · Cited by 1
“The predicted sentiment ( S p ) from the classifier is then compared with the ground truth sentiment ( S gt ), and CE Loss [20], [21] is used to optimize the model.”
Self-citationSun, H., Lu, X., Tian, J., Zhou, J., He, J., Wang, H., Kong, X., Hu, X., & Qin, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3640882 · Cited by 4
“CIF-MMIN [47] is an improved version of IF-MMIN.”
Emphasis rendering for conversational text-to-speech with multi-modal multi-scale context modeling
Liu, R., Zhenqi, J., Yang, J., Hu, Y., & Li, H. (2026) · Speech Communication · doi:10.1016/j.specom.2026.103353 · Cited by 3
Self-citationYongwei, L. I., Gao, W., & Li, J. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113224 · Cited by 1
Yu, Y., Fang-ai, L., Zhang, Y., Gao, X., & Yang, H. (2026) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105880 · Cited by 2
Chien, G., Liu, J., Wang, J., & Chen, L. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115654 · Cited by 7
Liu, Y., Shou, Y., Ai, W., Meng, T., & Li, K. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133306 · Cited by 2
“As the core carrier of emotional seman tics, the textual modality can serve as a guiding modality in the fusion process.”
ALMansour, A. G. M., & Alshomrani, F. (2026) · Scientific Reports · doi:10.1038/s41598-026-48187-x · Cited by 3
“One of these imputation techniques uses generative adversarial networks (GAN) and variational autoencoders (VAE) to generate the missing modalities before segmentation 11–14 .”
Lv, W., Hu, H., & Kang, Y. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133655 · Cited by 1
Luo, J., Miao, L., Guan, Q., et al. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461625 · Cited by 1
EMPATHIA Med: A Framework for Emotion Conditioned Multimodal Medical Intelligence
Aarthi, D., Ismoilov, M., Nivedha, R., Jeevitha, R., qizi, A. Z. F., & Priya, M. (2026) · International Conference on Machine Learning and Autonomous Systems (ICMLAS) · doi:10.1109/icmlas67792.2026.11483808 · Cited by 1
“In order to solve these issues, contrastive learning-based methods have been suggested to learn modality-invariant features, which enhance resilience in multimodal emotion recognition in the context of an incomplete set of data [8].”
Pre-training Language Model for Mongolian with Agglutinative Linguistic Knowledge Injection
Na, M., Liu, R., Bao, F., & Gao, G. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650812 · Cited by 1
Self-citationCFin-NER: A Study on Chinese Financial Entity Recognition Based on Comparative Learning
Zhao, J., Qian, Y., Hou, S., Chen, J., & Wang, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6963-9_31 · Cited by 1
Yuan, M., Li, J., Zhao, Y., Wang, J., Ye, J., & Jia, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0030-7_22 · Cited by 1
Zhao, Y. (2025) · Applied and Computational Engineering · doi:10.54254/2755-2721/2025.po24716
TranSTYLer: Multimodal behavioural style transfer for facial and body gestures generation
Fares, M., Pélachaud, C., & Obin, N. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103286 · Cited by 3
Zhou, Q., Song, J., Zhao, Y., Zhang, S., Du, Q., & Li, K. (2025) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-025-03430-x
“In order to address that problem, contrastive learning has been implemented in emotion recognition [23–26].”
Kaur, D., Misra, A., & Vyas, O. P. (2025) · doi:10.1109/icscsa66339.2025.11170723
“[2] have developed a framework that learned features that are unaffected by different input types, using a missing modality imagination network.”
Liu, P., Sun, F., Zhao, L., Fang, W., & Huang, F. (2025) · The Computer Journal · doi:10.1093/comjnl/bxaf110
Wang, D., Li, Z., Xia, Y., & Yu, Z. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.071656
Tang, X., Xiang, Y., Yu, Z., & Huang, Y. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209332
“Zuo et al. [10] introduced modality-invariant features, further improved by Liu et al. [11] with contrastive learning.”
InfluentialFan, J., Chen, W., Fan, Z., & Yu, R. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209913 · Cited by 2
Optimization Strategy of Multimodal Emotional Interaction System Based on GPT-5
Zhang, Q. (2025) · doi:10.1109/iacis65746.2025.11211043
“Liu et al. [9] introduced the concept of modality-invariant features in the missing modality imagination network, which consists of two key modules: 1) a novel module based on contrastive learning to extract modality-invariant features under all modalities; 2) a robust imagination module based on…”
MDA-ML:A Multi-source Domain Adaptation Network with Multi-view Learning for Emotion Recognition
Wang, X., Song, J., Zhao, Y., et al. (2025) · doi:10.1109/ijcnn64981.2025.11227591
“[ 11].”
MDA-ML:A Multi-source Domain Adaptation Network with Multi-view Learning for Emotion Recognition
Wang, X., Song, J., Zhao, Y., et al. (2025) · doi:10.1109/ijcnn64981.2025.11228709
Wang, Y., & Zhou, Y. (2025) · doi:10.1145/3778450.3778513
“[9] developed a contrastive learning module to extract modality-invariant features, enabling robust emotion recognition even with incomplete inputs.”
Temporal Multimodal Knowledge Distillation for Modality-Missing RGBT TrackingPreprint
Ruan, R., Kang, Y., Liu, L. N., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5953538
Fu, J., Li, Z., Weng, H., Li, C., Luo, Y., Li, X., & Chen, C. (2025) · International Conference on Electronic Information, Computer and Aerospace Remote Sensing (EICARS) · doi:10.1109/eicars68214.2025.11320234
“As a result, EEG data are often unavailable in practical applications, leading to severe performance degradation of traditional multimodal methods and posing a major challenge for real-world deployment[12], [13].”
Comprehensive Insights Into Multimodal Emotion Recognition Using Machine Learning and Deep Learning
Sanku, R., Singireddy, S., Nandini, M. R., et al. (2025) · doi:10.1109/ic3it66137.2025.11340782
“Liu R, et al. [10] developed a modality-invariant feature-based approach to multimodal emotion recognition with absent modalities.”
Nguyen, D., & Le‐Khac, N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6216286
SADGR: Adaptive Cross-Modal Emotion Recognition via Self-Supervised Alignment and Dynamic Gating
Zhang, J., Mao, J., Hou, Y., Noh, G., & Zhang, F. (2026) · IEEE Access · doi:10.1109/access.2026.3663574 · Cited by 1
“[41] shows that contrastive learning enables robust modality-invariant representations for multimodal emotion recognition.”
Ren, J., Chen, X., Cai, X., Guo, K., Chen, L., & Wei, J. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3668129
“Recent works employ contrastive learning [19], [20], [21], [22], [23], [24], [25] to enhance alignment or improve the discriminability between emotions.”
DGGD: A Unified and Discriminative Representation Learning Method for Multimodal Sentiment Analysis
Ye, L., Zheng, N., Li, Y., & Luo, N. (2025) · International Conference on Computer, Internet of Things and Smart City (CIoTSC) · doi:10.1109/ciotsc67482.2025.11412996
“Liu et al. [4] leveraged contrastive learning and a modal imagery module to achieve modal decoupling and reconstruction, demonstrating strong robustness under missing modality scenarios.”
Deep reinforcement learning for adaptive music emotion recognition and generation
Zang, H., & Chen, Z. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-00968-z
Dynamic Memory-Augmented Retrieval and Synthesis for Robust Incomplete Multimodal LearningPreprint
Chen, C., Wang, Z., Zhang, H., Sun, J., Ma, J., Chen, B., & Zhang, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6388116
A Multimodel Deep Learning Framework for Emotion Aware Mental Health Assessment
Pranav Chandran, T., Prabhavathy, P., Kanipriya, M., et al. (2025) · International Conference on Signal Processing, Computation, Electronics, Power and Telecommunication (IConSCEPT) · doi:10.1109/iconscept66142.2025.11437254
“Liu et al. [9] even extended this by using contrastive learning for modality-invariant feature learning, to enhance its robustness towards missing or noisy input conditions which is a major drawback in real world multimodal settings.”
Zhao, H., Yu, Z., Wei, Y., Li, B., & Gao, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462640 · Cited by 1
Ye, M., Yin, M., & Guo, Y. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3687649 · Cited by 1
“Using autoencoders and contrastive learning, a cross-modal transfer network was proposed to recover the missing data [17].”
Shuai, Q., Chen, X., Hu, F., & Cheng, Y. (2026) · Complex & Intelligent Systems · doi:10.1007/s40747-026-02318-z
“Referring to previous works on contrastive learning [40], we construct the following loss function for contrastive learning on X and Y pairs: where ( X , Y ) ∈ (( ˆ H a , ˆ H t ), ( ˆ H v , ˆ H t ), ( ˆ H a , ˆ H v )) , and X is the anchor point, X i , Y i denote the positive pair, the X i , Y j…”
Cross-Attention Audio–Visual Fusion Based on Multi-Scale Vision Transformers for Emotion Recognition
Bao, C., Chen, L., Li, M., Wu, M., Witold Pedrycz, C., & Hirota, K. (2026) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2026.p0749 · Cited by 1
Temporal multimodal knowledge distillation for modality-missing RGBT tracking
Ruan, R., Kang, Y., Liu, L., Sun, J., Li, C., & Tang, J. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113991 · Cited by 1
Shou, Y., Meng, T., Ai, W., & Li, K. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810726
“For MERC under incomplete multimodality, a common solu-tion is to use an encoder–decoder framework to recover missing modalities from observed ones.”
GS³-ICL: Graph-Structured Sparse Selection with Invariance-Consistent Learning for multimodal SER
Zhou, Y., Ma, Z., Wang, L. S., Ren, H., Zhong, Z., Wang, S., Zhang, C., & Yang, H. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810766
“Driven by the need to compensate for such signal degradation and ensure robustness amidst real-world imperfections, recent research has focused on robustness against coupled uncertainty involving both data noise and label subjectivity [26, 27, 49].”
Personalized Federated Multi-Center Medical Data Analysis with Local and Global Uncertainty
Li, S., Wang, M. Y., Xu, S., Zhang, D., & Zhu, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-3827-0_19
Fan, J., Chen, W., & M, X. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134334
Align and Adapt: Multimodal Multiview Human Activity Recognition Under Arbitrary View Combinations
Nguyen, D., & Le‐Khac, N. (2026) · IEEE Sensors Journal · doi:10.1109/jsen.2026.3718255
“Digital Object Identifier 10.1109/JSEN.2026.3718255 combination can degrade the task performance when the available views do not align with the system’s original design.”
InfluentialMulti-modal CNN Based Emotion Recognition with Prior Pose Estimation Using YOLOv5
Khan, S. S., Sengupta, D., Ghosh, A., & Chaudhuri, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-25446-7_37
-, B. P., Deshpande, U. U., & Michael, G. K. O. (2026) · Discover Computing · doi:10.1007/s10791-026-10471-9
Bastidas-Benalcazar, N., Santos-Solís, D., Cueva-Argudo, D., Lechón-Santacruz, E., Almeida-Galárraga, D. A., Navas, P., Mora, H. C., Orozco-Garzón, N., & Tirado-Espín, A. (2026) · Technologies · doi:10.3390/technologies14090523
Feature Disentanglement for Multimodal Emotion Recognition with Missing Modalities
Zhang, J., Tao, X., & Gan, Y. (2026) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat71030.2026.11682632
“Following the approach of Liu et al. [14], the contrastive learning loss is defined as: where ( X, Y ) ∈ ( H s , H e ) , ( H s , H p ) , ( H e , H p ) , X denotes the anchor modality, ( X i , Y i ) represents a positive pair, ( X i , Y j ) denotes a negative pair, N is batch size, and sim( · )…”
Emotion Emerges from Shared Latent Context
Yang, Z., Pei, G., & Fu, X. (2026) · International Conference on Image, Vision and Computing (ICIVC) · doi:10.1109/icivc70586.2026.11686507
Wang, S., Li, J., Wu, Z., Wang, B., & Wang, Q. (2026) · iScience · doi:10.1016/j.isci.2026.117593
Li, J., Gong, J., Chen, J., Chen, R., Wang, Y., Fu, Y., & Zhao, Y. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.117118
Zhou, C., & Kang, W. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1927988
Metadata
- Authors
- Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H.
- Affiliations
- Inner Mongolia University (CN)
- University of Bremen (DE)
- Chinese Academy of Sciences (CN)
- Imperial College London (GB)
- Shenzhen Research Institute of Big Data (CN)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2024-10
- Volume
- 15
- Issue
- 4
- Pages
- 1856-1873
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W4392903647
- Cited by
- 96
- References
- 98
- Citations by year
- 2026: 40 · 2025: 48 · 2024: 8
- FWCI
- 39.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion)
- Funders
- National Natural Science Foundation of China (62206136, 62201572)
- Guangdong Provincial Key Laboratory of Human Digital Twin (2022B121201 0004)
- Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone
- University Collaborative Innovation Base
- University Entrepreneurship Training Base (21300-231510)
- Internal Project Fund from Shenzhen Research Institute of Big Data (T00120220002)
- Shenzhen Science and Technology Research Fund Fundamental Research Key Project (JCYJ20220818103001002)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal emotion recognition, missing modalities, modality gap, modality-invariant features, contrastive learning, cross-modal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.