Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities
The DOI matches its Crossref record; 88 of the 98 references it cites were found in a registry or an index.
Abstract
Multimodal emotion recognition (MER) aims to understand the way that humans express their emotions by exploring complementary information across modalities. However, it is hard to guarantee that full-modality data is always available in real-world scenarios. To deal with missing modalities, researchers focused on meaningful joint multimodal representation learning during cross-modal missing modality imagination. However, the cross-modal imagination mechanism is highly susceptible to errors due to the “modality gap” issue, which affects the imagination accuracy, thus, the final recognition performance. To this end, we introduce the concept of a modality-invariant feature into the missing modality imagination network, which contains two key modules: 1) a novel contrastive learning-based module to extract modality-invariant features under full modalities; 2) a robust imagination module based on imagined invariant features to reconstruct missing information under missing conditions. Finally, we incorporate imagined and available modalities for emotion recognition. Experimental results on benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art strategies. Compared with our previous work, our extended version is more effective on multimodal emotion recognition with missing modalities. The code is released athttps://github.com/ZhuoYulang/CIF-MMIN.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
(2008) · Journal of Machine Learning Research · Cited by 33,051
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Self-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,941
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073 · cited 6×
“In full-modality scenarios, some works tried to mine the intrinsic correlation of different modalities to alleviate the “ modality gap ” problem [16], [17], [18].”
InfluentialContext-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation LearningPreprint
Liang, W., Zhang, Y., Kwon, Y., Yeung, S., & Zou, J. Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2203.02053 · Cited by 908
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852
Sak, H., Senior, A., & Beaufays, F. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.1128 · Cited by 838
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695
Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512
Neural Networks and Deep Learning
(2016) · Computer Age Statistical Inference · doi:10.1017/cbo9781316576533.019 · Cited by 509
MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391
Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
Wu, N., Green, B., Ben, X., & O’Banion, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08317 · Cited by 335
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S., Cui, Y., & Gong, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11178 · Cited by 326
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities
Zhao, J., Li, R., & Jin, Q. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.203 · Cited by 275 · cited 10×
“Furthermore, we adopt the partition method of [7], i.e., four sessions as the training set, and the remaining session is divided into the validation and testing sets according to the speaker’s gender.”
InfluentialDeep Partial Multi-View Learning
Zhang, C., Cui, Y., Han, Z., Zhou, J. T., Fu, H., & Hu, Q. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3037734 · Cited by 270
Sun, L., Lian, Z., Liu, B., & Tao, J. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 258
Self-citationMissing Modalities Imputation via Cascaded Residual Autoencoder
Tran, L., Liu, X., Zhou, J., & Jin, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.528 · Cited by 256 · cited 4×
“To achieve this, several neural network models, which include a Transformer [9], [39], Generative Adversarial Network (GAN) [40], [41], AutoEncoder (AE) [42], [43], and a Variational AutoEncoder (VAE) [44], [45] amongst others have emerged and performed promising results.”
InfluentialVideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples
Pan, T., Song, Y., Yang, T., Jiang, W., & Liu, W. (2021) · arXiv · doi:10.48550/arxiv.2103.05905 · Cited by 238
VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples
Pan, T., Song, Y., Yang, T., Jiang, W., & Liu, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01105 · Cited by 231
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
Li, W., Gao, C., Niu, G., Xiao, X., Liu, H., Liu, J., Wu, H., & Wang, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.202 · Cited by 231
Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis
Yuan, Z., Li, W., Xu, H., & Yu, W. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475585 · Cited by 211
Domain Adaptation via Prompt Learning
Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., & Huang, G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3327962 · Cited by 199
Supporting Clustering with Contrastive Learning
Zhang, D., Nan, F., Wei, X., Li, S.-W., Zhu, H., McKeown, K., Nallapati, R., Arnold, A. O., & Xiang, B. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.427 · Cited by 185
Deep Adversarial Learning for Multi-Modality Missing Data Completion
Cai, L., Wang, Z., Gao, H., Shen, D., & Ji, S. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219963 · Cited by 180
ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis
Yang, J., Yu, Y., Niu, D., Guo, W., & Xu, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.421 · Cited by 173
Learning Alignment for Multimodal Emotion Recognition from Speech
Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., & Li, X. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3247 · Cited by 170
GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
Lian, Z., Chen, L., Sun, L., Liu, B., & Tao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234553 · Cited by 160
Self-citationPartial Multi-view Clustering via Consistent GAN
Wang, Q., Ding, Z., Tao, Z., Gao, Q., & Fu, Y. (2018) · 2018 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2018.00174 · Cited by 151
CM-BERT: Cross-Modal BERT for Text-Audio Sentiment Analysis
Yang, K., Xu, H., & Gao, K. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413690 · Cited by 129
Key-Sparse Transformer for Multimodal Speech Emotion Recognition
Chen, W., Xing, X., Xu, X., Yang, J., & Pang, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746598 · Cited by 112
Variational Autoencoder with Arbitrary ConditioningPreprint
Ivanov, O., Figurnov, M., & Vetrov, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02382 · Cited by 107
Learning in Audio-visual Context: A Review, Analysis, and New PerspectivePreprint
Wei, Y., Hu, D., Tian, Y., & Li, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.09579 · Cited by 106
Zuo, H., Liu, R., Zhao, J., Gao, G., & Li, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095836 · Cited by 105
Self-citationLearning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q., Salakhutdinov, R., & Morency, L.-P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1152 · Cited by 95 · cited 1×
“Some further approaches [38] take into account the correlation between different modalities.”
Tang, J., Li, K., Jin, X., Cichocki, A. S., Zhao, Q., & Kong, W. (2021) · doi:10.18653/v1/2021.acl-long.412 · Cited by 92
A deep learning based approach for traffic data imputation
Duan, Y., Lv, Y., Kang, W., & Zhao, Y. (2014) · 17th International IEEE Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2014.6957805 · Cited by 90
Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification
Lee, S., Han, D. K., & Ko, H. (2021) · IEEE Access · doi:10.1109/access.2021.3092735 · Cited by 89
Padi, S., Sadjadi, S. O., Manocha, D., & Sriram, R. D. (2022) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2022-57 · Cited by 88
NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better
Wu, C., Wu, F., Qi, T., & Huang, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.76 · Cited by 80
Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition
Parthasarathy, S., & Sundaram, S. (2020) · Companion Publication of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3395035.3425202 · Cited by 78
Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
Du, C., Du, C., Wang, H., Li, J., Zheng, W.-L., Lu, B.-L., & He, H. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240528 · Cited by 74
Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
Liang, J., Li, R., & Jin, Q. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413579 · Cited by 73
MEmoR: A Dataset for Multimodal Emotion Reasoning in Videos
Shen, G., Wang, X. Y., Duan, X., Li, H., & Zhu, W. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413909 · Cited by 73
Self-citationUnderstanding Hard Negatives in Noise Contrastive Estimation
Zhang, W., & Stratos, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.86 · Cited by 73
Cross-Modal Discrete Representation Learning
Liu, A. H., Jin, S., Lai, C.-I., Rouditchenko, A., Oliva, A., & Glass, J. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.215 · Cited by 68
Tellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 66
Self-citationMake Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64 · cited 2×
“One of the possible reasons may be that the textual modality includes more semantic information than the others do [82], [83], resulting in a much weaker representation of features when the textual modality is missing.”
InfluentialTDFNet: Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition
Zhao, Z., Wang, Y., shen, G. Z., Xu, Y., & Zhang, J. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3316458 · Cited by 63
Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability
Liu, R., Şişman, B., & Li, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1236 · Cited by 61
Self-citationGoncalves, L., & Busso, C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3216993 · Cited by 61
Gu, D., Wang, J., Cai, S., Yang, C., Song, Z., Zhao, H., Xiao, L., & Wang, H. (2021) · IEEE Access · doi:10.1109/access.2021.3126782 · Cited by 60
Implicit Fusion by Joint Audiovisual Training for Emotion Recognition in Mono Modality
Han, J., Zhang, Z., Ren, Z. X., & Schuller, B. W. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682773 · Cited by 54
Self-citationLiu, R., Hu, Y., Ren, Y., Yin, X., & Li, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29833 · Cited by 49
Self-citationMetric Learning on Healthcare Data with Incomplete Modalities
Suo, Q., Zhong, W., Ma, F., Yuan, Y., Gao, J., & Zhang, A. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/490 · Cited by 46
Multimodal Analogical Reasoning over Knowledge GraphsPreprint
Zhang, N., Li, L., Chen, X., Liang, X., Deng, S., & Chen, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.00312 · Cited by 46
Cross-modal Retrieval and Synthesis (X-MRS): Closing the Modality Gap in Shared Subspace Learning
Guerrero, R., Pham, H., & Pavlović, V. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475465 · Cited by 45
NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis
Fu, Z., Liu, F., Xu, Q., Qi, J., Fu, X., Zhou, A., & Li, Z. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859836 · Cited by 44
Liu, R., Hu, Y., Zuo, H., Luo, Z., Wang, L., & Gao, G. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3348762 · Cited by 43
Self-citationSelf-Supervised Learning with Cross-Modal Transformers for Emotion Recognition
Khare, A., Parthasarathy, S., & Sundaram, S. (2021) · 2021 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383618 · Cited by 41
Pitch-synchronous single frequency filtering spectrogram for speech emotion recognition
Gupta, S., Fahad, M. S., & Deepak, A. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09068-1 · Cited by 39
Understanding Multimodal Contrastive Learning and Incorporating Unpaired DataPreprint
Nakada, R., Gulluk, H. I., Deng, Z., Ji, W., Zou, J., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.06232 · Cited by 31
2D vs. 3D LiDAR-based Person Detection on Mobile Robots
Jia, D. P., Hermans, A., & Leibe, B. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981519 · Cited by 27
Accuracy weighted diversity-based online boosting
Baidari, I., & Honnikoll, N. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113723 · Cited by 19
Leveraging Contrastive Learning and Knowledge Distillation for Incomplete Modality Rumor Detection
Xu, F., Fu, P., Huang, Q., Zou, B., Aw, A. T., & Wang, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.900 · Cited by 13
Unsupervised Metric Learning with Synthetic Examples
Dutta, U. K., Harandi, M. T., & Sekhar, C. C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5795 · Cited by 11
Domain and Modality Gaps for LiDAR-based Person Detection on Mobile Robots.Preprint
Jia, D. P., Hermans, A., & Leibe, B. (2021) · arXiv (Cornell University) · Cited by 9
Topic and Style-aware Transformer for Multimodal Emotion Recognition
Qiu, S., Sekhar, N., & Singhal, P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.130 · Cited by 8
Mongolian emotional speech synthesis based on transfer learning and emotional embedding
Huang, A., Bao, F., Gao, G., Yu, S., & Liu, R. (2021) · International Conference on Asian Language Processing · doi:10.1109/ialp54817.2021.9675192 · Cited by 7
Self-citationAccurate Emotion Strength Assessment for Seen and Unseen Speech Based on Data-Driven Deep Learning
Liu, R., Şişman, B., Schuller, B. W., et al. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-534 · Cited by 7
Self-citationSammut, C., & Webb, G. I. (2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_953 · Cited by 5
CMI-Net: a Unified Framework for Physiological Time Series Classification with Incomplete ModalitiesPreprint
Shen, Q. (2023) · doi:10.36227/techrxiv.24099339.v1 · Cited by 1
Research works printed but not linked: 2
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Maas, Rectifier nonlinearities improve neural network acoustic models
Cited by
Xiang, J., Zhu, X., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103809 · Cited by 58
A systematic literature review on incomplete multimodal learning: techniques and challenges
Zhan, Y., Yang, R., You, J., Huang, M., Liu, W., & Liu, X. (2025) · Systems Science & Control Engineering · doi:10.1080/21642583.2025.2467083 · Cited by 72
“In sentiment analysis, multimodal learning can combine text, audio, and video information to assess the user’s emotional state, remarkably improving the accuracy of the analysis (Z. Liu et al., 2024; Y. Sun et al., 2024).”
InfluentialGenerative technology for human emotion recognition: A scoping review
Ma, F., Yuan, Y., Xie, Y., Ren, H., Liu, I., He, Y., Ren, F., Yu, F. R., & Ni, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102753 · Cited by 61
Multimodal fusion in speech emotion recognition: A comprehensive review of methods and technologies
Nguyen, N. M., Nguyen, T. T., Tran, P.-N., Lim, C. P., Pham, N. T., & Dang, D. N. M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112624 · Cited by 18
Multisensory Neuromorphic Devices: From Physics to Integration
Gui, A., Mu, H., Yang, R., Zhang, G., & Lin, S. (2026) · Nano-Micro Letters · doi:10.1007/s40820-025-01940-9 · Cited by 20
Metadata
- Authors
- Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H.
- Affiliations
- Inner Mongolia University (CN)
- University of Bremen (DE)
- Chinese Academy of Sciences (CN)
- Imperial College London (GB)
- Shenzhen Research Institute of Big Data (CN)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2024-10
- Volume
- 15
- Issue
- 4
- Pages
- 1856-1873
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W4392903647
- Cited by
- 96
- References
- 98
- Citations by year
- 2026: 40 · 2025: 48 · 2024: 8
- FWCI
- 39.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion)
- Funders
- National Natural Science Foundation of China (62206136, 62201572)
- Guangdong Provincial Key Laboratory of Human Digital Twin (2022B121201 0004)
- Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone
- University Collaborative Innovation Base
- University Entrepreneurship Training Base (21300-231510)
- Internal Project Fund from Shenzhen Research Institute of Big Data (T00120220002)
- Shenzhen Science and Technology Research Fund Fundamental Research Key Project (JCYJ20220818103001002)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal emotion recognition, missing modalities, modality gap, modality-invariant features, contrastive learning, cross-modal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.