The DOI matches its Crossref record; 87 of the 98 references it cites were found in a registry or an index.
Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities
Abstract
Multimodal emotion recognition (MER) aims to understand the way that humans express their emotions by exploring complementary information across modalities. However, it is hard to guarantee that full-modality data is always available in real-world scenarios. To deal with missing modalities, researchers focused on meaningful joint multimodal representation learning during cross-modal missing modality imagination. However, the cross-modal imagination mechanism is highly susceptible to errors due to the “modality gap” issue, which affects the imagination accuracy, thus, the final recognition performance. To this end, we introduce the concept of a modality-invariant feature into the missing modality imagination network, which contains two key modules: 1) a novel contrastive learning-based module to extract modality-invariant features under full modalities; 2) a robust imagination module based on imagined invariant features to reconstruct missing information under missing conditions. Finally, we incorporate imagined and available modalities for emotion recognition. Experimental results on benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art strategies. Compared with our previous work, our extended version is more effective on multimodal emotion recognition with missing modalities. The code is released athttps://github.com/ZhuoYulang/CIF-MMIN.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
(2008) · Journal of Machine Learning Research · Cited by 33,051
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Research works printed but not linked: 2
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Maas, Rectifier nonlinearities improve neural network acoustic models
Cited by
Xiang, J., Zhu, X., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103809 · Cited by 58
A systematic literature review on incomplete multimodal learning: techniques and challenges
Zhan, Y., Yang, R., You, J., Huang, M., Liu, W., & Liu, X. (2025) · Systems Science & Control Engineering · doi:10.1080/21642583.2025.2467083 · Cited by 73
“In sentiment analysis, multimodal learning can combine text, audio, and video information to assess the user’s emotional state, remarkably improving the accuracy of the analysis (Z. Liu et al., 2024; Y. Sun et al., 2024).”
InfluentialGenerative technology for human emotion recognition: A scoping review
Ma, F., Yuan, Y., Xie, Y., Ren, H., Liu, I., He, Y., Ren, F., Yu, F. R., & Ni, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102753 · Cited by 59
Multimodal fusion in speech emotion recognition: A comprehensive review of methods and technologies
Nguyen, N. M., Nguyen, T. T., Tran, P.-N., Lim, C. P., Pham, N. T., & Dang, D. N. M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112624 · Cited by 15
Multisensory Neuromorphic Devices: From Physics to Integration
Gui, A., Mu, H., Yang, R., Zhang, G., & Lin, S. (2026) · Nano-Micro Letters · doi:10.1007/s40820-025-01940-9 · Cited by 20
Metadata
- Authors
- Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H.
- Affiliations
- Inner Mongolia University (CN)
- University of Bremen (DE)
- Chinese Academy of Sciences (CN)
- Imperial College London (GB)
- Shenzhen Research Institute of Big Data (CN)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2024-10
- Volume
- 15
- Issue
- 4
- Pages
- 1856-1873
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W4392903647
- Cited by
- 96
- References
- 98
- Citations by year
- 2026: 40 · 2025: 48 · 2024: 8
- FWCI
- 39.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion)
- Funders
- National Natural Science Foundation of China (62206136, 62201572)
- Guangdong Provincial Key Laboratory of Human Digital Twin (2022B121201 0004)
- Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone
- University Collaborative Innovation Base
- University Entrepreneurship Training Base (21300-231510)
- Internal Project Fund from Shenzhen Research Institute of Big Data (T00120220002)
- Shenzhen Science and Technology Research Fund Fundamental Research Key Project (JCYJ20220818103001002)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal emotion recognition, missing modalities, modality gap, modality-invariant features, contrastive learning, cross-modal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities. https://app.cochat.ai/papers/10.1109/taffc.2024.3378570
- The paper
- Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H. (2024). Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities. IEEE Transactions on Affective Computing. https://doi.org/10.1109/taffc.2024.3378570
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.