Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model
Open accessAbstract
With the proliferation of user-generated videos in online websites, it becomes particularly important to achieve automatic perception and understanding of human emotion/sentiment from these videos. In this paper, we present our solutions to the MuSe-Wilder and MuSe-Sent sub-challenges in MuSe 2021 Multimodal Sentiment Analysis Challenge. MuSe-Wilder focuses on continuous emotion (i.e., arousal and valence) recognition while the task of MuSe-Sent concentrates on discrete sentiment classification. To this end, we first extract a variety of features from three common modalities (i.e., audio, visual, and text), including both low-level handcrafted features and high-level deep representations from supervised/unsupervised pre-trained models. Then, the long short-term memory recurrent neural network, as well as the self-attention mechanism is employed to model the complex temporal dependencies in the feature sequence. The concordance correlation coefficient (CCC) loss and F1-loss are used to guide continuous regression and discrete classification, respectively. To further boost the model's performance, we adopt late fusion to exploit complementary information from different modalities. Our proposed method achieves CCCs of 0.4117 and 0.6649 for arousal and valence respectively on the test set of MuSe-Wilder, which outperforms the baseline system (i.e., 0.3386 and 0.5974) by a large margin. For MuSe-Sent, F1-scores of 0.3614 and 0.4451 for arousal and valence are obtained, which also outperforms the baseline system significantly (i.e., 0.3512 and 0.3291). With these promising results, we ranked top3 in both sub-challenges.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sun et al. employed LSTM and a self-attention mechanism to capture complex temporal dependencies in feature sequences and fused the prediction results of several unimodal affect recognition models using a second-level LSTM as the fusion model.
“Recently, Sun et al. [88,89] employed LSTM and a self-attention mechanism to capture complex temporal dependencies in feature sequences. They then fused the prediction results of several unimodal affect recognition models using a second-level LSTM as the fusion model.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [89]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,605 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · see citing works
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338 · see citing works
Openface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861 · see citing works
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · see citing works
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., & Plumbley, M. D. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3030497 · Cited by 1,235 · see citing works
DeBERTa: Decoding-enhanced BERT with Disentangled AttentionPreprint
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 1,224 · see citing works
Training deep networks for facial expression recognition with crowd-sourced label distribution
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993165 · Cited by 926 · see citing works
Schuller, B. W., Steidl, S., Batliner, A. M., Vinciarelli, A., Scherer, K. R., Ringeval, F., Chétouani, M., Weninger, F. J., Eyben, F., Marchi, E., et al. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-56 · Cited by 732 · see citing works
pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis
Γιαννακόπουλος, Θ. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144610 · Cited by 460 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020) · neural information processing systems · Cited by 386 · see citing works
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
Albanie, S., Nagrani, A., Vedaldi, A., & Zisserman, A. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240578 · Cited by 293 · see citing works
Estimation of continuous valence and arousal levels from faces in naturalistic conditions
Toisoul, A., Kossaifi, J., Bulat, A., Tzimiropoulos, G., & Pantić, M. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00280-0 · Cited by 233 · see citing works
The MuSe 2021 Multimodal Sentiment Analysis Challenge
Stappen, L., Baird, A., Christ, L., Schumann, L., Sertolli, B., Meßner, E.-M., Cambria, E., Zhao, G., & Schuller, B. W. (2021) · doi:10.1145/3475957.3484450 · Cited by 96 · see citing works
Stappen, L., Baird, A., Schumann, L., & Schuller, B. W. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3097002 · Cited by 92 · see citing works
Affective Computing for Large-scale Heterogeneous Multimedia Data
Zhao, S., Wang, S., Soleymani, M., Joshi, D., & Ji, Q. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3363560 · Cited by 87 · see citing works
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 1
- Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). · arXiv
Cited by
Sorted by citations
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224 · see citing works
“Recently, Sun et al. [88,89] employed LSTM and a self-attention mechanism to capture complex temporal dependencies in feature sequences.”
Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning
Luna-Jiménez, C., Griol, D., Callejas, Z., Kleinlein, R., Montero, J. M., & Fernández-Martínez, F. (2021) · Sensors · doi:10.3390/s21227665 · Cited by 168 · see citing works
Luna-Jiménez, C., Kleinlein, R., Griol, D., Callejas, Z., Montero, J. M., & Fernández-Martínez, F. (2021) · Applied Sciences · doi:10.3390/app12010327 · Cited by 125 · see citing works
Sun, L., Lian, Z., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102382 · Cited by 78 · see citing works
Self-citationLi, M., Shi, J., Bai, L., Huang, C., Jiang, Y., Lü, K., Wang, S., & Hancock, E. R. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2024.111340 · Cited by 69 · see citing works
Khan, U. A., Xu, Q., Liu, Y. M., Lagstedt, A., Alamäki, A. V., & Kauttonen, J. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01302-2 · Cited by 62 · see citing works
Lin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 56 · see citing works
Boitel, E., Mohasseb, A., & Haig, E. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126236 · Cited by 52 · see citing works
Multimodal Temporal Attention in Sentiment Analysis
He, Y. L., Sun, L., Lian, Z., Liu, B., Tao, J., Wang, M., & Cheng, Y. (2022) · doi:10.1145/3551876.3554811 · Cited by 30 · see citing works
Self-citationTime-Continuous Audiovisual Fusion with Recurrence vs Attention for In-The-Wild Affect Recognition
Karas, V., Tellamekala, M. K., Mallol-Ragolta, A., Valstar, M., & Schuller, B. W. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00266 · Cited by 25 · see citing works
Vaiani, L., Quatra, M. L., Cagliero, L., & Garza, P. (2022) · doi:10.1145/3551876.3554806 · Cited by 17 · see citing works
Affective Video Content Analysis: Decade Review and New Perspectives
Xue, J., Wang, J., Liu, X., Zhang, Q., & Wu, X. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020048 · Cited by 16 · see citing works
Emotion Recognition from Videos Using Multimodal Large Language Models
Vaiani, L., Cagliero, L., & Garza, P. (2024) · Future Internet · doi:10.3390/fi16070247 · Cited by 15 · see citing works
Integrating Cross-modal Interactions via Latent Representation Shift for Multi-modal Humor Detection
Chen, C., & Zhang, P. (2022) · doi:10.1145/3551876.3554805 · Cited by 11 · see citing works
Exploring Multimodal Sentiment Analysis Models: A Comprehensive Survey
Quoc, P. D., Nguyen-Tat, T. B., Roantree, M., & Ngo, V. M. (2024) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr63514.2024.10660747 · Cited by 10 · see citing works
Improving Dimensional Emotion Recognition via Feature-wise Fusion
Liu, Y., Sun, W., Zhang, X., & Qin, Y. (2022) · doi:10.1145/3551876.3554804 · Cited by 9 · see citing works
Multimodal Emotion Captioning Using Large Language Model with Prompt Engineering
Xu, Y., Zhou, Y., Cai, Y., Xie, J., Ye, R., & Wu, Z. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689403 · Cited by 8 · see citing works
Li, C., Xie, L., Shao, X., Pan, H., & Wang, Z. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108413 · Cited by 7 · see citing works
Wang, C., Zhang, Q., Dong, J., Fang, H., Schaefer, G., Liu, R., & Yi, P. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113638 · Cited by 6 · see citing works
Building Robust Multimodal Sentiment Recognition via a Simple yet Effective Multimodal Transformer
Zong, D., Ding, C., Li, B., Zhou, D., Li, J., Zheng, K., & Zhou, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612872 · Cited by 6 · see citing works
Siamese capsule gorilla troops network-based multimodal sentiment analysis for car reviews
Kothuri, S. R., & Rajalakshmi, N. R. (2024) · Soft Computing · doi:10.1007/s00500-024-09754-4 · Cited by 6 · see citing works
Multimodal reaching-position prediction for ADL support using neural networks
Takase, Y., & Yamazaki, K. (2024) · ROBOMECH Journal · doi:10.1186/s40648-024-00282-2 · Cited by 3 · see citing works
Xlnet Parallel Hybrid Network Sentiment Analysis Based on Sentiment Word Augmentation
Xu, Z., Liao, Y., & Zhan, S. (2022) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs55155.2022.9845940 · Cited by 3 · see citing works
First-order Multi-label Learning with Cross-modal Interactions for Multimodal Emotion Recognition
Cai, Y., Xie, J., Tang, B., Wang, Y., Chen, J., Xue, H., & Wu, Z. (2023) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3607865.3613181 · Cited by 2 · see citing works
Emoji’s sentiment score estimation using convolutional neural network with multi-scale emoji images
Kulkongkoon, T., Cooharojananone, N., & Lipikorn, R. (2023) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i1.pp698-710 · Cited by 2 · see citing works
Liu, M., Chen, K., Ye, Q., & Wu, H. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104757 · Cited by 2 · see citing works
Kothuri, S. R., & Rajalakshmi, N. R. (2025) · International Journal of Uncertainty Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488525500035 · Cited by 2 · see citing works
Exclusive Modeling for MuSe-Personalisation Challenge
Sun, H., Wen, Z., Xu, M., Lian, Z., Sun, L., Liu, B., & Tao, J. (2023) · doi:10.1145/3606039.3613108 · Cited by 2 · see citing works
Self-citationAn Improved Vectorization-Based Emotion Detection Using Tuned Inverse Document Frequency Approach
Vanitha, R. (2024) · International Journal of Electronics and Communication Engineering · doi:10.14445/23488549/ijece-v11i3p111 · Cited by 2 · see citing works
Multimodal continuous emotion analysis
Zhang, S. (2023) · doi:10.32657/10356/166783
Zhang, Y. (2024) · Journal of Electrical Systems · doi:10.52783/jes.3055
Content-Aware Efficient Learner for Audio-Visual Emotion Recognition
Huang, G., Lin, W., & Liu, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1151-5_4
LGFN: A Dynamic Gating Framework for Lyrics-Audio Alignment in Music Emotion RecognitionPreprint
Wang, Y., Zang, M., Zhang, H., Shan, X., & Wang, X. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.11.669801
A Dual-Branch Ensemble Framework for Personality Recognition Based on Multimodal Emotion Features
Yu, R., Cai, Y., Zhou, Y., Ye, R., & Wu, Z. (2025) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3746270.3760217
Reinforced Context Augmentation for Multimodal Emotion Analysis
Gan, R., Tian, Y., Pan, K., Song, Y., & Zhang, Y. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3673543
A Quantum-Inspired Residual Self-Attention Network for Multimodal Sentiment Analysis
Liu, Y., Feng, X., & Zhong, Y. (2026) · Entropy · doi:10.3390/e28091014
- Authors
- Sun, L., Xu, M., Lian, Z., Liu, B., Tao, J., Wang, M., & Cheng, Y.
- Affiliations
- Chinese Academy of Sciences (CN)
- Ant Group (China) (CN)
- Venue
- Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge
- Publisher
- ACM
- Published
- 2021-10-20
- Pages
- 15-20
- OpenAlex
- W3206842948
- Cited by
- 36
- References
- 23
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 12 · 2023: 7 · 2022: 6 · 2021: 2
- FWCI
- 7.51
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Funders
- National Key Research & Development Plan of China (No.2017YFB1002804)
- National Natural Science Foundation of China (No.61831022, No.61771472, No.61773379, No.61901473)
- Key Program of the Natural Science Foundation of Tianjin (No. 18JCZDJC36300)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal emotion recognition, sentiment analysis, continuous emotion recognition, valence and arousal, long short-term memory, self-attention, decision fusion, user-generated video
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
