Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Automatic perception and understanding of human emotion or sentiment has a wide range of applications and has attracted increasing attention nowadays. The Multimodal Sentiment Analysis in Real-life Media (MuSe) 2020 provides a testing bed for recognizing human emotion or sentiment from multiple modalities (audio, video, and text) in the wild scenario. In this paper, we present our solutions to the MuSe-Wild sub-challenge of MuSe 2020. The goal of this sub-challenge is to perform continuous emotion (arousal and valence) predictions on a car review database, Muse-CaR. To this end, we first extract both handcrafted features and deep representations from multiple modalities. Then, we utilize the Long Short-Term Memory (LSTM) recurrent neural network as well as the self-attention mechanism to model the complex temporal dependencies in the sequence. The Concordance Correlation Coefficient (CCC) loss is employed to guide the model to learn local variations and the global trend of emotion simultaneously. Finally, two fusion strategies, early fusion and late fusion, are adopted to further boost the model's performance by exploiting complementary information from different modalities. Our proposed method achieves CCC of 0.4726 and 0.5996 for arousal and valence respectively on the test set, which outperforms the baseline system with corresponding CCC of 0.2834 and 0.2431.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recently, Sun et al. [88,89] employed LSTM and a self-attention mechanism to capture complex temporal dependencies in feature sequences. They then fused the prediction results of several unimodal affect recognition models using a second-level LSTM as the fusion model.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [88]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“Compared to other representations, such as the one-hot vector, the word vector is more compact and can capture linguistic regularities in language [32].”
BackgroundMethodA concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · cited 8×
“We can notice that “CCC” loss outperforms “MSE” and “L1” loss.”
MethodInfluentialOpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929257 · Cited by 5,536
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · cited 2×
“It is trained on a large-scale audio dataset, AudioSet [17], which contains over 2 million human-labeled video soundtracks with more than 600 audio event classes.”
MethodOpensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 2×
“We use the eGeMAPS feature provided by the organizers of MuSe 2020, which utilize the freely available openSMILE toolkit [16] to extract it.”
MethodMultimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 1×
“Besides, Transformer-like [42] architectures can be employed to model the complex temporal dynamics of emotion in the future.”
MethodTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 1×
“More advanced fusion methods such as attention-based fusion [11, 29] or tensor fusion [46] can be explored.”
BackgroundCNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952132 · Cited by 2,023
Openface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 1×
“eGeMAPS Feature: The extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) contains 23 acoustic low-level descriptors (LLDs), covering spectral, cepstral, and prosodic features [15].”
BackgroundA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Training deep networks for facial expression recognition with crowd-sourced label distribution
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993165 · Cited by 926
Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904
Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., Chetouani, M., Weninger, F., Eyben, F., Marchi, E., et al. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-56 · Cited by 731
pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis
Giannakopoulos, T. (2015) · PLOS ONE · doi:10.1371/journal.pone.0144610 · Cited by 454
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
Albanie, S., Nagrani, A., Vedaldi, A., & Zisserman, A. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240578 · Cited by 290
LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288 · cited 1×
“Wöllmer et al. [44] and Chen et al. [12] both utilize SVR and LSTM to perform regression analysis.”
MethodAffective computing in education: A systematic review and future research
Yadegaridehkordi, E., Noor, N. F. B. M., Ayub, M. N. B., Affal, H. B., & Hussin, N. B. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103649 · Cited by 239
Multimodal Transformer Fusion for Continuous Emotion Recognition
Huang, J., Tao, J., Liu, B., Lian, Z., & Niu, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053762 · Cited by 173
Self-citationMultimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition
Chen, S., Jin, Q., Zhao, J., & Wang, S. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133949 · Cited by 147 · cited 4×
“needs to align multiple features due to the different sampling rates and may suffer from the problems caused by high dimensionality [12].”
BackgroundMethodInfluentialLong Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2808196.2811634 · Cited by 120
Self-citationEmotion Recognition in the Wild with Feature Fusion and Multiple Kernel Learning
Chen, J., Chen, Z., Chi, Z., & Fu, H. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2666277 · Cited by 105
Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
Chen, S., & Jin, Q. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2967286 · Cited by 86
Luo, D., Zou, Y., & Huang, D. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1832 · Cited by 77
Affective Computing for Large-scale Heterogeneous Multimedia Data: A Survey
Zhao, S., Wang, S., Soleymani, M., Joshi, D., & Ji, Q. (2019) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3363560 · Cited by 77
Stappen, L., Baird, A., Rizos, G., Tzirakis, P., Du, X., Hafner, F., Schumann, L., Mallol-Ragolta, A., Schuller, B. W., Lefter, I., et al. (2020) · doi:10.1145/3423327.3423673 · Cited by 67
Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network
Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., & Yi, J. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133946 · Cited by 54 · cited 5×
“At first, competitors use the classic bag-of-words features [12, 23].”
BackgroundMethodInfluentialSelf-citationConversational Emotion Analysis via Attention Mechanisms
Lian, Z., Tao, J., Liu, B., & Huang, J. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1577 · Cited by 46
Self-citationDu, Z., Wu, S., Huang, D., Li, W., & Wang, Y. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2940224 · Cited by 44
Zhao, J., Li, R., Liang, J., Chen, S., & Jin, Q. (2019) · doi:10.1145/3347320.3357692 · Cited by 43
Multi-modal Multi-cultural Dimensional Continues Emotion Recognition in Dyadic Interactions
Zhao, J., Li, R., Chen, S., & Jin, Q. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266313 · Cited by 42 · cited 11×
“Particularly, we extract contextual word representations from textual modality, which has not been utilized in corresponding sub-challenges of previous AVECs.”
BackgroundMethodResultInfluentialMultimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks
Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266304 · Cited by 40
Self-citationEfficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition
Chen, H., Deng, Y., Cheng, S., Wang, Y., Jiang, D., & Sahli, H. (2019) · doi:10.1145/3347320.3357690 · Cited by 40
Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2014) · Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2661806.2661811 · Cited by 39
Self-citationAutomatic Depression Level Detection via ℓp-Norm Pooling
Niu, M., Tao, J., Liu, B., & Fan, C. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1617 · Cited by 39
Self-citationAudiovisual three-level fusion for continuous estimation of Russell's emotion circumplex
Sánchez, E., Lopez‐Otero, P., Docío-Fernández, L., Argones-Rúa, E., & Alba‐Castro, J. L. (2013) · doi:10.1145/2512530.2512534 · Cited by 31
Efficient Modeling of Long Temporal Contexts for Continuous Emotion Recognition
Huang, J., Tao, J., Liu, B., Lian, Z., & Niu, M. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925452 · Cited by 10
Self-citation
Research works printed but not linked: 4
- Jimmy Lei Ba , Jamie Ryan Kiros, and Geoffrey E Hinton . 2016 . Layer normalization. arXiv preprint arXiv:1607.06450 (2016). Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv
- Yann LeCun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. nature , Vol. 521 , 7553 ( 2015 ), 436--444. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature , Vol. 521, 7553 (2015), 436--444.
- Qi Liu , Matt J Kusner , and Phil Blunsom . 2020. A Survey on Contextual Embeddings. arXiv preprint arXiv:2003.07278 ( 2020 ). Qi Liu, Matt J Kusner, and Phil Blunsom. 2020. A Survey on Contextual Embeddings. arXiv preprint arXiv:2003.07278 (2020). · arXiv
Cited by
Disentangled Representation Learning for Multimodal Emotion Recognition
Yang, D., Huang, S., Kuang, H., Du, Y., & Zhang, L. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547754 · Cited by 266
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Recently, Sun et al. [88,89] employed LSTM and a self-attention mechanism to capture complex temporal dependencies in feature sequences.”
Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning
Luna-Jiménez, C., Griol, D., Callejas, Z., Kleinlein, R., Montero, J. M., & Fernández-Martínez, F. (2021) · Sensors · doi:10.3390/s21227665 · Cited by 161
A study on computer vision for facial emotion recognition
Huang, Z.-Y., Chiang, C., Chen, J., Chen, Y.-C., Chung, H.-L., Cai, Y. A., & Hsu, H.-C. (2023) · Scientific Reports · doi:10.1038/s41598-023-35446-4 · Cited by 218
Luna-Jiménez, C., Kleinlein, R., Griol, D., Callejas, Z., Montero, J. M., & Fernández-Martínez, F. (2021) · Applied Sciences · doi:10.3390/app12010327 · Cited by 149
Metadata
- Authors
- Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M.
- Affiliations
- Chinese Academy of Sciences (CN)
- Venue
- Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop
- Publisher
- ACM
- Published
- 2020-10-15
- Pages
- 27-34
- OpenAlex
- W3097075955
- Cited by
- 81
- References
- 44
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 16 · 2023: 20 · 2022: 10 · 2021: 18
- FWCI
- 11.68
- Type
- conference paper
- Language
- en
- Open access
- closed
- Funders
- Key Program of the Natural Science Foundation of Tianjin (No. 18JCZDJC36300)
- National Key Research & Development Plan of China (No.2017YFB1002804)
- National Natural Science Foundation of China (No.61831022, No.61771472, No.61773379, No.61901473)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- continuous emotion recognition, multimodal sentiment analysis, LSTM, self-attention, feature-level fusion, decision fusion, temporal dependencies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.