Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks
The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index.
Abstract
This paper presents our effects for Cross-cultural Emotion Sub-challenge in the Audio/Visual Emotion Challenge (AVEC) 2018, whose goal is to predict the level of three emotional dimensions time-continuously in a cross-cultural setup. We extract the emotional features from audio, visual and textual modalities. The state of art regressor for continuous emotion recognition, long short term memory recurrent neural network (LSTM-RNN) is utilized. We augment the training data by replacing the original training samples with shorter overlapping samples extracted from them, thus multiplying the number of training samples and also beneficial to train emotional temporal model with LSTM-RNN. In addition, two strategies are explored to decrease the interlocutor influence to improve the performance. We also compare the performance of feature level fusion and decision level fusion. The experimental results show the efficiency of the proposed method and competitive results are obtained.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In a following study, Huang et al. [83] enhanced data richness to evaluate the efficacy of early fusion. This was achieved during the 2018 Audio/Visual Emotion Challenge (AVEC), in which the researchers augmented the training data by replacing the original training samples with shorter overlapping samples extracted from them, thus multiplying the number of training samples.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [83]
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Word vectors are distributional word representations learned from massive textual dataset [42], which are not only more compact than BoW representation but also are related to the semantic meanings of the words.”
BackgroundADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Universals and cultural differences in facial expressions of emotion.
Ekman, P. (1972) · Medical Entomology and Zoology · Cited by 2,273
Nebraska Symposium on Motivation, 1969.
Schwartz, S., Arnold, W. J., & Levine, D. (1971) · American Sociological Review · doi:10.2307/2093748 · Cited by 2,171
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 2×
“This challenge adopts eGeMAPS [12] as the baseline audio features.”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477553 · Cited by 1,747
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179
Audio augmentation for speech recognition
Ko, T., Peddinti, V., Povey, D., & Khudanpur, S. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-711 · Cited by 1,104
AVEC 2013: the continuous audio/visual emotion and depression recognition challenge
Valstar, M., Schuller, B., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., & Pantic, M. (2013) · Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge · doi:10.1145/2512530.2512533 · Cited by 581
The INTERSPEECH 2010 paralinguistic challenge
Schuller, B., Steidl, S., Batliner, A., Burkhardt, F., Devillers, L., Müller, C., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-739 · Cited by 571
AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge
Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., & Pantic, M. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988258 · Cited by 503
AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge
Valstar, M., Schuller, B., Smith, K., Almaev, T., Eyben, F., Krajewski, J., Cowie, R., & Pantic, M. (2014) · Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2661806.2661807 · Cited by 472
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 426
Gunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360 · cited 1×
“Continuous emotion recognition as a function of time assigns an emotional value to every frame in a sequence [1, 2], usually consisting of arousal, valence, and dominance [3].”
BackgroundProceedings of Interspeech 2008
Moore, J. D., Tietze, M. I., & Demberg, V. (2008) · Cited by 356
AVEC 2017: Real-life Depression, and Affect Recognition Workshop and Challenge
Ringeval, F., Schuller, B., Valstar, M., Gratch, J., Cowie, R., Scherer, S., Mozgai, S., Cummins, N., Schmitt, M., & Pantic, M. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133953 · Cited by 332
LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288 · cited 1×
“Wöllmer et al. [25] utilize LSTM-RNN to perform regression analysis on arousal and valence dimension, which improves recognition performance significantly.”
MethodAVEC 2011–The First International Audio/Visual Emotion Challenge
Schuller, B., Valstar, M., Eyben, F., McKeown, G., Cowie, R., & Pantic, M. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24571-8_53 · Cited by 271
AVEC 2012: the continuous audio/visual emotion challenge
Schuller, B., Valster, M., Eyben, F., Cowie, R., & Pantic, M. (2012) · Proceedings of the 14th ACM international conference on Multimodal interaction · doi:10.1145/2388676.2388776 · Cited by 226
SoundNet: Learning Sound Representations from Unlabeled Video
Aytar, Y., Vondrick, C., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09001 · Cited by 222
Local Gabor Binary Patterns from Three Orthogonal Planes for Automatic Facial Expression Recognition
Almaev, T. R., & Valstar, M. F. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.65 · Cited by 201
Vocal Emotion Recognition Across Disparate Cultures
Bryant, G., & Barrett, H. C. (2008) · Journal of Cognition and Culture · doi:10.1163/156770908x289242 · Cited by 153
AVEC 2018 Workshop and Challenge: Bipolar Disorder and Cross-Cultural Affect Recognition
Ringeval, F., Schuller, B., Valstar, M., Cowie, R., Kaya, H., Schmitt, M., Amiriparian, S., Cummins, N., Lalanne, D., Michaud, A., et al. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266316 · Cited by 148
Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition
Chen, S., Jin, Q., Zhao, J., & Wang, S. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133949 · Cited by 147
Long Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2808196.2811634 · Cited by 120
Self-citationCorrecting Time-Continuous Emotional Labels by Modeling the Reaction Lag of Evaluators
Mariooryad, S., & Busso, C. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2334294 · Cited by 106
Emotion Recognition in the Wild with Feature Fusion and Multiple Kernel Learning
Chen, J., Chen, Z., Chi, Z., & Fu, H. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2666277 · Cited by 105
Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
Chen, S., & Jin, Q. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2967286 · Cited by 86
Multimodal Emotion Recognition for AVEC 2016 Challenge
Povolny, F., Matejka, P., Hradis, M., Popková, A., Otrusina, L., Smrz, P., Wood, I., Robin, C., & Lamel, L. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988268 · Cited by 80
Weninger, F. J., Ringeval, F., Marchi, E., & Schuller, B. W. (2016) · OPUS (Augsburg University) · Cited by 74
Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions
Lee, C.-C., Busso, C., Lee, S., & Narayanan, S. S. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-480 · Cited by 71
Huang, Z., Dang, T., Cummins, N., Stasak, B., Le, P., Sethu, V., & Epps, J. (2015) · Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2808196.2811640 · Cited by 69
Cross-language acoustic emotion recognition: An overview and some tendencies
Feraru, S. M., Schuller, D., & Schuller, B. W. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344561 · Cited by 67
Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network
Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., & Yi, J. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133946 · Cited by 54 · cited 3×
“Compared with SVR, LSTM can capture the temporal information of the emotional dimension better, which can achieve superior information [15][26].”
BackgroundMethodSelf-citationEmotion recognition using acoustic and lexical features
Rozgić, V., Ananthakrishnan, S., Saleem, S., Kumar, R., Vembu, A. N., & Prasad, R. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-118 · Cited by 50
AVEC 2015: The 5th International Audio/Visual Emotion Challenge and Workshop
Ringeval, F., Schuller, B., Valstar, M., Cowie, R., & Pantic, M. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2806408 · Cited by 50
Bhaykar, M., Yadav, J., & Rao, K. S. (2013) · National Conference on Communications · doi:10.1109/ncc.2013.6487998 · Cited by 43
Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2014) · Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2661806.2661811 · Cited by 39
Self-citationExploring Multimodal Visual Features for Continuous Affect Recognition
Sun, B., Cao, S., Li, L., He, J., & Yu, L. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988270 · Cited by 31
Convolutional Neural Networks with Data Augmentation for Classifying Speakers’ Native Language
Keren, G., Deng, J., Pohjalainen, J., & Schuller, B. W. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-261 · Cited by 25
Interaction and Transition Model for Speech Emotion Recognition in Dialogue
Zhang, R., Ando, A., Kobashikawa, S., & Aono, Y. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-713 · Cited by 18
Barrière, V., Clavel, C., & Essid, S. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462160 · Cited by 14
Investigation of Bottle-Neck Features for Emotion Recognition
Popková, A., Povolný, F., Matějka, P., Glembek, O., Grézl, F., & Černocký, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45510-5_49 · Cited by 7
Research works printed but not linked: 1
- H. W. Chung C. L. Jen L. W. Wen. 2014. Survey on audiovisual emotion recognition: databases features and data fusion strategies. APSIPA transactions on signal and information processing. H. W. Chung C. L. Jen L. W. Wen. 2014. Survey on audiovisual emotion recognition: databases features and data fusion strategies. APSIPA transactions on signal and information processing.
Cited by
A Survey of Data Augmentation Approaches for NLP
Feng, S., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.84 · Cited by 670
Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409
“nt detection to identify emotion markers invariant to socioculturalcontext. The other two entrants employed data driven approaches based on long short-term memory recurrent neural networks (LSTM-RNN) [27, 82]. Matching with similar results in the literature [21, 56], all entrants in the AVEC 2018 CES observed a drop i…”
ResultA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“In a following study, Huang et al. [83] enhanced data richness to evaluate the efficacy of early fusion.”
BackgroundMultimodal Transformer Fusion for Continuous Emotion Recognition
Huang, J., Tao, J., Liu, B., Lian, Z., & Niu, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053762 · Cited by 173
Self-citationTransformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Sahli, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 158
Metadata
- Authors
- Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M.
- Affiliations
- Chinese Academy of Sciences (CN)
- Venue
- Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop
- Publisher
- ACM
- Published
- 2018-10-15
- Pages
- 57-64
- OpenAlex
- W2896480997
- Cited by
- 40
- References
- 47
- Citations by year
- 2025: 6 · 2024: 5 · 2023: 9 · 2022: 4 · 2021: 5 · 2020: 5 · 2019: 6
- FWCI
- 4.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Proceedings
- Funders
- National Natural Science Foundation of China (No.61332017, No.61773379, NO.61425017, No.61603390, No.61771472)
- National Key Research & Development Plan of China (No. 2017YFB1002804)
- Major Program for the National Social Science Fund of China (13&ZD189)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- continuous emotion recognition, multimodal emotion recognition, cross-cultural emotion recognition, data augmentation, LSTM, feature-level fusion, decision fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.