Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks
The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index.
Abstract
This paper presents our effects for Cross-cultural Emotion Sub-challenge in the Audio/Visual Emotion Challenge (AVEC) 2018, whose goal is to predict the level of three emotional dimensions time-continuously in a cross-cultural setup. We extract the emotional features from audio, visual and textual modalities. The state of art regressor for continuous emotion recognition, long short term memory recurrent neural network (LSTM-RNN) is utilized. We augment the training data by replacing the original training samples with shorter overlapping samples extracted from them, thus multiplying the number of training samples and also beneficial to train emotional temporal model with LSTM-RNN. In addition, two strategies are explored to decrease the interlocutor influence to improve the performance. We also compare the performance of feature level fusion and decision level fusion. The experimental results show the efficiency of the proposed method and competitive results are obtained.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In a following study, Huang et al. [83] enhanced data richness to evaluate the efficacy of early fusion. This was achieved during the 2018 Audio/Visual Emotion Challenge (AVEC), in which the researchers augmented the training data by replacing the original training samples with shorter overlapping samples extracted from them, thus multiplying the number of training samples.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [83]
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Word vectors are distributional word representations learned from massive textual dataset [42], which are not only more compact than BoW representation but also are related to the semantic meanings of the words.”
BackgroundADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Universals and cultural differences in facial expressions of emotion.
Ekman, P. (1972) · Medical Entomology and Zoology · Cited by 2,273
Nebraska Symposium on Motivation, 1969.
Schwartz, S., Arnold, W. J., & Levine, D. (1971) · American Sociological Review · doi:10.2307/2093748 · Cited by 2,171
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 2×
“This challenge adopts eGeMAPS [12] as the baseline audio features.”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477553 · Cited by 1,747
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179
Research works printed but not linked: 1
- H. W. Chung C. L. Jen L. W. Wen. 2014. Survey on audiovisual emotion recognition: databases features and data fusion strategies. APSIPA transactions on signal and information processing. H. W. Chung C. L. Jen L. W. Wen. 2014. Survey on audiovisual emotion recognition: databases features and data fusion strategies. APSIPA transactions on signal and information processing.
Cited by
A Survey of Data Augmentation Approaches for NLP
Feng, S., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.84 · Cited by 670
Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“In a following study, Huang et al. [83] enhanced data richness to evaluate the efficacy of early fusion.”
BackgroundMultimodal Transformer Fusion for Continuous Emotion Recognition
Huang, J., Tao, J., Liu, B., Lian, Z., & Niu, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053762 · Cited by 173
Self-citationTransformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Sahli, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 158
Metadata
- Authors
- Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M.
- Affiliations
- Chinese Academy of Sciences (CN)
- Venue
- Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop
- Publisher
- ACM
- Published
- 2018-10-15
- Pages
- 57-64
- OpenAlex
- W2896480997
- Cited by
- 40
- References
- 47
- Citations by year
- 2025: 6 · 2024: 5 · 2023: 9 · 2022: 4 · 2021: 5 · 2020: 5 · 2019: 6
- FWCI
- 4.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Proceedings
- Funders
- National Natural Science Foundation of China (No.61332017, No.61773379, NO.61425017, No.61603390, No.61771472)
- National Key Research & Development Plan of China (No. 2017YFB1002804)
- Major Program for the National Social Science Fund of China (13&ZD189)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- continuous emotion recognition, multimodal emotion recognition, cross-cultural emotion recognition, data augmentation, LSTM, feature-level fusion, decision fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks. https://app.cochat.ai/papers/10.1145/3266302.3266304
- The paper
- Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M. (2018). Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks. Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop. https://doi.org/10.1145/3266302.3266304
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.