Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks
The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index.
Abstract
This paper presents our effects for Cross-cultural Emotion Sub-challenge in the Audio/Visual Emotion Challenge (AVEC) 2018, whose goal is to predict the level of three emotional dimensions time-continuously in a cross-cultural setup. We extract the emotional features from audio, visual and textual modalities. The state of art regressor for continuous emotion recognition, long short term memory recurrent neural network (LSTM-RNN) is utilized. We augment the training data by replacing the original training samples with shorter overlapping samples extracted from them, thus multiplying the number of training samples and also beneficial to train emotional temporal model with LSTM-RNN. In addition, two strategies are explored to decrease the interlocutor influence to improve the performance. We also compare the performance of feature level fusion and decision level fusion. The experimental results show the efficiency of the proposed method and competitive results are obtained.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In a following study, Huang et al. [83] enhanced data richness to evaluate the efficacy of early fusion. This was achieved during the 2018 Audio/Visual Emotion Challenge (AVEC), in which the researchers augmented the training data by replacing the original training samples with shorter overlapping samples extracted from them, thus multiplying the number of training samples.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [83]
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Word vectors are distributional word representations learned from massive textual dataset [42], which are not only more compact than BoW representation but also are related to the semantic meanings of the words.”
BackgroundADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Universals and cultural differences in facial expressions of emotion.
Ekman, P. (1972) · Medical Entomology and Zoology · Cited by 2,273
Nebraska Symposium on Motivation, 1969.
Schwartz, S., Arnold, W. J., & Levine, D. (1971) · American Sociological Review · doi:10.2307/2093748 · Cited by 2,171
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 2×
“This challenge adopts eGeMAPS [12] as the baseline audio features.”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477553 · Cited by 1,747
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179
Research works printed but not linked: 1
- H. W. Chung C. L. Jen L. W. Wen. 2014. Survey on audiovisual emotion recognition: databases features and data fusion strategies. APSIPA transactions on signal and information processing. H. W. Chung C. L. Jen L. W. Wen. 2014. Survey on audiovisual emotion recognition: databases features and data fusion strategies. APSIPA transactions on signal and information processing.
Cited by
A Survey of Data Augmentation Approaches for NLP
Feng, S., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.84 · Cited by 670
Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409
“nt detection to identify emotion markers invariant to socioculturalcontext. The other two entrants employed data driven approaches based on long short-term memory recurrent neural networks (LSTM-RNN) [27, 82]. Matching with similar results in the literature [21, 56], all entrants in the AVEC 2018 CES observed a drop i…”
ResultA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“In a following study, Huang et al. [83] enhanced data richness to evaluate the efficacy of early fusion.”
BackgroundMultimodal Transformer Fusion for Continuous Emotion Recognition
Huang, J., Tao, J., Liu, B., Lian, Z., & Niu, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053762 · Cited by 173
Self-citationTransformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Sahli, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 160
An Active Learning Paradigm for Online Audio-Visual Emotion Recognition
Kansizoglou, I., Bampis, L., & Gasteratos, A. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961089 · Cited by 121
SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing
Chen, W., Xing, X., Xu, X., Pang, J., & Du, L. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3235194 · Cited by 117
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81
Self-citationPham, N. T., Dang, D. N. M., Nguyen, N. D., Nguyen, T. T., Nguyen, V., Manavalan, B., Lim, C. P., & Nguyen, S. D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120608 · Cited by 71
Huang, Z., Epps, J., Joachim, D., & Sethu, V. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2949419 · Cited by 70
Kaya, H., Fedotov, D., Dresvyanskiy, D., Doyran, M., Mamontov, D., Markitantov, M., Salah, A. A. A., Kavcar, E., Karpov, A. A., & Salah, A. A. (2019) · doi:10.1145/3347320.3357691 · Cited by 53
Continuous Emotion Recognition in Speech — Do We Need Recurrence?
Schmitt, M., Cummins, N., & Schuller, B. W. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2710 · Cited by 47
PriMonitor: An adaptive tuning privacy-preserving approach for multimodal emotion detection
Yin, L., Lin, S., Sun, Z., Wang, S., Li, R., & He, Y. (2024) · World Wide Web · doi:10.1007/s11280-024-01246-7 · Cited by 44
Xu, N., Mao, W., Wei, P., & Zeng, D. D. (2020) · IEEE Intelligent Systems · doi:10.1109/mis.2020.3026715 · Cited by 44
Speech-Based Depression Assessment: A Comprehensive Survey
Leal, S. S., Ntalampiras, S., & Sassi, R. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3521327 · Cited by 41
Cross-modal contrastive learning for multimodal sentiment recognition
Yang, S., Cui, L., Wang, L., & Wang, T. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05355-8 · Cited by 38
Hybrid Mutimodal Fusion for Dimensional Emotion Recognition
Ma, Z., Ma, F., Sun, B., & Li, S. (2021) · doi:10.1145/3475957.3484457 · Cited by 38
Modularized composite attention network for continuous music emotion recognition
Zhang, M., Zhu, Y., Zhang, W., Zhu, Y., & Feng, T. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13577-6 · Cited by 26
Analysis and Recognition of Voluntary Facial Expression Mimicry Based on Depressed Patients
Ye, J., Yu, Y., Fu, G., Zheng, Y., Liu, Y., Zhu, Y., & Wang, Q. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3260816 · Cited by 26
Liu, Y., Xing-yun, L., Wang, M., Bi, J., Lin, S., Wang, Q., Yu, Y., Ye, J., & Zheng, Y. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.03.155 · Cited by 26
Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders
Hamieh, S., Heiries, V., Osman, H. A., & Godin, C. (2021) · doi:10.1145/3475957.3484455 · Cited by 19
Learning long-term temporal contexts using skip RNN for continuous emotion recognition
Huang, J., Liu, B., & Tao, J. (2021) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2020.11.005 · Cited by 17
Self-citationGao, X., Bansal, S., Gowda, K., Li, Z., Nayak, S., Kumar, N., & Coler, M. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3639406 · Cited by 15
AffectFAL: Federated Active Affective Computing with Non-IID Data
Zhang, Z., Qi, F., Li, S., & Xu, C. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612442 · Cited by 12
Sun, L., Lei, Y., Zhang, Z., Tang, Y., Wang, J., Ye, L., & Li, P. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107811 · Cited by 12
Ghiasi, S., Patanè, A., Laurenti, L., Gentili, C., Scilingo, E. P., Greco, A., & Kwiatkowska, M. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3224775 · Cited by 10
Branch-Fusion-Net for Multi-Modal Continuous Dimensional Emotion Recognition
Li, C., Xie, L., & Pan, H. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3160373 · Cited by 9
Depression and anxiety detection method based on serialized facial expression imitation
Lü, L., Jiang, Y. L., Xing-yun, L., Wang, H., Zou, Q., & Wang, Q. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110354 · Cited by 8
LSTM and GPT-2 Synthetic Speech Transfer Learning for Speaker Recognition to Overcome Data ScarcityPreprint
Bird, J. J., Faria, D. R., Ekárt, A., Premebida, C., & Ayrosa, P. P. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00659 · Cited by 8
Pham, N. T., Dang, D. N. M., & Nguyen, S. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09026 · Cited by 8
Emotion recognition based on multi-modal electrophysiology multi-head attention Contrastive LearningPreprint
Guo, Y., Zhang, T., & Huang, W. X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01919 · Cited by 8
An Effective Data Fusion Methodology for Multi-modal Emotion Recognition: A Survey
(2024) · International Journal of Emerging Trends in Engineering Research · doi:10.30534/ijeter/2024/011272024 · Cited by 5
Raju, J., Gaus, Y. F. A., & Breckon, T. P. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00115 · Cited by 5
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot LearningPreprint
Dogan, M., Kesen, İ., Calixto, I., Erdem, A., & Erdem, E. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.12498 · Cited by 5
Progress in Multimodal Affective Computing: From Machine Learning to Deep Learning
Chanchal, M., & Kumar, B. V. (2023) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-031-20541-5_6 · Cited by 4
Ayoub, I. (2019) · uO Research (University of Ottawa) · doi:10.20381/ruor-23584 · Cited by 4
Multimodal transformers for emotion recognition
Vazquez-Rodriguez, J. (2023) · theses.fr (ABES) · doi:10.70675/a1927be2zdba6z405azbe6fz6856529b46bb · Cited by 3
Shuffling Augmented Decoupled Features for Multimodal Emotion Recognition
Cho, S. (2025) · IEEE Access · doi:10.1109/access.2025.3572925 · Cited by 2
Continuous Emotion Prediction from Speech: Modelling Ambiguity in Emotion
Bose, D. (2023) · UNSWorks (University of New South Wales, Sydney, Australia) · doi:10.26190/unsworks/25132 · Cited by 2
Orthogonal Synthesis: Data Augmentation for Multimodal Sentiment Analysis
Zheng, T., & Liu, W. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650640 · Cited by 1
Multimodal Affect Recognition Using Temporal Convolutional Neural Networks
Ayoub, I., Heiries, V., & Osman, H. A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892145 · Cited by 1
SLMF: Adaptive Short-Long Multi-Stage Fusion for Emotion Recognition in Conversation
Zhao, Y., Lin, Z., Wang, W., & Zhou, X. (2023) · IEEE International Conference on Medical Artificial Intelligence (MedAI) · doi:10.1109/medai59581.2023.00011 · Cited by 1
Multimodal continuous emotion analysis
Zhang, S. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/166783 · Cited by 1
Macary, M. (2022) · theses.fr (ABES) · doi:10.70675/900e2862z85aez4bfbz91f3za4e7a5bebbe8 · Cited by 1
On the Adversarial Robustness ofGaussian Processes
Patanè, A. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5092157
Decoding Hostility from Conversations Through Speech and Text Integration
Xin, J., Gump, B. B., Maisto, S. A., Jorgensen, R. S., Bhatia, T. K., Phoha, V. V., & Salekin, A. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00032
LGFN: A Dynamic Gating Framework for Lyrics-Audio Alignment in Music Emotion RecognitionPreprint
Wang, Y., Zang, M., Zhang, H., Shan, X., & Wang, X. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.11.669801
Empowering Emotion Recognition with Flexible Modality Information
LI, D., & LI, D. (2024) · Institutional Repositories DataBase (IRDB)
Neural recommender models for sparse and skewed behavioral data
Krishnan, A. (2021) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
On the adversarial robustness of Gaussian processes
Patane, A. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-dxxpay2za
Head- and eye-based features for continuous core affect prediction
O'Dwyer, J. (. (2020) · Research@THEA
Multimodal Emotion Detection and Classification of Audio-Visual Using Deep Learning
Srivastava, M., Singh, P., & Sharma, M. K. (2025) · International Conference on Advanced Computing and Emerging Technologies (ACET) · doi:10.1109/acet67282.2025.11430197
Leveraging Anomaly Detection for Affective Computing Applications
Hamieh, S. (2024) · theses.fr (ABES) · doi:10.70675/e2afb792zceb2z4868z9e58z09a90764b943
Metadata
- Authors
- Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M.
- Affiliations
- Chinese Academy of Sciences (CN)
- Venue
- Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop
- Publisher
- ACM
- Published
- 2018-10-15
- Pages
- 57-64
- OpenAlex
- W2896480997
- Cited by
- 40
- References
- 47
- Citations by year
- 2025: 6 · 2024: 5 · 2023: 9 · 2022: 4 · 2021: 5 · 2020: 5 · 2019: 6
- FWCI
- 4.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Proceedings
- Funders
- National Natural Science Foundation of China (No.61332017, No.61773379, NO.61425017, No.61603390, No.61771472)
- National Key Research & Development Plan of China (No. 2017YFB1002804)
- Major Program for the National Social Science Fund of China (13&ZD189)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- continuous emotion recognition, multimodal emotion recognition, cross-cultural emotion recognition, data augmentation, LSTM, feature-level fusion, decision fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.