An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods
The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
Abstract
Continuous emotion recognition is a challenging task and a key part of human-computer interaction, especially multimodal emotion recognition can effectively improve the accuracy and robustness of recognition. But there are limited emotion data sets, and it is difficult to extract emotion features. We present a multi-level segmented decision level fusion emotion recognition model to improve the performance of emotion recognition. In this paper, we predict multi-modal dimensional emotional state on AVEC2017 dataset. Our model uses Bidirectional Long Short-Term Memory (BLSTM) as multi-level segmented emotional feature learning model, and uses the SVR model as fusion model of the decision layer. The BLSTM can model different forms of emotional information in time, and can also consider the impact of previous and later emotional features on current results. The SVR model can compensate for the redundant information of emotion recognition. At the same time, we also consider annotation delay and temporal pooling in our multi-modal dimensional emotion recognition model. Our multi-modal emotion recognition model achieves significant recognition improvements and provide the robustness. Finally, we compare the baseline methods which used the same dataset, and find that the CCC performance of our method is the best on arousal, which is 0.685. Our research shows that the proposed multi-layer segmentation decision level fusion emotion recognition model is conducive to improving performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In an attempt to augment performance, Su et al. [87] proposed a multi-level segmented decision-level fusion model for affect recognition. Their model incorporated Bidirectional LSTM (BLSTM) as a multi-level segmental”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [87]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 1×
“2) IS10 Feature: We utilize the OpenSMILE toolkit [12] to extract the expert-knowledge based acoustic features with IS10 configuration [13], which can better express the characteristics of paralinguistic information.”
MethodThe Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 1×
“The following are specific feature introductions: 1) Bag-of-audio-words (BoAW): The eGeMAPS acoustic feature set is an extension of the Geneva minimal acoustic parameter set feature set [11].”
BackgroundSchuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.011 · Cited by 758
AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge
Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., & Pantic, M. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988258 · Cited by 503
Tao, J., & Tan, T. (2005) · Lecture Notes in Computer Science · doi:10.1007/11573548_125 · Cited by 494
Self-citationSpeech Emotion Recognition Using CNN
Huang, Z., Dong, M., Mao, Q., & Zhan, Y. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654984 · Cited by 398
AVEC 2017: Real-life Depression, and Affect Recognition Workshop and Challenge
Ringeval, F., Schuller, B., Valstar, M., Gratch, J., Cowie, R., Scherer, S., Mozgai, S., Cummins, N., Schmitt, M., & Pantic, M. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133953 · Cited by 332
Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.11.007 · Cited by 202 · cited 1×
“The factor of annotation delay is processed in many emotion recognition systems of AVEC [9][17][18]to improve the prediction performance.”
BackgroundLearning Alignment for Multimodal Emotion Recognition from Speech
Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., & Li, X. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3247 · Cited by 170
Cited by
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“In an attempt to augment performance, Su et al. [87] proposed a multi-level segmented decision-level fusion model for affect recognition.”
BackgroundA Visual–Audio-Based Emotion Recognition System Integrating Dimensional Analysis
Tian, J., & She, Y. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3200060 · Cited by 24
Review on Multimodal Fusion Techniques for Human Emotion Recognition
Karani, R. B., & Desai, S. S. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.0131035 · Cited by 32
Lu, Y.-Y., Chen, Z., Zheng, Q., Zhu, Y., & Wang, M. (2023) · Soft Computing · doi:10.1007/s00500-023-08760-2 · Cited by 5
RetractedSharma, D., Jayabalan, M., Sultanova, N., Mustafina, J., & Yao, D. N. L. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-0293-0_15 · Cited by 3
Metadata
- Authors
- Su, H., Liu, B., Tao, J., Dong, Y., Huang, J., Lian, Z., & Song, L.
- Affiliations
- Hebei University of Technology (CN)
- Institute of Automation (CN)
- Venue
- 2020 15th IEEE International Conference on Signal Processing (ICSP)
- Publisher
- IEEE
- Published
- 2020-12-06
- Pages
- 257-261
- OpenAlex
- W3127502324
- Cited by
- 10
- References
- 26
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 3 · 2023: 2 · 2022: 2
- FWCI
- 0.97
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- International Conference on Signal Processing (publishedVersion)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal emotion recognition, decision fusion, bidirectional LSTM, support vector regression, arousal prediction, temporal pooling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods. https://app.cochat.ai/papers/10.1109/icsp48669.2020.9321008
- The paper
- Su, H., Liu, B., Tao, J., Dong, Y., Huang, J., Lian, Z., & Song, L. (2020). An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods. 2020 15th IEEE International Conference on Signal Processing (ICSP). https://doi.org/10.1109/icsp48669.2020.9321008
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.