Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 2 of the 6 sources its claims cite were checked for support.
Abstract
In recent years, deep learning has been increasingly used in the field of multimodal emotion recognition in conjunction with electroencephalogram. Considering the complexity of recording electroencephalogram signals, some researchers have applied deep learning to find new features for emotion recognition. In previous studies, convolutional neural network model was used to automatically extract features and complete emotion recognition, and certain results were obtained. However, the extraction of hierarchical features with convolutional neural network for multimodal emotion recognition remains unexplored. Therefore, this paper proposes a hierarchical fusion convolutional neural network model to mine the potential information in the data by constructing different network hierarchical structures, extracting multiscale features, and using feature-level fusion to fuse the global features formed by combining weights with manually extracted statistical features to form the final feature vector. This paper conducts binary classification experiments on the valence and arousal dimensions of the DEAP and MAHNOB-HCI data sets to evaluate the performance of the proposed model. The results show that the model proposed in this paper can achieve accuracies of 84.71% and 89.00% on the two corresponding data sets, indicating that the model proposed in this paper is superior to other deep learning emotion classification models in feature extraction and fusion.
Overview
CoChat Summary
TL;DR
The paper proposes a hierarchical fusion convolutional neural network (HFCNN) for multimodal emotion recognition from EEG and peripheral physiological signals, using weighted fusion of multiscale convolutional features plus manual statistical features. It reports binary valence/arousal classification on DEAP and MAHNOB-HCI, with the best accuracies of 84.71% and 89.00%, respectively.
Key findings
- HFCNN achieved average accuracies on DEAP of 83.28% for arousal and 84.71% for valence; on MAHNOB-HCI, 88.28% for arousal and 89.00% for valence (Table 2).
- HFCNN outperformed CNN-alone feature extraction under optimal RF settings: DEAP arousal 83.28% vs 75.98%, DEAP valence 84.71% vs 79.52%; MAHNOB-HCI arousal 88.28% vs 70.17%, MAHNOB-HCI valence 89.00% vs 71.38% (Table 2).
- In comparison with prior methods on DEAP, HFCNN’s 84.71% exceeded SAE+LSTM (81.10%), multi-hypergraph neural networks (82.95%), CapsNet (68.28%), and hierarchical bidirectional GRU (67.90%) (Table 3).
- Valence performance was better than arousal on the same dataset, and MAHNOB-HCI accuracy was higher than DEAP accuracy when classifier parameters were optimal.
Methods
- The model uses observation-level fusion of EEG and peripheral physiological signals (PPS), then a hierarchical incremental CNN with three convolutional layers: 8 filters of size 7×7, 12 filters of size 5×5, and 16 filters of size 3×3; max pooling is 2×2 with stride 2. Features MPs-1, MPs-2, and MPs-3 are extracted after pooling, weighted, and combined into MPs-global, then fused with manually extracted statistical features. A random forest (RF) classifier is used with ten-fold cross-validation for binary valence and arousal classification.
- Data: DEAP included 32 subjects watching 40 videos; MAHNOB-HCI included 25 subjects watching 20 videos after excluding 3 subjects with missing data and 2 with incomplete records from the original 30. Inputs used six forehead EEG channels (FP1, FP2, AF3, AF4, F3, F4) and four PPS signals (GSR, RESP, TEMP, PLET). The middle 30 s of each video was used, with 6-s windows and 50% overlap applied to five frequency bands.
- Manual features included Hjorth parameters in the time domain, mean power spectral density and mean difference entropy after Fourier transform for EEG, and mean, variance, maximum, minimum, and max-min differences of PPS signals and their first- and second-order differences. RF decision-tree parameters were fixed at 450 and 200 for DEAP arousal and valence, and 600 and 500 for MAHNOB-HCI arousal and valence.
Limitations
- Stated: the authors call for further research on extracting more discriminative multimodal features for cross-subject emotion classification, selecting/constructing/optimizing deep learning models with higher accuracy, robustness, and generalization, incorporating emotion-related brain neurogenic analysis, and reducing sensory-induced interference.
- Evident from the text: MAHNOB-HCI results are based on only 25 of 30 subjects because of missing or incomplete data records, and class imbalance was handled by expanding the smaller categories, which may affect dataset representativeness.
- Evident from the text: evaluation is limited to two datasets (DEAP and MAHNOB-HCI) and binary valence/arousal tasks, so broader generalization is not demonstrated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 4 Not checked · 25 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, each layer of features extracted by the proposed HFCNN was tested separately, and the results proved that only the weighted feature fusion method that fuses the features of the three incremental network structures can achieve good experimental results.”
- Also stated in: § B. MULTIMODAL EMOTION RECOGNITION RESULT · plain
“each layer of features extracted by the proposed HFCNN was tested separately, and the results proved that only the weighted feature fusion method that fuses the features of the three incremental network structures can achieve good experimental results.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § B. MULTIMODAL EMOTION RECOGNITION RESULT · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“in the DEAP data set, the average accuracy of arousal is 83.28% and valence is 84.71%”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that the model proposed in this paper can achieve accuracies of 84.71% and 89.00% on the two corresponding data sets, indicating that the model proposed in this paper is superior to other deep learning emotion classi cation models in feature extraction and fusion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the MAHNOB-HCI data set, the accuracy of arousal is 88.28% and valence is 89.00%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can be concluded that using the same data set in the parameter optimization case, the performance of valence is better than that of arousal”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
11 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 9
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ V. CONCLUSION
“We presented a subject-independent emotion recognition system that is suitable for real-time operation”
§ V. CONCLUSION
“Finally, a 10-fold cross-validation data set was created for each subject, and the HFCNN model was used to perform binary emotion classi cation experiments in the valence and arousal dimensions.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ V. CONCLUSION
“Then, the dataset was balanced to ensure that the number of the two classes was equal to improve the normalization ability of the classi cation model.”
§ A. DATA PREPROCESSING AND OBSERVATION-LEVEL FUSION OF MULTIMODAL SIGNALS
“| HA | 33930 | HV | 32580 | HA | 11475 | HV | 12015 | | LA | 33670 | LV | 32020 | LA | 11025 | LV | 11985 |”
Citations
Cites
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
EEG-Based Emotion Recognition Using Regularized Graph Neural Networks
Zhong, P., Wang, D., & Miao, C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2994159 · Cited by 910
Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion
Amin, S. U., Alsulaiman, M., Muhammad, G., Mekhtiche, M. A., & Shamim Hossain, M. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.06.027 · Cited by 523
Domain Adaptation for EEG Emotion Recognition Based on Latent Representation Similarity
Li, J., Qiu, S., Du, C., Wang, Y., & He, H. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2019.2949306 · Cited by 291
Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E., & Arunkumar, N. (2019) · IEEE Access · doi:10.1109/access.2018.2883213 · Cited by 289 · cited 1×
“[21] used a CNN to automatically extract features from a variety of physiological signals and perform sentiment prediction through fully connected network layers.”
MethodEmotion Recognition from Multiband EEG Signals Using CapsNet
Chao, H., Dong, L., Liu, Y., & Lu, B. (2019) · Sensors · doi:10.3390/s19092212 · Cited by 287 · cited 1×
“[31] extracted multiband features from multichannel EEG signals to constitute the multiband feature matrix (MFM), and entered the capsule network to mine three related features.”
MethodSAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG
Xing, X., Li, Z., Xu, T., Shu, L., Hu, B., & Xu, X. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00037 · Cited by 269
A machine learning model for emotion recognition from physiological signals
Domínguez-Jiménez, J. A., Campo-Landines, K. C., Martínez-Santos, J. C., Delahoz, E. J., & Contreras-Ortiz, S. H. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.101646 · Cited by 232
Emotion Recognition using Multimodal Residual LSTM Network
Ma, J., Tang, H., Zheng, W.-L., & Lu, B.-L. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350871 · Cited by 230 · cited 1×
“…results show that automatic feature extraction performs better than manual feature extraction; and various deep learning technologies [10], including autoen-coders (AEs), convolutional neural networks (CNNs) [11], and recurrent neural networks (RNNs) [12], are widely used in different domains.”
Background
Cited by
Recognition of human emotions using EEG signals: A review
Rahman, M. M., Sarkar, A. K., Hossain, M. A., Hossain, M. S., Islam, M. R., Hossain, M. B., Quinn, J. M. W., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104696 · Cited by 227
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Deep learning-based EEG emotion recognition: Current trends and future perspectives
Wang, X., Ren, Y., Luo, Z., He, W., Hong, J. S., & Huang, Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1126994 · Cited by 218
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 167
Metadata
- Authors
- Zhang, Y., Cheng, C., & Zhang, Y.
- Affiliations
- Liaoning Normal University (CN)
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 7943-7951
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3120929420
- Semantic Scholar
- 96cb58a7255129284cae50002651fc01b7b74080
- Cited by
- 84
- References
- 32
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 27 · 2023: 20 · 2022: 11 · 2021: 3
- FWCI
- 14.74
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61772252)
- Natural Science Foundation of Liaoning Province of China (2019-MS-216)
- Program for Liaoning Innovative Talents in University (LR2017044)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, EEG, feature-level fusion, multi-scale feature extraction, valence and arousal classification, DEAP dataset, automatic feature extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network. https://app.cochat.ai/papers/10.1109/access.2021.3049516
- The paper
- Zhang, Y., Cheng, C., & Zhang, Y. (2021). Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network. IEEE Access. https://doi.org/10.1109/access.2021.3049516
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.