Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 2 of the 6 sources its claims cite were checked for support.
Abstract
In recent years, deep learning has been increasingly used in the field of multimodal emotion recognition in conjunction with electroencephalogram. Considering the complexity of recording electroencephalogram signals, some researchers have applied deep learning to find new features for emotion recognition. In previous studies, convolutional neural network model was used to automatically extract features and complete emotion recognition, and certain results were obtained. However, the extraction of hierarchical features with convolutional neural network for multimodal emotion recognition remains unexplored. Therefore, this paper proposes a hierarchical fusion convolutional neural network model to mine the potential information in the data by constructing different network hierarchical structures, extracting multiscale features, and using feature-level fusion to fuse the global features formed by combining weights with manually extracted statistical features to form the final feature vector. This paper conducts binary classification experiments on the valence and arousal dimensions of the DEAP and MAHNOB-HCI data sets to evaluate the performance of the proposed model. The results show that the model proposed in this paper can achieve accuracies of 84.71% and 89.00% on the two corresponding data sets, indicating that the model proposed in this paper is superior to other deep learning emotion classification models in feature extraction and fusion.
Overview
CoChat Summary
TL;DR
The paper proposes a hierarchical fusion convolutional neural network (HFCNN) for multimodal emotion recognition from EEG and peripheral physiological signals, using weighted fusion of multiscale convolutional features plus manual statistical features. It reports binary valence/arousal classification on DEAP and MAHNOB-HCI, with the best accuracies of 84.71% and 89.00%, respectively.
Key findings
- HFCNN achieved average accuracies on DEAP of 83.28% for arousal and 84.71% for valence; on MAHNOB-HCI, 88.28% for arousal and 89.00% for valence (Table 2).
- HFCNN outperformed CNN-alone feature extraction under optimal RF settings: DEAP arousal 83.28% vs 75.98%, DEAP valence 84.71% vs 79.52%; MAHNOB-HCI arousal 88.28% vs 70.17%, MAHNOB-HCI valence 89.00% vs 71.38% (Table 2).
- In comparison with prior methods on DEAP, HFCNN’s 84.71% exceeded SAE+LSTM (81.10%), multi-hypergraph neural networks (82.95%), CapsNet (68.28%), and hierarchical bidirectional GRU (67.90%) (Table 3).
- Valence performance was better than arousal on the same dataset, and MAHNOB-HCI accuracy was higher than DEAP accuracy when classifier parameters were optimal.
Methods
- The model uses observation-level fusion of EEG and peripheral physiological signals (PPS), then a hierarchical incremental CNN with three convolutional layers: 8 filters of size 7×7, 12 filters of size 5×5, and 16 filters of size 3×3; max pooling is 2×2 with stride 2. Features MPs-1, MPs-2, and MPs-3 are extracted after pooling, weighted, and combined into MPs-global, then fused with manually extracted statistical features. A random forest (RF) classifier is used with ten-fold cross-validation for binary valence and arousal classification.
- Data: DEAP included 32 subjects watching 40 videos; MAHNOB-HCI included 25 subjects watching 20 videos after excluding 3 subjects with missing data and 2 with incomplete records from the original 30. Inputs used six forehead EEG channels (FP1, FP2, AF3, AF4, F3, F4) and four PPS signals (GSR, RESP, TEMP, PLET). The middle 30 s of each video was used, with 6-s windows and 50% overlap applied to five frequency bands.
- Manual features included Hjorth parameters in the time domain, mean power spectral density and mean difference entropy after Fourier transform for EEG, and mean, variance, maximum, minimum, and max-min differences of PPS signals and their first- and second-order differences. RF decision-tree parameters were fixed at 450 and 200 for DEAP arousal and valence, and 600 and 500 for MAHNOB-HCI arousal and valence.
Limitations
- Stated: the authors call for further research on extracting more discriminative multimodal features for cross-subject emotion classification, selecting/constructing/optimizing deep learning models with higher accuracy, robustness, and generalization, incorporating emotion-related brain neurogenic analysis, and reducing sensory-induced interference.
- Evident from the text: MAHNOB-HCI results are based on only 25 of 30 subjects because of missing or incomplete data records, and class imbalance was handled by expanding the smaller categories, which may affect dataset representativeness.
- Evident from the text: evaluation is limited to two datasets (DEAP and MAHNOB-HCI) and binary valence/arousal tasks, so broader generalization is not demonstrated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 4 Not checked · 25 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, each layer of features extracted by the proposed HFCNN was tested separately, and the results proved that only the weighted feature fusion method that fuses the features of the three incremental network structures can achieve good experimental results.”
- Also stated in: § B. MULTIMODAL EMOTION RECOGNITION RESULT · plain
“each layer of features extracted by the proposed HFCNN was tested separately, and the results proved that only the weighted feature fusion method that fuses the features of the three incremental network structures can achieve good experimental results.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § B. MULTIMODAL EMOTION RECOGNITION RESULT · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“in the DEAP data set, the average accuracy of arousal is 83.28% and valence is 84.71%”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that the model proposed in this paper can achieve accuracies of 84.71% and 89.00% on the two corresponding data sets, indicating that the model proposed in this paper is superior to other deep learning emotion classi cation models in feature extraction and fusion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the MAHNOB-HCI data set, the accuracy of arousal is 88.28% and valence is 89.00%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can be concluded that using the same data set in the parameter optimization case, the performance of valence is better than that of arousal”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
11 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 9
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ V. CONCLUSION
“We presented a subject-independent emotion recognition system that is suitable for real-time operation”
§ V. CONCLUSION
“Finally, a 10-fold cross-validation data set was created for each subject, and the HFCNN model was used to perform binary emotion classi cation experiments in the valence and arousal dimensions.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ V. CONCLUSION
“Then, the dataset was balanced to ensure that the number of the two classes was equal to improve the normalization ability of the classi cation model.”
§ A. DATA PREPROCESSING AND OBSERVATION-LEVEL FUSION OF MULTIMODAL SIGNALS
“| HA | 33930 | HV | 32580 | HA | 11475 | HV | 12015 | | LA | 33670 | LV | 32020 | LA | 11025 | LV | 11985 |”
Citations
Cites
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
EEG-Based Emotion Recognition Using Regularized Graph Neural Networks
Zhong, P., Wang, D., & Miao, C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2994159 · Cited by 910
Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion
Amin, S. U., Alsulaiman, M., Muhammad, G., Mekhtiche, M. A., & Shamim Hossain, M. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.06.027 · Cited by 523
Domain Adaptation for EEG Emotion Recognition Based on Latent Representation Similarity
Li, J., Qiu, S., Du, C., Wang, Y., & He, H. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2019.2949306 · Cited by 291
Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E., & Arunkumar, N. (2019) · IEEE Access · doi:10.1109/access.2018.2883213 · Cited by 289 · cited 1×
“[21] used a CNN to automatically extract features from a variety of physiological signals and perform sentiment prediction through fully connected network layers.”
MethodEmotion Recognition from Multiband EEG Signals Using CapsNet
Chao, H., Dong, L., Liu, Y., & Lu, B. (2019) · Sensors · doi:10.3390/s19092212 · Cited by 287 · cited 1×
“[31] extracted multiband features from multichannel EEG signals to constitute the multiband feature matrix (MFM), and entered the capsule network to mine three related features.”
MethodSAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG
Xing, X., Li, Z., Xu, T., Shu, L., Hu, B., & Xu, X. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00037 · Cited by 269
A machine learning model for emotion recognition from physiological signals
Domínguez-Jiménez, J. A., Campo-Landines, K. C., Martínez-Santos, J. C., Delahoz, E. J., & Contreras-Ortiz, S. H. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.101646 · Cited by 232
Emotion Recognition using Multimodal Residual LSTM Network
Ma, J., Tang, H., Zheng, W.-L., & Lu, B.-L. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350871 · Cited by 230 · cited 1×
“…results show that automatic feature extraction performs better than manual feature extraction; and various deep learning technologies [10], including autoen-coders (AEs), convolutional neural networks (CNNs) [11], and recurrent neural networks (RNNs) [12], are widely used in different domains.”
BackgroundXie, S., & Hu, H. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2844085 · Cited by 214
Amin, S. U., Alsulaiman, M., Muhammad, G., Bencherif, M. A., & Hossain, M. S. (2019) · IEEE Access · doi:10.1109/access.2019.2895688 · Cited by 209
Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020) · IEEE Access · doi:10.1109/access.2020.3023871 · Cited by 200
A Hierarchical Bidirectional GRU Model With Attention for EEG-Based Emotion Classification
Chen, J. X., Jiang, D. M., & Zhang, Y. N. (2019) · IEEE Access · doi:10.1109/access.2019.2936817 · Cited by 194
Self-citationEEG-based emotion recognition using simple recurrent units network and ensemble learning
Wei, C., Chen, L.-L., Song, Z.-Z., Lou, X.-G., & Li, D.-D. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.101756 · Cited by 192
Jia, Z., Lin, Y., Cai, X., Chen, H., Gou, H., & Wang, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413724 · Cited by 167
Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Huang, Y., Yang, J., Liu, S., & Pan, J. (2019) · Future Internet · doi:10.3390/fi11050105 · Cited by 124 · cited 2×
“The expression of emotions includes not only some physiological responses, such as skin temperature and heart rate, but also some nonphysiological responses, such as facial expressions and body language [1].”
BackgroundExpression-EEG Based Collaborative Multimodal Emotion Recognition Using Deep AutoEncoder
Zhang, H. (2020) · IEEE Access · doi:10.1109/access.2020.3021994 · Cited by 122
Chen, D.-W., Miao, R., Yang, W.-Q., Liang, Y., Chen, H.-H., Huang, L., Deng, C.-J., & Han, N. (2019) · Sensors · doi:10.3390/s19071631 · Cited by 104 · cited 2×
“[8] proposed a multichannel EEG feature extraction method that combines differential entropy and linear discriminant analysis, and the method achieved an accuracy rate of 82.”
BackgroundMethodMultimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition
Ghaleb, E., Popa, M., & Asteriadis, S. (2019) · 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2019.8925444 · Cited by 93
Wu, M., Su, W., Chen, L., Liu, Z., Cao, W., & Hirota, K. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2019.2897330 · Cited by 90
Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video
Wang, Z., Zhou, X., Wang, W., & Chen, L. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-019-01056-8 · Cited by 79
Dynamic entropy-based pattern learning to identify emotions from EEG signals across individuals
Lu, Y., Wang, M., Wu, W., Han, Y., Zhang, Q., & Chen, S. (2020) · Measurement · doi:10.1016/j.measurement.2019.107003 · Cited by 73 · cited 1×
“Lu et al. [6] proposed a pattern learning framework based on dynamic entropy.”
BackgroundMultimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks
Guo, J.-J., Zhou, R., Zhao, L.-M., & Lu, B.-L. (2019) · 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8856563 · Cited by 73
Multimodal Emotion Recognition Based on Ensemble Convolutional Neural Network
Huang, H., Hu, Z., Wang, W., & Wu, M. (2020) · IEEE Access · doi:10.1109/access.2019.2962085 · Cited by 66
A Multiscale Deep Middle-level Feature Fusion Network for Hyperspectral Classification
Li, Z., Huang, L., & He, J. (2019) · Remote Sensing · doi:10.3390/rs11060695 · Cited by 55
Wu, D., Zhang, J., & Zhao, Q. (2020) · IEEE Access · doi:10.1109/access.2020.3010311 · Cited by 51 · cited 1×
“Feature extraction and fusion are the key steps in multimodal emotion recognition [3].”
BackgroundMultimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Choi, D. Y., Kim, D.-H., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3036877 · Cited by 39 · cited 1×
“Each video clip was about 34 to 117 seconds [30].”
BackgroundEmotion Recognition from Physiological Signals using Multi-Hypergraph Neural Networks
Zhu, J., Zhao, X., Hu, H., & Gao, Y. (2019) · 2019 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00111 · Cited by 23 · cited 1×
“Zhu et al. [28] used the multi-hypergraph neural network to identify the emotional status from physiological signals.”
BackgroundInvestigating Sex Differences in Classification of Five Emotions from EEG and Eye Movement Signals
Bao, L.-Q., Qiu, J.-L., Tang, H., Zheng, W.-L., & Lu, B.-L. (2019) · 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8857476 · Cited by 22
Design of a hierarchy modular neural network and its application in multimodal emotion recognition
Li, W., Chu, M., & Qiao, J. (2019) · Soft Computing · doi:10.1007/s00500-018-03735-0 · Cited by 13
Cross-Subject EEG Signal Classification with Deep Neural Networks Applied to Motor Imagery
Riyad, M., Khalil, M., & Adib, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-22885-9_12 · Cited by 12
Cited by
Recognition of human emotions using EEG signals: A review
Rahman, M. M., Sarkar, A. K., Hossain, M. A., Hossain, M. S., Islam, M. R., Hossain, M. B., Quinn, J. M. W., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104696 · Cited by 227
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Deep learning-based EEG emotion recognition: Current trends and future perspectives
Wang, X., Ren, Y., Luo, Z., He, W., Hong, J. S., & Huang, Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1126994 · Cited by 219
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 167
Metadata
- Authors
- Zhang, Y., Cheng, C., & Zhang, Y.
- Affiliations
- Liaoning Normal University (CN)
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 7943-7951
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3120929420
- Semantic Scholar
- 96cb58a7255129284cae50002651fc01b7b74080
- Cited by
- 84
- References
- 32
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 27 · 2023: 20 · 2022: 11 · 2021: 3
- FWCI
- 14.74
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61772252)
- Natural Science Foundation of Liaoning Province of China (2019-MS-216)
- Program for Liaoning Innovative Talents in University (LR2017044)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, EEG, feature-level fusion, multi-scale feature extraction, valence and arousal classification, DEAP dataset, automatic feature extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.