CoChat

Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network

Open access
Zhang, Yong, Cheng, Cheng, Zhang, Yidie · 2021 · IEEE Access

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 2 of the 6 sources its claims cite were checked for support.

Abstract

In recent years, deep learning has been increasingly used in the field of multimodal emotion recognition in conjunction with electroencephalogram. Considering the complexity of recording electroencephalogram signals, some researchers have applied deep learning to find new features for emotion recognition. In previous studies, convolutional neural network model was used to automatically extract features and complete emotion recognition, and certain results were obtained. However, the extraction of hierarchical features with convolutional neural network for multimodal emotion recognition remains unexplored. Therefore, this paper proposes a hierarchical fusion convolutional neural network model to mine the potential information in the data by constructing different network hierarchical structures, extracting multiscale features, and using feature-level fusion to fuse the global features formed by combining weights with manually extracted statistical features to form the final feature vector. This paper conducts binary classification experiments on the valence and arousal dimensions of the DEAP and MAHNOB-HCI data sets to evaluate the performance of the proposed model. The results show that the model proposed in this paper can achieve accuracies of 84.71% and 89.00% on the two corresponding data sets, indicating that the model proposed in this paper is superior to other deep learning emotion classification models in feature extraction and fusion.

Overview

CoChat Summary

TL;DR

The paper proposes a hierarchical fusion convolutional neural network (HFCNN) for multimodal emotion recognition from EEG and peripheral physiological signals, using weighted fusion of multiscale convolutional features plus manual statistical features. It reports binary valence/arousal classification on DEAP and MAHNOB-HCI, with the best accuracies of 84.71% and 89.00%, respectively.

Key findings

  • HFCNN achieved average accuracies on DEAP of 83.28% for arousal and 84.71% for valence; on MAHNOB-HCI, 88.28% for arousal and 89.00% for valence (Table 2).
  • HFCNN outperformed CNN-alone feature extraction under optimal RF settings: DEAP arousal 83.28% vs 75.98%, DEAP valence 84.71% vs 79.52%; MAHNOB-HCI arousal 88.28% vs 70.17%, MAHNOB-HCI valence 89.00% vs 71.38% (Table 2).
  • In comparison with prior methods on DEAP, HFCNN’s 84.71% exceeded SAE+LSTM (81.10%), multi-hypergraph neural networks (82.95%), CapsNet (68.28%), and hierarchical bidirectional GRU (67.90%) (Table 3).
  • Valence performance was better than arousal on the same dataset, and MAHNOB-HCI accuracy was higher than DEAP accuracy when classifier parameters were optimal.

Methods

  • The model uses observation-level fusion of EEG and peripheral physiological signals (PPS), then a hierarchical incremental CNN with three convolutional layers: 8 filters of size 7×7, 12 filters of size 5×5, and 16 filters of size 3×3; max pooling is 2×2 with stride 2. Features MPs-1, MPs-2, and MPs-3 are extracted after pooling, weighted, and combined into MPs-global, then fused with manually extracted statistical features. A random forest (RF) classifier is used with ten-fold cross-validation for binary valence and arousal classification.
  • Data: DEAP included 32 subjects watching 40 videos; MAHNOB-HCI included 25 subjects watching 20 videos after excluding 3 subjects with missing data and 2 with incomplete records from the original 30. Inputs used six forehead EEG channels (FP1, FP2, AF3, AF4, F3, F4) and four PPS signals (GSR, RESP, TEMP, PLET). The middle 30 s of each video was used, with 6-s windows and 50% overlap applied to five frequency bands.
  • Manual features included Hjorth parameters in the time domain, mean power spectral density and mean difference entropy after Fourier transform for EEG, and mean, variance, maximum, minimum, and max-min differences of PPS signals and their first- and second-order differences. RF decision-tree parameters were fixed at 450 and 200 for DEAP arousal and valence, and 600 and 500 for MAHNOB-HCI arousal and valence.

Limitations

  • Stated: the authors call for further research on extracting more discriminative multimodal features for cross-subject emotion classification, selecting/constructing/optimizing deep learning models with higher accuracy, robustness, and generalization, incorporating emotion-related brain neurogenic analysis, and reducing sensory-induced interference.
  • Evident from the text: MAHNOB-HCI results are based on only 25 of 30 subjects because of missing or incomplete data records, and class imbalance was handled by expanding the smaller categories, which may affect dataset representativeness.
  • Evident from the text: evaluation is limited to two datasets (DEAP and MAHNOB-HCI) and binary valence/arousal tasks, so broader generalization is not demonstrated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2021.3049516

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 0 Not valid · 4 Not checked · 25 Own results and interpretations

11 more claims

Create a free account to see all 31 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 10
  • It may state two separate claims as one. 9

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 32 references

Cited by

See all 85

Metadata

Authors
Zhang, Y., Cheng, C., & Zhang, Y.
Affiliations
Liaoning Normal University (CN)
Venue
IEEE Access
Published
2021
Volume
9
Pages
7943-7951
ISSN
2169-3536 (ISSN-L)
OpenAlex
W3120929420
Cited by
84
References
32
Citations by year
2026: 7 · 2025: 16 · 2024: 27 · 2023: 20 · 2022: 11 · 2021: 3
FWCI
14.74
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Natural Science Foundation of China (61772252)
  • Natural Science Foundation of Liaoning Province of China (2019-MS-216)
  • Program for Liaoning Innovative Talents in University (LR2017044)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
Keywords
multimodal emotion recognition, EEG, feature-level fusion, multi-scale feature extraction, valence and arousal classification, DEAP dataset, automatic feature extraction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network. https://app.cochat.ai/papers/10.1109/access.2021.3049516
The paper
Zhang, Y., Cheng, C., & Zhang, Y. (2021). Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network. IEEE Access. https://doi.org/10.1109/access.2021.3049516

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.