The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 5 of the 12 sources its claims cite were checked for support.
Multimodal Emotion Recognition From EEG Signals and Facial Expressions
Open accessAbstract
Emotion recognition has attracted a lot of attention in recent years. It is widely used in health care, teaching, human-computer interaction, and other fields. Various human emotional features are often used to recognize different emotions. Currently, there is more and more research on multimodal emotion recognition based on the fusion of multiple features. This paper proposes a deep learning model of multimodal emotion recognition based on the fusion of electroencephalogram (EEG) signals and facial expressions to achieve an excellent classification effect. First, the pre-trained convolution neural network (CNN) is used to extract the facial features from the facial expressions. Next, the attention mechanism is introduced to extract more critical facial frame features. Then, we apply CNNs to extract spatial features from original EEG signals, which use a local convolution kernel and a global convolution kernel to learn the features of left and right hemispheres channels and all EEG channels. After feature-level fusion, the fusion features of the facial expression features and EEG features are fed into the classifier for emotion recognition. This paper conducted experiments on the DEAP and MAHNOB-HCI datasets to evaluate the performance of the proposed model. The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset. The experimental results show that the proposed model can effectively carry out emotion recognition.
Overview
CoChat Summary
TL;DR
This paper proposes a deep learning model that fuses EEG signals and facial expressions for multimodal emotion recognition, using a pre-trained CNN with attention for facial features and CNNs with local/global kernels for EEG spatial features. It reports high binary valence/arousal classification accuracy on DEAP and MAHNOB-HCI, with fusion outperforming either modality alone.
Key findings
- On DEAP, the proposed model achieved 96.63% accuracy for valence and 97.15% for arousal in binary classification.
- On MAHNOB-HCI, it achieved 96.69% accuracy for valence and 96.26% for arousal.
- Ablation results show multimodal fusion outperformed single modalities: DEAP valence was 93.59% for facial expressions, 85.72% for EEG, and 96.63% for fusion; DEAP arousal was 94.03%, 87.97%, and 97.15%, respectively. MAHNOB-HCI valence was 93.34%, 85.98%, and 96.69%; arousal was 93.06%, 85.23%, and 96.26%.
- The paper reports that its model outperformed several existing approaches on DEAP and MAHNOB-HCI, such as Huang et al. (DEAP valence 80.30%, arousal 74.23%), Zhu et al. (70.01%, 77.08%), Li et al. (71.00%, 58.75%), and Koelstra et al. (74.00%, 70.00%).
Methods
- The model extracts facial features using a pre-trained CNN (DeepVANet, pre-trained on AFEW-VA) as an encoder, then applies an attention mechanism to weight important facial frames; input samples are T × 3 × 64 × 64 with T = 5 frames per second.
- EEG spatial features are extracted with CNNs using a local convolution kernel for left/right brain channels and a global convolution kernel for all EEG channels; in DEAP and MAHNOB-HCI, c = 32 channels, h = 64, and K = 20 kernels.
- Facial and EEG features are fused at the feature level and fed into a fully connected classifier. Experiments use DEAP (32 subjects, 40 trials each, 2400 samples per subject) and MAHNOB-HCI (25 participants used, 20 trials each, 1611 samples per subject), with 5-fold cross-validation; training used PyTorch, batch size 128, learning rate 0.00001, Adam optimizer, and binary cross entropy.
Limitations
- The paper does not include an explicit limitations section; its conclusion states future work to explore a more reliable pre-training model to reduce resources and time, and to introduce more modalities such as non-physiological signals.
- Evident from the text: MAHNOB-HCI experiments used only 25 of 30 participants because three recording files were corrupted and two participants’ data were incomplete.
- Evident from the text: the tasks are binary high/low valence and arousal classifications, not continuous or multi-class emotion recognition; also, DEAP facial expressions were recorded for the first 22 subjects, while the experiments state both datasets use data of all subjects, suggesting a possible sample mismatch.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 0 Not valid · 4 Not checked · 16 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
What the sources say
- Not checked — no full text of the sourcecites [10] source
- Supportedcites [11] source
The Introduction says recent research uses multimodal data to improve emotion-classification performance and that multiple sources provide complementary emotion-related information. The Multimodal Data Fusion section states that multimodal inputs improve recognition accuracy over single-modality inputs by providing additional information. (LLM, conf=0.93)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Through the above optimization methods, the model proposed in this paper outperforms other similar studies mentioned above in the classification tasks of DEAP and MAHNOB-HCI datasets.”
What the sources say
- Could not verifycites [29] sourceExperimental — accuracy not yet measured
The cited paper reports 2D face-alignment comparisons on 300-VW, 300-W and Menpo (Section 5, p. 5), not classification results on DEAP or MAHNOB-HCI. Searches for those datasets and classification tasks found no passage addressing the citing paper’s claim about its own model. (LLM, conf=0.78)
- Could not verifycites [30] sourceExperimental — accuracy not yet measured
The Discussion (p. 11) describes EEGLAB as a toolbox for EEG processing, while the comparison on p. 12 concerns ICA algorithms, not classification models. Searches for DEAP, MAHNOB-HCI, and classification performance found no passage assessing the citing paper’s proposed model or its claimed outperformance. (LLM, conf=0.78)
- Supportedcites [31] source
Section 3.2, “Comparison with Other Literature” (page not provided), compares the method with earlier studies on DEAP and MAHNOB-HCI and states that its performance surpassed previous results for both valence and arousal classification. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Could not verify
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
What the sources say
- Not checked — no full text of the sourcecites [10] source
- Supportedcites [11] source
The Introduction says data from multiple sources provide complementary emotion-related information. Section B, “Multimodal Data Fusion for Emotion Classification,” explains that features from each modality can be combined and that the additional information improves emotion-recognition accuracy. This supports the attributed claim; page numbers are not provided in our copy. (LLM, conf=0.93)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
9 more claims
Create a free account to see all 23 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 8
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The proposed study demonstrated a marginally higher performance, achieving an accuracy of 97.80%, compared to Wang et al.'s [23] 96.89% using the DEAP dataset by combining EEG and facial expressions.”
From Multimodal Emotion Recognition: Emotion Classification Through the Integration of EEG and Facial Expressions (2025) · cites this paper as [23]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“the number of samples per subject in the DEAP dataset is 2400”
Reported 2400 · recomputed 1200
Citations
Cites
Delorme, A., & Makeig, S. (2004) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2003.10.009 · Cited by 24,371 · cited 1×
“For EEG data, the sampling rate is reduced to 128Hz, and the band-pass filter in the toolbox of EEGLAB [30] is used to filter and remove artifacts.”
MethodAn argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“In the dimensional model, the emotions are mapped into the valence, arousal, and The associate editor coordinating the review of this manuscript and approving it for publication was Seifedine Kadry . dominance dimensions [2].”
BackgroundCore affect and the psychological construction of emotion.
Russell, J. A. (2003) · Psychological Review · doi:10.1037/0033-295x.110.1.145 · Cited by 6,113 · cited 2×
“Currently, the most commonly used dimensional model is the valence-arousal model proposed by Russell [3], which only includes valence and arousal.”
BackgroundDEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 1×
“This paper uses two standard datasets: DEAP [26] and MAHNOB-HCI [27] datasets to verify the model performance.”
MethodUniversals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · cited 1×
“This paper uses two standard datasets: DEAP [26] and MAHNOB-HCI [27] datasets to verify the model performance.”
MethodBulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464
Bulat, A., & Tzimiropoulos, G. (2017) · Nottingham ePrints (University of Nottingham) · Cited by 1,277
Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers
Atkinson, J., & Campos, D. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.10.049 · Cited by 506 · cited 1×
“Traditional emotion recognition often uses machine learning methods, such as support vector machine (SVM), decision tree, and k -nearest neighbor (KNN) to classify [4], [5].”
MethodAFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236 · cited 1×
“The AFEW-VA database [25] contains 600 videos selected from movies showing various facial expressions.”
Background
Cited by
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 156
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 141
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
A survey on state-of-the-art deep learning applications and challenges
Noor, M. H. M., & Ige, A. O. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111225 · Cited by 106
Metadata
- Authors
- Wang, S., Qu, J., Zhang, Y., & Zhang, Y.
- Affiliations
- Liaoning Normal University (CN)
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 33061-33068
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4361987556
- Semantic Scholar
- 00cf5823372d502b5d81f35177757c06201d458d
- Cited by
- 124
- References
- 36
- Citations by year
- 2026: 29 · 2025: 56 · 2024: 32 · 2023: 7
- FWCI
- 29.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61772252)
- Natural Science Foundation of Liaoning Province of China (2019-MS-216)
- Scientific Research Foundation of the Education Department of Liaoning Province (LJKZ0965)
- Huzhou Science and Technology Plan Project (2022GZ08)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- multimodal emotion recognition, EEG signals, facial expressions, feature-level fusion, attention mechanism, convolutional neural networks, valence and arousal classification, DEAP dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Emotion Recognition From EEG Signals and Facial Expressions. https://app.cochat.ai/papers/10.1109/access.2023.3263670
- The paper
- Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023). Multimodal Emotion Recognition From EEG Signals and Facial Expressions. IEEE Access. https://doi.org/10.1109/access.2023.3263670
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.