The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 3 of the 10 sources its claims cite were checked for support.
Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Open accessAbstract
Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.
Overview
CoChat Summary
TL;DR
The paper proposes Deep-Emotion, a multimodal emotion recognition framework that combines facial expressions, speech, and EEG through decision-level fusion. It reports strong results on CK+, EMO-DB, and MAHNOB-HCI, while noting that fusion did not significantly outperform facial-expression or EEG modes on MAHNOB-HCI.
Key findings
- The improved GhostNet achieved 98.27% average accuracy on CK+ under tenfold cross-validation, outperforming traditional GhostNet at 90.21%; fear expression accuracy was only 95%, possibly because fear and contempt have similar features.
- The LFCNN for speech achieved 94.36% average accuracy and 94.38% F1-score on EMO-DB, with a model size of only 2.28 MB.
- On MAHNOB-HCI, the tLSTM obtained advanced results in EEG emotion recognition, and the fusion method improved both arousal and valence dimensions; fusion was significantly higher than the speech mode (\(p<0.05\), paired \(t\) test) but not significantly better than facial expressions or EEG (\(p>0.05\)).
- The paper claims this is the first attempt to combine facial expressions, speech, and EEG for multimodal emotion recognition.
Methods
- Deep-Emotion has three branches: an improved GhostNet for facial expression recognition, a lightweight fully convolutional neural network (LFCNN) for speech emotion recognition, and a tree-like LSTM (tLSTM) for EEG emotion recognition.
- Decision-level fusion uses an optimal weight distribution search algorithm: weights are enumerated from 0.00 to 1.00 with step 0.01, and RMSE is minimized to select the best weight distribution.
- Experiments use CK+, EMO-DB, and MAHNOB-HCI; CK+ used tenfold cross-validation. Data preprocessing, full LFCNN structure, experiment setup, and algorithm pseudo-code are stated to be in the Supplementary Materials.
Limitations
- The paper states that on MAHNOB-HCI the fusion method was only significantly higher than the speech mode, not facial expressions or EEG; it suggests this may be because subjects behaved normally rather than mimicking facial expressions.
- The paper states that SER on MAHNOB-HCI is challenging because the speech signal includes both the subject’s voice and the stimulus material’s voice, possibly explaining the relatively average SER recognition rate.
- The paper states that the work is based on open-source datasets and has not conducted independent data collection; it also notes future work should explore dynamic weight allocation. The text makes evident that many methodological details are relegated to Supplementary Materials, limiting full standalone reproducibility from the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
2 Valid · 6 Not valid · 4 Not checked · 34 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“EEG signals extracted from the central nervous system can more accurately and objectively reflect changes in people's emotions than other signals [2], [11].”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Introduction
What the sources say
- Not checked — no full text of the sourcecites [2] source
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 2.2 says EEG records central nervous system activity and is an effective measure for studying emotion. The introduction (p. 1:2) notes that facial and speech cues can be affected when people conceal their feelings, but it does not establish that EEG reflects emotional changes more accurately and objectively than other signals generally. (LLM, conf=0.84)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“emotions are considered a complex representation that cannot be reliably captured with unimodal signals, since genuine emotion can be hidden by different facial expressions or tones [11].”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Introduction
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 5.3 says people may conceal feelings behind social masks and that a single modality may be insufficient for accurate recognition; the introduction (1:2) says deliberate concealment can affect facial- or speech-based methods. The claim overstates this as a categorical inability to capture emotion reliably with any unimodal signal, whereas the review discusses limitations that may arise. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The size of our proposed model is only 2.28 MB”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Results and Discussion > Results on EMO-DB
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network.”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“outperforming the traditional GhostNet (90.21”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Results and Discussion > Results on CK+
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 46 claims and every source verdict.
- It may state two separate claims as one. 6
- The quote only partly states this claim. 2
- Probably not a substantial claim. 9
Citations
Cites
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
GhostNet: More Features From Cheap Operations
Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00165 · Cited by 5,222
GhostNet: More Features From Cheap OperationsPreprint
Han, K., Wang, Y.-H., Tian, Q., Guo, J.-Y., Xu, C.-J., & Xu, C. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.11907 · Cited by 4,540
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Russell, J. A. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.3.345 · Cited by 857 · cited 1×
“In contrast, the most commonly used multi-dimensional paradigm is the arousal-valence 2D model proposed by Russell [7], arousal and valence are levels of excitement and positivity, respectively, and this definition method is conducive to our quantitative research on emotions.”
MethodEmotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
EEG Based Emotion Recognition: A Tutorial and Review
Li, X., Zhang, Y., Tiwari, P., Song, D., Hu, B., Yang, M., Zhao, Z., Kumar, N., & Marttinen, P. (2023) · ACM Computing Surveys · doi:10.1145/3524499 · Cited by 508
3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition
Chen, M., He, X., Yang, J., & Zhang, H. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2860246 · Cited by 503
Deep learning-based facial emotion recognition for human–computer interaction applications
Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2023) · Neural Computing and Applications · doi:10.1007/s00521-021-06012-8 · Cited by 416
Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM
Mustaqeem, Sajjad, M., & Kwon, S. (2020) · IEEE Access · doi:10.1109/access.2020.2990405 · Cited by 402
Cited by
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
A Review of Key Technologies for Emotion Analysis Using Multimodal Information
Zhu, X., Guo, C., Feng, H., Huang, Y., Feng, Y., Wang, X., & Wang, R. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10287-z · Cited by 193
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 166
Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features
Mamieva, D., Abdusalomov, A. B., Kutlimuratov, A., Muminov, B. B., & Whangbo, T. K. (2023) · Sensors · doi:10.3390/s23125475 · Cited by 144
Convergence of artificial intelligence with social media: A bibliometric & qualitative analysis
Saheb, T., Sidaoui, M., & Schmarzo, B. (2024) · Telematics and Informatics Reports · doi:10.1016/j.teler.2024.100146 · Cited by 105
Halkiopoulos, C., Gkintoni, E., Aroutzidis, A., & Antonopoulou, H. (2025) · Diagnostics · doi:10.3390/diagnostics15040456 · Cited by 119
Advances in EEG-based emotion recognition: Challenges, methodologies, and future directions
Chen, J., Cui, Y., Wei, C., Polat, K., & Alenezi, F. S. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113478 · Cited by 101
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
A survey on state-of-the-art deep learning applications and challenges
Noor, M. H. M., & Ige, A. O. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111225 · Cited by 120
Ali, M., Ali, S., Abbas, Q., Abbas, Z., & Lee, S. (2025) · Digital Health · doi:10.1177/20552076251395548 · Cited by 109
“58 Wearable technology integration.”
Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures
Makhmudov, F., Kultimuratov, A., & Cho, Y. I. (2024) · Applied Sciences · doi:10.3390/app14104199 · Cited by 73
EEG-based affective brain–computer interfaces: recent advancements and future challenges
Chen, Y., Peng, Y. K., Tang, J., Camilleri, T. A., Camilleri, K. P., Kong, W., & Cichocki, A. S. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/ade290 · Cited by 49
Hierarchical convolutional neural networks with post-attention for speech emotion recognition
Fan, Y., Huang, H., & Han, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128879 · Cited by 23
Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition
Jin, M., & Li, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612074 · Cited by 22
“Liu et al. [24] proposed Deep-Emotion to adaptively integrate the most discriminative features from facial expression, speech and EEG.”
BackgroundFang, L., Xing, S. P., Ma, Z., Zhang, Z., Long, Y., Lee, K.-P., & Wang, S. J. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2228983 · Cited by 24
“Pan et al. (2023) used an LSTM model to extract features from EEGs, facial expressions, and speeches.”
Self-citationEmotion recognition via affective EEG signals: State of the art
Meng, W., Hou, F., Zhao, M., Kong, J., Wu, J., Zuo, J., & Liu, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130418 · Cited by 22
Facial Emotion Recognition for Mobile Devices: A Practical Review
Krumnikl, M., & Maiwald, V. (2024) · IEEE Access · doi:10.1109/access.2024.3358455 · Cited by 15
“The design of this network is suitable for multimodal emotion recognition [67].”
Multimodal Daily-Life Emotional Recognition Using Heart Rate and Speech Data From Wearables
Moon, E., Sagar, A. S. M. S., & Kim, H. S. (2024) · IEEE Access · doi:10.1109/access.2024.3427111 · Cited by 14
“Pan et al. similarly integrated facial, speech, and EEG data for improved emotion recognition [17].”
BackgroundEvidential dissonance measure in robust multi-view classification to resist adversarial attack
Yue, X., Dong, Z., Chen, Y., & Xie, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102605 · Cited by 14
Sedehi, J. F., Dabanloo, N. J., Maghooli, K., & Sheikhani, A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36411 · Cited by 17
“[31], proposed a deep learning-based multimodal emotion recognition (MER) framework called Deep-Emotion, which integrates features from facial expressions, speech, and EEG to enhance MER performance.”
BackgroundShen, J., Liang, H., Ma, R., Xu, Y., Zhu, K., Gao, H., Hou, K., Zhang, Y., Zhang, X., & Hu, B. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104224 · Cited by 25
Multimodal Emotion Recognition Using Speech and Facial Dynamics Transformer-Based Architecture
(2025) · doi:10.1109/icicnct66124.2025.11232705 · Cited by 10
“Although unimodal methods based primarily on speech or facial expressions have been examined by researchers, they typically do not encompass the breadth of human affective communication [1].”
Kapase·, A. B., & Uke, N. (2025) · International Journal of Speech Technology · doi:10.1007/s10772-025-10202-3 · Cited by 11
Sarikaya, M. A., & İnce, G. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2649 · Cited by 11
Hamid, M. A., & Singh, J. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10606-0 · Cited by 9
Kalsum, T., & Mehmood, Z. (2023) · Journal of Sensors · doi:10.1155/2023/6987708 · Cited by 13
Hassan, C. A. U., Ehatisham-ul-Haq, M., Murtaza, F., Yasin, A. U., & Ullah, S. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-98404-2 · Cited by 10
“Contrarily 55 , proposed a Spectral Adversarial MixUp method for improving EEG emotion recognition through domain adaptation.”
A novel signal channel attention network for multi-modal emotion recognition
Du, Z., Xia, Y., & Zhao, P. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1442080 · Cited by 13
“In recent studies, features of human facial data, speech data, and EEG signals have been fused at the decision level through three branches, and very significant breakthroughs have been made in utilizing such data for multi-modal emotion recognition tasks (Pan et al., 2023).”
BackgroundReDDIT: Regret detection and domain identification from text
Balouchzahi, F., Butt, S., Sidorov, G., & Gelbukh, A. F. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120099 · Cited by 19
Gupta, C., Gill, N. S., Gulia, P., Kumar, A., Karamti, H., Moges, D. M., & Safra, I. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01300-9 · Cited by 17
“Hence, multi-modal modeling has become a critical research model for human emotion recognition [3].”
EEG-based Multimodal Emotion Recognition: Recent Progress, Challenges, and Future Directions
Muhammad, G., Almuntasheri, S. A., Alenezi, F., Alhadi, N., & Leung, V. C. M. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3774428 · Cited by 14
EEG-based fusion approaches in multimodal emotion recognition: An in-depth review
Yılmaz, B. H., Yılmaz, Ç. M., & Köse, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132235 · Cited by 13
Enhancing emotion recognition through multi-modal data fusion and graph neural networks
Devarajan, K., Ponnan, S., & Perumal, S. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100291 · Cited by 11
Exploring Emotions in EEG: Deep Learning Approach with Feature Fusion
Mridula, D. T., Ferdaus, A. A., & Pias, T. S. (2023) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit60459.2023.10441204 · Cited by 8
Torcate, A. S., de Santana, M. A., & dos Santos, W. P. (2023) · IEEE Latin American Conference on Computational Intelligence (LA-CCI) · doi:10.1109/la-cci58595.2023.10409408 · Cited by 5
“Facial expressions and speech are the most dominant external channels for transmitting emotions, since these two modalities account for about 93% of emotional information in human communication [4].”
BackgroundChen, M., Liu, J., Li, X., Zhang, Y., Liu, H., Jiao, J., & He, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3569089 · Cited by 11
“The deep emotion framework proposed by Pan et al. [35] consists of three branches: face, speech, and EEG.”
Sreehari, P., Raghavendra, U., & Gudigar, A. (2025) · F1000Research · doi:10.12688/f1000research.171170.1 · Cited by 13
“Pan et al. 352 propose Deep-Emotion, a multi-branch architecture that fuses EEG, facial, and speech inputs using GhostNet, Lightweight Full CNN (LFCNN), and tree-like LSTM (tLSTM), respectively.”
Khan, M. U., Hirachan, N., Joseph, C., Murtagh, L., Chetty, G., Goecke, R., & Rojas, R. F. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108532 · Cited by 6
Pan, J., Chen, Y., Xiao, Q., Chen, Z., Cai, H., You, Q., Qiu, L., & Xie, Q. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2024.3442788 · Cited by 5
Self-citationA Review of Advancements in Facial Emotion Recognition and Detection Using Deep Learning
Harika, R., Uday, T. Y., Sirisha, M. L., Sahitya, M. S. L., Drugaanjali, K., & Srinivas, M. S. (2024) · International Conference on Social and Sustainable Innovations in Technology and Engineering (SASI-ITE) · doi:10.1109/sasi-ite58663.2024.00062 · Cited by 5
Torcate, A. S., de Santana, M. A., & dos Santos, W. P. (2024) · Research on Biomedical Engineering · doi:10.1007/s42600-024-00363-6 · Cited by 4
“Among these channels, facial expressions and speech prevail as the primary conduits through which emotional information is conveyed, accounting for approximately 93% of emotional content in human communication (Pan et al. 2023).”
Enhancing Emergency Response Through Speech Emotion Recognition: A Machine Learning Approach
Deb, P., Mahrin, H., & Bhuiyan, A. R. (2023) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit60459.2023.10440971 · Cited by 4
“Some recent studies, in order to create emotion recognition models for facial expressions, speech, and EEG, propose a new MER approach called Deep-Emotion, based on deep learning techniques [6].”
BackgroundA Cross-Modal Adaptive Masked Autoencoder for Decoding Emotions With Multimodal Data
Cheng, C., Liu, W., Zhang, Y., Feng, L., & Jia, Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3415613 · Cited by 7
“Pan et al. [25] proposed a deep-emotion framework that adaptively integrates discriminative features from facial expressions, speech, and EEG, showcasing the power of combined modalities.”
Recent Trends in Machine and Deep Learning for Verbal and Non-verbal Emotion Detection
Chawla, M., Panda, S. N., Khullar, V., Kansal, I., & Kumar, R. (2024) · Recent Advances in Electrical & Electronic Engineering (Formerly Recent Patents on Electrical & Electronic Engineering) · doi:10.2174/0123520965303142240430101645 · Cited by 3
MKDF: Knowledge Distillation Based Lightweight Multimodal Framework for Emotion Recognition
Wang, Y., Sun, Y., Qiao, B., Wang, C., Li, Z., & Yang, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_6 · Cited by 1
Kim, T., Moon, E., Kang, H., & Kim, H. S. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11368-2 · Cited by 2
“Multimodal ER emulates the predictive abilities of combinations of human biological and behavioral features, providing an advantage in situations where accurately detecting emotions through a single method (unimodal) is challenging [36, 37].”
Human Emotion Quantification using Facial Expression and Deep Learning
Hamid, M. A., & Singh, J. (2025) · World Conference on Communication & Computing (WCONF) · doi:10.1109/wconf64849.2025.11233451 · Cited by 1
AI in Emergency Responses: Distress & Lie Detection Using Audio & EEG
(2024) · doi:10.1109/icses63760.2024.10910378 · Cited by 1
“In line with previous work on EEG branches, a tree-like LSTM (tLSTM) model is proposed for emotion feature extraction in EEGs [36].”
Exploring Emotions in EEG: Deep Learning Approach with Feature FusionPreprint
Mridula, D. T., Ferdaus, A. A., & Pias, T. S. (2023) · medRxiv · doi:10.1101/2023.11.17.23298680 · Cited by 1
“At first, we explore the SJTU SEED-V emotion dataset.”
MethodChawla, M., Panda, S. N., & Khullar, V. (2025) · Proceedings of the Institution of Mechanical Engineers Part H Journal of Engineering in Medicine · doi:10.1177/09544119251325331 · Cited by 1
“52,53 The input gate enables the addition of new information to the memory cell, the output gate regulates the output of information from the memory cell, and the forget gate enables the network to discard obsolete information.”
Sreenivasarao, D., & Saheb, S. K. (2026) · Iran Journal of Computer Science · doi:10.1007/s42044-025-00365-2 · Cited by 1
“Identify emotions to provide in the assignment, without trying to identify them in living life [4–6].”
AI for Learners’ Emotions — A Perspective Approach of Analysis During Online Assessments
Sharon, S. J. S., Chitra, R. M. S., & Santhiya, C. (2025) · Wiley eBooks · doi:10.1002/9781394302666.ch12 · Cited by 1
EEG-Based Emotion Dynamics Recognition Using Hybrid AI Models for Cybersecurity
Pleshakova, E., Osipov, A. V., Yudin, A., & Gataullin, S. T. (2026) · Technologies · doi:10.3390/technologies14040209 · Cited by 4
Zhang, Y., Zhang, H., Wang, Y., Gao, X., & Yang, C. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104281 · Cited by 1
A Survey on Multi-modal Emotion Detection TechniquesPreprint
Chatterjee, C., Shah, N., Bhatt, S., & Chandi, S. (2024) · Research Square · doi:10.21203/rs.3.rs-3942784/v1 · Cited by 1
(2026) · doi:10.1109/tcss.2026.3653634 · Cited by 1
“Similarly, Liu et al. [47] designed a selective convolutional network (SCN), and Huang et al. [48] proposed a motion-aware CNN (MACNN), both showing apex-based superiority.”
Bharaneedharan, M., Kavya, G., Annalakshmi, T., & Bapu, B. R. T. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04736-x · Cited by 1
“Pan et al. [12 ] introduced DeepEmotion with a tree-structured LSTM (EEG), light CNN (audio), and GhostNet (facial features).”
EEG-Based Emotion Recognition Using Spatial-Temporal Graph-Aware Network With Channel Selection
(2026) · doi:10.1109/jbhi.2026.3665517 · Cited by 1
“Besides EEG, other physiological signals like facial expressions and speech [22] can also aid emotion recognition.”
Audiovisual integration of faces and voices in Chinese individuals with social anxiety
Yuan, J., Liu, X., Chen, X., Li, M., Liu, Z., & Li, B. (2025) · Behavioural Brain Research · doi:10.1016/j.bbr.2025.115905 · Cited by 1
FPGA-Based Automated EEG Analysis for Efficient Emotion Detection
(2024) · doi:10.1109/icdcece60827.2024.10548113 · Cited by 1
“In this work, to provide a novel method for automated FPGA-based EEG-based emotion recognition [3].”
MethodKaur, D., Misra, A., & Vyas, O. P. (2025) · doi:10.1109/icscsa66339.2025.11170723
“Human-machine interaction has spread across a variety of environments where systems need to interpret user commands and anticipate behaviour [7].”
Bastidas-Benalcazar, N., Santos-Solís, D., Cueva-Argudo, D., Lechón-Santacruz, E., Almeida-Galárraga, D. A., Navas, P., Mora, H. C., Orozco-Garzón, N., & Tirado-Espín, A. (2026) · Technologies · doi:10.3390/technologies14090523
AI-powered Non-contact Emotion Recognition: Recent progress, Advances, and Future prospects *
(2025) · doi:10.1109/icia64617.2025.11277829
“A multi-modal emotion detection model is constructed based on an in-depth understanding and analysis of physiological signals, speech, and facial expressions [18].”
Decoding Cognitive States and Emotions Using the Electroencephalogram
Ullah, I., Zafar, R., Dilpazir, H., Rehman, M. J. U., Waqas, A., & Ahmad, R. F. (2025) · International Journal of Innovations in Science and Technology · doi:10.33411/ijist/1652
An efficient multimodal deep learning model for emotion recognition in social media videos
Sangeetha, J., & Anu, V. M. (2026) · Discover Computing · doi:10.1007/s10791-026-09914-0
TDGN: A text-guided dual-gated network for multimodal sentiment analysis
Xiao, W., Zhang, L., & Xu, Y. (2026) · PLoS ONE · doi:10.1371/journal.pone.0349024 · Cited by 1
Zhu, K., Zhao, Q., Wang, H., & Zhao, Y. (2026) · User Modeling and User-Adapted Interaction · doi:10.21203/rs.3.rs-9400088/v1
Modelling Human-Like Emotion Perception Using Multimodal Deep LearningPreprint
Shanker, A., & Pandey, S. C. (2026) · Research Square · doi:10.21203/rs.3.rs-9572322/v1
“The amount of road traffic accidents has been steadily increasing to 1.35 million [1].”
BackgroundInfluentialRetractedSmart Emotion Recognition for Personalized Mental Health Support
Charaan, R. M. D., Rahul, B., Reddy, M. P., et al. (2026) · International Conference on Image Processing and Capsule Networks (ICIPCN) · doi:10.1109/icipcn67432.2026.11438798
“They find empirical results from their investigation to support the assertion that EEG functions to enhance recognizability, and is particularly useful in ambiguous cases for either facial or speech signals [14].”
Falahzadeh, M. R., Vakili, Y. Z., Harimi, A., Farsa, E. Z., Ahmadi, A., & Abraham, A. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20956-2 · Cited by 1
Swain, M. R., Nayak, R. K., Ghosh, K., Jain, R., Banthia, M., Bisen, M., & Sharma, P. (2026) · International Conference on Connected Intelligence for Industrial Applications (CI2A) · doi:10.1109/ci2a69097.2026.11576896
Deep Learning-Driven Emotion Recognition for Personalized and Adaptive E-Learning Systems
Khasanova, S., qizi, O. N. O., Aarthi, S., Ravikumar, R. N., Urozboev, K., & Usmonov, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-6581-7.ch006
Denoise and Attend: An AE-Transformer Approach to EEG Emotion Recognition
Luo, M. (2025) · International Conference on Electrical Engineering and Computer Technology (IC2ECT) · doi:10.1109/ic2ect66838.2025.11291002
“Beyond single-modality EEG, multimodal systems fusing EEG with facial, speech, and peripheral signals demonstrate complementary gains [10–15].”
AIGC-driven virtual patients for diabetes: a systematic review
Tang, Y., Wen, G., & Wu, H. (2026) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2026.113261
(2025) · doi:10.1109/ic3it66137.2025.11341646
“Fusion strategies that integrate text, speech, and visual inputs continue to depend on internal representation modalities whose reliability remains a challenge, particularly in dynamic or noisy environments [9].”
A multimodal meta-learning-augmented EmoTriSense for emotion recognition
Sowmya, V., & Rajeswari, A. (2026) · Journal of Combinatorial Optimization · doi:10.1007/s10878-026-01402-x · Cited by 2
Mind Your Mind: Real-Time Emotional Insights from Voice
Manoharan, H., Keerthana, R. E., Selvaganesh, N., & Logeswari, P. (2025) · doi:10.38124/ijisrt/25may894
Multimodal Sentiment Analysis via Modal Experts and Missing-Modal Prompt Generation
Liu, S., & Wu, X. (2026) · IEEE Conference on Cybernetics and Intelligent Systems · doi:10.1109/cis-ram71614.2026.11668537
(2025) · doi:10.1371/journal.pone.0332130
“Multimodal approaches improve robustness by integrating facial expressions, speech, and physiological signals [14].”
Micro-expression recognition method based on improved GhostNet combined with attention mechanism
(2025) · doi:10.1109/bdee67464.2025.00025
“The research team performed ten-fold cross-validation on the CASME II dataset using the improved GhostNet V1 method proposed by Pan et al. [10] and the improved GhostNet V2 combined with the attention mechanism.”
Sharma, S., Dhingra, A., Bhanot, N., & Kharb, S. (2025) · OPJU International Technology Conference on Smart Computing for Innovation and Advancement in Industry (OTCON) · doi:10.1109/otcon65728.2025.11071221
AI- Powered Emotion Recognition: A Multimodal Approach to Analyzing Human Sentiment
Shalma, H., Mukherjee, S., Jainulabudeen, S. A. K., Reddy, B. S. T., Budhati, B., & Krishna, N. S. (2025) · International Conference on Visual Analytics and Data Visualization (ICVADV) · doi:10.1109/icvadv63329.2025.10961939
GhostBlock-Augmented Lightweight Gaze Tracking via Depthwise Separable Convolution
Guo, J.-M., Cheng, Y.-S., Zeng, Y., & Yang, Z.-Y. (2025) · Electronics · doi:10.3390/electronics14152978
“…designing a modality-aware attention mechanism and a proxy-based multimodal loss function to address class bias in data fusion; Pan J et al. [5] fused facial expression, speech, and EEG features to construct a multi-dimensional emotion recognition framework; Ueda L et al. [6] optimized…”
Metadata
- Authors
- Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S.
- Affiliations
- South China Normal University (CN)
- University of Leicester (GB)
- Corresponding author
- Chen, Bingzhi
- Venue
- IEEE Open Journal of Engineering in Medicine and Biology
- Published
- 2024
- Volume
- 5
- Pages
- 396-403
- ISSN
- 2644-1276 (ISSN-L)
- PMC
- PMC11186647
- OpenAlex
- W4320002631
- Semantic Scholar
- cbdc02c9cfa1d14a88447691f7b511712350b010
- Cited by
- 121
- References
- 35
- Citations by year
- 2026: 35 · 2025: 48 · 2024: 25
- FWCI
- 31.22
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Open Journal of Engineering in Medicine and Biology (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- STI 2030-Major (2022ZD0208900)
- National Natural Science Foundation of China (62076103, 62271217)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, facial expression recognition, speech emotion recognition, EEG-based emotion recognition, GhostNet, lightweight convolutional neural network, decision fusion, real-time emotion recognition, human-machine interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.