The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 2 of the 6 sources its claims cite were checked for support.
Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Open accessAbstract
In recent years, deep learning has been increasingly used in the field of multimodal emotion recognition in conjunction with electroencephalogram. Considering the complexity of recording electroencephalogram signals, some researchers have applied deep learning to find new features for emotion recognition. In previous studies, convolutional neural network model was used to automatically extract features and complete emotion recognition, and certain results were obtained. However, the extraction of hierarchical features with convolutional neural network for multimodal emotion recognition remains unexplored. Therefore, this paper proposes a hierarchical fusion convolutional neural network model to mine the potential information in the data by constructing different network hierarchical structures, extracting multiscale features, and using feature-level fusion to fuse the global features formed by combining weights with manually extracted statistical features to form the final feature vector. This paper conducts binary classification experiments on the valence and arousal dimensions of the DEAP and MAHNOB-HCI data sets to evaluate the performance of the proposed model. The results show that the model proposed in this paper can achieve accuracies of 84.71% and 89.00% on the two corresponding data sets, indicating that the model proposed in this paper is superior to other deep learning emotion classification models in feature extraction and fusion.
Overview
CoChat Summary
TL;DR
The paper proposes a hierarchical fusion convolutional neural network (HFCNN) for multimodal emotion recognition from EEG and peripheral physiological signals, using weighted fusion of multiscale convolutional features plus manual statistical features. It reports binary valence/arousal classification on DEAP and MAHNOB-HCI, with the best accuracies of 84.71% and 89.00%, respectively.
Key findings
- HFCNN achieved average accuracies on DEAP of 83.28% for arousal and 84.71% for valence; on MAHNOB-HCI, 88.28% for arousal and 89.00% for valence (Table 2).
- HFCNN outperformed CNN-alone feature extraction under optimal RF settings: DEAP arousal 83.28% vs 75.98%, DEAP valence 84.71% vs 79.52%; MAHNOB-HCI arousal 88.28% vs 70.17%, MAHNOB-HCI valence 89.00% vs 71.38% (Table 2).
- In comparison with prior methods on DEAP, HFCNN’s 84.71% exceeded SAE+LSTM (81.10%), multi-hypergraph neural networks (82.95%), CapsNet (68.28%), and hierarchical bidirectional GRU (67.90%) (Table 3).
- Valence performance was better than arousal on the same dataset, and MAHNOB-HCI accuracy was higher than DEAP accuracy when classifier parameters were optimal.
Methods
- The model uses observation-level fusion of EEG and peripheral physiological signals (PPS), then a hierarchical incremental CNN with three convolutional layers: 8 filters of size 7×7, 12 filters of size 5×5, and 16 filters of size 3×3; max pooling is 2×2 with stride 2. Features MPs-1, MPs-2, and MPs-3 are extracted after pooling, weighted, and combined into MPs-global, then fused with manually extracted statistical features. A random forest (RF) classifier is used with ten-fold cross-validation for binary valence and arousal classification.
- Data: DEAP included 32 subjects watching 40 videos; MAHNOB-HCI included 25 subjects watching 20 videos after excluding 3 subjects with missing data and 2 with incomplete records from the original 30. Inputs used six forehead EEG channels (FP1, FP2, AF3, AF4, F3, F4) and four PPS signals (GSR, RESP, TEMP, PLET). The middle 30 s of each video was used, with 6-s windows and 50% overlap applied to five frequency bands.
- Manual features included Hjorth parameters in the time domain, mean power spectral density and mean difference entropy after Fourier transform for EEG, and mean, variance, maximum, minimum, and max-min differences of PPS signals and their first- and second-order differences. RF decision-tree parameters were fixed at 450 and 200 for DEAP arousal and valence, and 600 and 500 for MAHNOB-HCI arousal and valence.
Limitations
- Stated: the authors call for further research on extracting more discriminative multimodal features for cross-subject emotion classification, selecting/constructing/optimizing deep learning models with higher accuracy, robustness, and generalization, incorporating emotion-related brain neurogenic analysis, and reducing sensory-induced interference.
- Evident from the text: MAHNOB-HCI results are based on only 25 of 30 subjects because of missing or incomplete data records, and class imbalance was handled by expanding the smaller categories, which may affect dataset representativeness.
- Evident from the text: evaluation is limited to two datasets (DEAP and MAHNOB-HCI) and binary valence/arousal tasks, so broader generalization is not demonstrated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 4 Not checked · 25 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, each layer of features extracted by the proposed HFCNN was tested separately, and the results proved that only the weighted feature fusion method that fuses the features of the three incremental network structures can achieve good experimental results.”
- Also stated in: § B. MULTIMODAL EMOTION RECOGNITION RESULT · plain
“each layer of features extracted by the proposed HFCNN was tested separately, and the results proved that only the weighted feature fusion method that fuses the features of the three incremental network structures can achieve good experimental results.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § B. MULTIMODAL EMOTION RECOGNITION RESULT · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“in the DEAP data set, the average accuracy of arousal is 83.28% and valence is 84.71%”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that the model proposed in this paper can achieve accuracies of 84.71% and 89.00% on the two corresponding data sets, indicating that the model proposed in this paper is superior to other deep learning emotion classi cation models in feature extraction and fusion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the MAHNOB-HCI data set, the accuracy of arousal is 88.28% and valence is 89.00%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can be concluded that using the same data set in the parameter optimization case, the performance of valence is better than that of arousal”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
11 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 9
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ V. CONCLUSION
“We presented a subject-independent emotion recognition system that is suitable for real-time operation”
§ V. CONCLUSION
“Finally, a 10-fold cross-validation data set was created for each subject, and the HFCNN model was used to perform binary emotion classi cation experiments in the valence and arousal dimensions.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ V. CONCLUSION
“Then, the dataset was balanced to ensure that the number of the two classes was equal to improve the normalization ability of the classi cation model.”
§ A. DATA PREPROCESSING AND OBSERVATION-LEVEL FUSION OF MULTIMODAL SIGNALS
“| HA | 33930 | HV | 32580 | HA | 11475 | HV | 12015 | | LA | 33670 | LV | 32020 | LA | 11025 | LV | 11985 |”
Citations
Cites
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
EEG-Based Emotion Recognition Using Regularized Graph Neural Networks
Zhong, P., Wang, D., & Miao, C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2994159 · Cited by 910
Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion
Amin, S. U., Alsulaiman, M., Muhammad, G., Mekhtiche, M. A., & Shamim Hossain, M. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.06.027 · Cited by 523
Domain Adaptation for EEG Emotion Recognition Based on Latent Representation Similarity
Li, J., Qiu, S., Du, C., Wang, Y., & He, H. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2019.2949306 · Cited by 291
Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E., & Arunkumar, N. (2019) · IEEE Access · doi:10.1109/access.2018.2883213 · Cited by 289
Emotion Recognition from Multiband EEG Signals Using CapsNet
Chao, H., Dong, L., Liu, Y., & Lu, B. (2019) · Sensors · doi:10.3390/s19092212 · Cited by 287
SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG
Xing, X., Li, Z., Xu, T., Shu, L., Hu, B., & Xu, X. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00037 · Cited by 269
A machine learning model for emotion recognition from physiological signals
Domínguez-Jiménez, J. A., Campo-Landines, K. C., Martínez-Santos, J. C., Delahoz, E. J., & Contreras-Ortiz, S. H. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.101646 · Cited by 232
Emotion Recognition using Multimodal Residual LSTM Network
Ma, J., Tang, H., Zheng, W.-L., & Lu, B.-L. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350871 · Cited by 230
Cited by
Recognition of human emotions using EEG signals: A review
Rahman, M. M., Sarkar, A. K., Hossain, M. A., Hossain, M. S., Islam, M. R., Hossain, M. B., Quinn, J. M. W., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104696 · Cited by 227
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Deep learning-based EEG emotion recognition: Current trends and future perspectives
Wang, X., Ren, Y., Luo, Z., He, W., Hong, J. S., & Huang, Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1126994 · Cited by 216
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 166
Spatial-Temporal Feature Fusion Neural Network for EEG-Based Emotion Recognition
Wang, Z., Wang, Y., Zhang, J., Hu, C., Yin, Z., & Song, Y. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3165280 · Cited by 113
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
An attention-based hybrid deep learning model for EEG emotion recognition
Zhang, Y., Zhang, Y., & Wang, S. (2022) · Signal Image and Video Processing · doi:10.1007/s11760-022-02447-1 · Cited by 63
Self-citationZhang, Y., Cheng, C., Wang, S., & Xia, T. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103877 · Cited by 60
Self-citationEEG-based emotion analysis using non-linear features and ensemble learning approaches
Rahman, M. M., Sarkar, A. K., Hossain, M. A., & Moni, M. A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118025 · Cited by 62
Gong, L., Chen, W., Li, M., & Zhang, T. (2023) · Digital Signal Processing · doi:10.1016/j.dsp.2023.104278 · Cited by 55
Singh, P., Tripathi, M. K., Patil, M. B., Shivendra, & Neelakantappa, M. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19171-2 · Cited by 38
Dense Graph Convolutional With Joint Cross-Attention Network for Multimodal Emotion Recognition
Cheng, C., Liu, W., Feng, L., & Jia, Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3412074 · Cited by 41
Self-citationEmotion Recognition System Based on Two-Level Ensemble of Deep-Convolutional Neural Network Models
Hussain, M., Qazi, E.-U.-H., Aboalsamh, H., & Ullah, I. (2023) · IEEE Access · doi:10.1109/access.2023.3245830 · Cited by 31
An Efficient Machine Learning-Based Emotional Valence Recognition Approach Towards Wearable EEG
Abdel‐Hamid, L. (2023) · Sensors · doi:10.3390/s23031255 · Cited by 36
Gahlan, N., & Sethia, D. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19467-3 · Cited by 39
Improving Emotion Recognition Systems by Exploiting the Spatial Information of EEG Sensors
Gagliardi, G., Alfeo, A. L., Catrambone, V., Candia‐Rivera, D., Cimino, M. G. C. A., & Valenza, G. (2023) · IEEE Access · doi:10.1109/access.2023.3268233 · Cited by 19
Emotion Fusion-Sense (Emo Fu-Sense) – A novel multimodal emotion classification technique
Umair, M., Rashid, N., Khan, U. S., Hamza, A., & Iqbal, J. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106224 · Cited by 20
Emotion recognition based on a limited number of multimodal physiological signals channels
Wan, C., Xu, C., Chen, D., Wei, D., & Li, X. (2024) · Measurement · doi:10.1016/j.measurement.2024.115940 · Cited by 21
A novel 3D feature fusion network for EEG emotion recognition
(2024) · doi:10.1016/j.bspc.2024.107347 · Cited by 17
Nandini, D., Yadav, J., Rani, A., & Singh, V. (2024) · Iranian Journal of Science and Technology Transactions of Electrical Engineering · doi:10.1007/s40998-024-00710-4 · Cited by 17
Xefteris, V.-R., Tsanousa, A., Georgakopoulou, N., Diplaris, S., Vrochidis, S., & Kompatsiaris, I. Y. (2022) · Sensors · doi:10.3390/s22218198 · Cited by 22
Multimodal Emotion Classification Method and Analysis of Brain Functional Connectivity Networks
Sun, X., Zheng, X., Li, T., Li, Y., & Cui, L. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3192533 · Cited by 17
Reward-Penalty Weighted Ensemble for Emotion State Classification from Multi-Modal Data Streams
Nandi, A., Xhafa, F., Subirats, L., & Fort, S. (2022) · International Journal of Neural Systems · doi:10.1142/s0129065722500496 · Cited by 16
MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals
Zhu, L., Ding, Y., Huang, A., Tan, X., & Zhang, J. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03632-0 · Cited by 18
Ruchilekha, Singh, M. K., & Singh, M. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104928 · Cited by 18
Soft Voting Strategy for Multi-Modal Emotion Recognition Using Deep-learning- Facial Images and EEG
Chinta, U., Kalita, J. K., & Atyabi, A. (2023) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc57344.2023.10099070 · Cited by 14
State-of-the-art Multimodal Emotion Recognition: A comprehensive survey and taxonomy
Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., & Tyler, B. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200642 · Cited by 11
García-Moreno, F. M., Badenes-Sastre, M., Expósito, F., Rodríguez‐Fórtiz, M. J., & Bermúdez-Edo, M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109463 · Cited by 16
Kapase·, A. B., & Uke, N. (2025) · International Journal of Speech Technology · doi:10.1007/s10772-025-10202-3 · Cited by 11
IoT-based approach to multimodal music emotion recognition
Zhao, H., & Jin, L. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.10.059 · Cited by 14
Emotion recognition with hybrid attentional multimodal fusion framework using cognitive augmentation
Kulkarni, S. V., Khot, S. S., & Angal, Y. (2025) · International Journal of Information Technology · doi:10.1007/s41870-025-02402-6 · Cited by 10
Real-Time Multimodal Emotion Classification System in E-Learning Context
Nandi, A., Xhafa, F., Subirats, L., & Fort, S. (2021) · Proceedings of the international neural networks society · doi:10.1007/978-3-030-80568-5_35 · Cited by 15
Zhang, D., Shafiq, M., Tang, K., & Naseem, U. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3318419 · Cited by 8
EEG-based Multimodal Emotion Recognition: Recent Progress, Challenges, and Future Directions
Muhammad, G., Almuntasheri, S. A., Alenezi, F., Alhadi, N., & Leung, V. C. M. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3774428 · Cited by 14
Semisupervised Deep Features of Time‐Frequency Maps for Multimodal Emotion Recognition
Zali-Vargahan, B., Charmin, A., Kalbkhani, H., & Barghandan, S. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/3608115 · Cited by 9
Wu, Q., Zhu, M., Xu, W., Wang, J., Mao, Z., Gao, Q., & Song, Y. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3400341 · Cited by 6
Multimodal fusion approach for sports injury prevention and pose keypoint detection
Ji, C., Zhong, Y., & Gao, M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327911 · Cited by 6
Image-Evoked Emotion Recognition for Hearing-Impaired Subjects with EEG Signals
Zhu, M., Jin, H., Bai, Z., Li, Z., & Song, Y. (2023) · Sensors · doi:10.3390/s23125461 · Cited by 5
Emotion Recognition from Physiological Signals Using Continuous Wavelet Transform and Deep Learning
Jalal, L., & Peer, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17618-0_8 · Cited by 8
Building Privacy-Preserving Medical Text Models With a Pretrained Transformer
Shafiq, M., Ren, L., Srivastava, G., Zhang, D., Bourouis, S., & Gadekallu, T. R. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3520426 · Cited by 7
A Two-Stage Multi-Modal Multi-Label Emotion Recognition Decision System Based on GCN
Wu, W., Chen, D., & Li, Q. (2024) · International Journal of Decision Support System Technology · doi:10.4018/ijdsst.352398 · Cited by 6
Xavier, S. P., & John, S. P. (2024) · International Conference on Advancement in Renewable Energy and Intelligent Systems (AREIS) · doi:10.1109/areis62559.2024.10893614 · Cited by 4
EEG-based multimodal emotion recognition with optimal trained hybrid classifier
Chakravarthy, G. K., Suchithra, M., & Thatavarti, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17489-x · Cited by 4
Enhancing emotion recognition through brain asymmetry and multi-angle fusion network
Zhou, B., Lin, L., & Chen, J. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107324 · Cited by 5
Advances In EEG-Based Multimodal Emotion Recognition: A Comprehensive Review
Pillalamarri, R., & Shanmugam, U. (2023) · Innovations in Power and Advanced Computing Technologies (i-PACT) · doi:10.1109/i-pact58649.2023.10434480 · Cited by 3
Parihar, N., Kanan, Chawla, R., & Fortino, G. (2026) · Discover Robotics · doi:10.1007/s44430-026-00028-2 · Cited by 5
Wen, Y., & Chen, W. (2025) · Measurement · doi:10.1016/j.measurement.2025.119247 · Cited by 2
A survey on encoding and decoding technology of non-invasive brain-computer interface
Qiu, S., Yang, B., Chen, X., Wang, Y., Xu, M., Lyu, B., Gao, X., & He, H. (2023) · Journal of Image and Graphics · doi:10.11834/jig.230031 · Cited by 6
Saisanthiya, D., & Supraja, P. (2023) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v17i16.42735 · Cited by 3
Correlation-Driven Multi-Modality Graph Decomposition for Cross-Subject Emotion Recognition
Huang, W., Chen, Y., Jiang, X., Gao, C., Chen, Q., Zhang, T., Yan, B., Wang, Y., & Yang, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681579 · Cited by 5
Multimodal Sentiment Analysis and Multimodal Emotion Analysis: A Review
Sharma, S., Sharma, S., & Gupta, D. K. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-0892-5_29 · Cited by 3
A Survey on Multi-modal Emotion Detection TechniquesPreprint
Chatterjee, C., Shah, N., Bhatt, S., & Chandi, S. (2024) · Research Square · doi:10.21203/rs.3.rs-3942784/v1 · Cited by 1
Empirical study of emotional state recognition using multimodal fusion framework
Ruchilekha, & Singh, M. K. (2023) · International Conference on Image, Video and Signal Processing (IVSP) · doi:10.1145/3591156.3591160 · Cited by 1
MS-TimesNet: A temporal–spatial multimodal fusion network for emotion recognition
Ding, L., Yu, R., Guo, S., et al. (2026) · Measurement and Control · doi:10.1177/00202940261419905 · Cited by 1
Multimodal Emotion Recognition Using Computer Vision: A Comprehensive Approach
Mohan, G. B., Sarrvesh, N., Hrishikeasan, N. P., Boopalamani, J., & Doss, S. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10725638 · Cited by 2
EEG-based Emotion Classification - A Theoretical Perusal of Deep Learning Methods
Sana Parveen, K., Panachakel, J. T., Ranjana, H., Sidharth, S., & Samuel, A. A. (2023) · International Conference for Innovation in Technology (INOCON) · doi:10.1109/inocon57975.2023.10101002 · Cited by 3
A robot electronic device for multimodal emotional recognition of expressions
Nie, L. (2024) · Paladyn Journal of Behavioral Robotics · doi:10.1515/pjbr-2022-0127 · Cited by 1
Mohiddin, S. K., Sharmila, S., & Slimani, K. (2026) · Information systems engineering and management · doi:10.1007/978-3-032-14778-3_10 · Cited by 1
Heterogeneous Graph Convolutional Neural Networks for EEG-fNIRS Bimodal Emotion Recognition
Zhao, T., Xue, Y., Zhu, J., & Zheng, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888828 · Cited by 2
Khanra, M., & Roy, S. S. (2026) · Frontiers in Robotics and AI · doi:10.3389/frobt.2026.1810478 · Cited by 3
Hegh, A. N., Adeyelu, A. A., Iorliam, A., & Otor, S. U. (2025) · FUDMA Journal of Sciences · doi:10.33003/fjs-2025-0905-3412 · Cited by 1
Umair, M., Rashid, N., Khan, U. S., Hamza, A., Zeb, A., Nawaz, T. H., & Ansari, A. R. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.106601 · Cited by 1
Grafting Heterogeneous Neural Networks for a Hierarchical Object Classification
Heller, G., Perrin, E., Vrabie, V. D., Dusart, C., & Roux, S. L. (2022) · IEEE Access · doi:10.1109/access.2022.3144579 · Cited by 1
Exploring key technologies for multimodal emotion recognition: Research and application analysis
Gao, J. (2024) · AIP conference proceedings · doi:10.1063/5.0223702 · Cited by 2
Abimala, T., Narmadha, T. V., & Raamesh, L. (2022) · Algorithms for intelligent systems · doi:10.1007/978-981-19-2126-1_21
Multiscale Feature Extraction Techniques for Emotion Recognition from Physiological Signal
Taley, S. G., & Pund, M. A. (2024) · International Conference on Advances in Computing, Communication and Networking (ICAC2N) · doi:10.1109/icac2n63387.2024.10895032
Decoupled Multimodal Fusion Network Based on Peripheral Physiological Signals
Fan, T., Qiu, S., Wang, Y., Wang, Z., Zhao, H., Jiang, J., Xu, J., Sun, T., & Tian, F. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3668039
Multimodal Emotion Detection and Classification of Audio-Visual Using Deep Learning
Srivastava, M., Singh, P., & Sharma, M. K. (2025) · International Conference on Advanced Computing and Emerging Technologies (ACET) · doi:10.1109/acet67282.2025.11430197
A Comprehensive Review on Enhancing Emotional Understanding Through Multimodal Fusion
Boby, M., Singh, Y., & Nikita (2026) · International Conference on Communication, Computing and Electronics Systems (ICCCES) · doi:10.1109/iccces62661.2026.11437288
Detection of Characteristic Waves in Clinical Visual Evoked Potential Signals Based on Deep LearningPreprint
Chen, Y., Shen, M., Lu, D., Lin, J. P., Liu, Y., Wang, S., Yu, C., Li, M., Zhang, Y., Yao, W., et al. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4608217
Self-citationMultimodal Data Mining Trends, Achievements, and Challenges: Systematic Literature Review
Gradinaru, B. C. P., Danubianu, M., & Barila, A. (2025) · IEEE Access · doi:10.1109/access.2025.3590921 · Cited by 1
Zhang, S., Zheng, X., Zhang, M., Guo, G., & Ji, C. (2021) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-92635-9_36
Fine-grained Dual-space Context Analysis Network for Multimodal Emotion Recognition
Li, J., Tang, J., Kong, W., & Ying, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651517
Bharaneedharan, M., & Kavya, G. (2025) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126625503335
Multimodal Emotion Recognition from Audio and Visual Data
Bhanuprakash, E., Phanish, G., Adapa, S. R., George, J., & Bellamkonda, S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-0924-6_34
Metadata
- Authors
- Zhang, Y., Cheng, C., & Zhang, Y.
- Affiliations
- Liaoning Normal University (CN)
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 7943-7951
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3120929420
- Semantic Scholar
- 96cb58a7255129284cae50002651fc01b7b74080
- Cited by
- 84
- References
- 32
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 27 · 2023: 20 · 2022: 11 · 2021: 3
- FWCI
- 14.74
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61772252)
- Natural Science Foundation of Liaoning Province of China (2019-MS-216)
- Program for Liaoning Innovative Talents in University (LR2017044)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, EEG, feature-level fusion, multi-scale feature extraction, valence and arousal classification, DEAP dataset, automatic feature extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.