Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Abstract
Background and objective Using deep-learning methodologies to analyze multimodal physiological signals becomes increasingly attractive for recognizing human emotions. However, the conventional deep emotion classifiers may suffer from the drawback of the lack of the expertise for determining model structure and the oversimplification of combining multimodal feature abstractions. Methods In this study, a multiple-fusion-layer based ensemble classifier of stacked autoencoder (MESAE) is proposed for recognizing emotions, in which the deep structure is identified based on a physiological-data-driven approach. Each SAE consists of three hidden layers to filter the unwanted noise in the physiological features and derives the stable feature representations. An additional deep model is used to achieve the SAE ensembles. The physiological features are split into several subsets according to different feature extraction approaches with each subset separately encoded by a SAE. The derived SAE abstractions are combined according to the physiological modality to create six sets of encodings, which are then fed to a three-layer, adjacent-graph-based network for feature fusion. The fused features are used to recognize binary arousal or valence states. Results DEAP multimodal database was employed to validate the performance of the MESAE. By comparing with the best existing emotion classifier, the mean of classification rate and F-score improves by 5.26%. Conclusions The superiority of the MESAE against the state-of-the-art shallow and deep emotion classifiers has been demonstrated under different sizes of the available physiological instances.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Yin et al. [171] used stacked autoencoders to learn high-level representations from various peripheral sensors including skin temperature and blood volume pressure, improving the state-of-the-art by 5%.
“Yin et al. [171] used stacked autoencoders (SAEs) to learn high-level representations from various peripheral sensors including skin temperature and blood volume pressure, improving the state-of-the-art by 5%.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [171]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · see citing works
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · see citing works
Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering
Belkin, M., & Niyogi, P. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0080 · Cited by 4,534 · see citing works
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797 · see citing works
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · see citing works
Dimensionality Reduction: A Comparative Review
van der Maaten, L., Postma, E. O., & van den Herik, H. J. (2009) · Cited by 2,066 · see citing works
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · see citing works
Borghini, G., Laura Astolfi, A., Vecchiato, G., Donatella Mattia, A., & Babiloni, F. (2012) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2012.10.003 · Cited by 1,346 · see citing works
Emotion recognition based on physiological changes in music listening
Kim, J., & André, E. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.26 · Cited by 1,079 · see citing works
Breaking the silence: Brain-computer interfaces (BCI) for communication and motor control
Birbaumer, N. (2006) · Psychophysiology · doi:10.1111/j.1469-8986.2006.00456.x · Cited by 722 · see citing works
Affective video content representation and modeling
Hanjalić, A., & Xu, L.-Q. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2004.840618 · Cited by 593 · see citing works
Affective neuroscience and psychophysiology: Toward a synthesis
Davidson, R. J. (2003) · Psychophysiology · doi:10.1111/1469-8986.00067 · Cited by 535 · see citing works
Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers
Atkinson, J., & Campos, D. (2015) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.10.049 · Cited by 510 · see citing works
Jirayucharoensak, S., Pan-ngum, S., & Israsena, P. (2014) · The Scientific World JOURNAL · doi:10.1155/2014/627892 · Cited by 486 · see citing works
Ecg pattern analysis for emotion detection
Agrafioti, F., Hatzinakos, D., & Anderson, A. K. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.28 · Cited by 476 · see citing works
Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty
Chanel, G., Rebetez, C., Bétrancourt, M., & Pun, T. (2011) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2011.2116000 · Cited by 401 · see citing works
EEG-based emotion recognition during watching movies
Nie, D., Wang, X., Shi, L.-C., & Lu, B. (2011) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner.2011.5910636 · Cited by 375 · see citing works
Verma, G. K., & Tiwary, U. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.007 · Cited by 354 · see citing works
Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach
Christie, I. C., & Friedman, B. H. (2004) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2003.08.002 · Cited by 347 · see citing works
Frantzidιs, C. A., Bratsas, C., Papadelis, C. L., Konstantinidis, E. I., Pappas, C., & Bamidis, P. D. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2010.2041553 · Cited by 250 · see citing works
EEG correlates (event-related desynchronization) of emotional face elaboration: A temporal analysis
Balconi, M., & Lucchiari, C. (2005) · Neuroscience Letters · doi:10.1016/j.neulet.2005.09.004 · Cited by 214 · see citing works
Yoon, H. J., & Chung, S. Y. (2013) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2013.10.017 · Cited by 184 · see citing works
Balconi, M., & Mazza, G. (2009) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2009.08.006 · Cited by 176 · see citing works
Feature learning from incomplete EEG with denoising autoencoder
Li, J., Struzik, Z. R., Zhang, L., & Cichocki, A. S. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.08.092 · Cited by 165 · see citing works
A 3-D Audio-Visual Corpus of Affective Communication
Fanelli, G., Gall, J., Romsdorfer, H., Weise, T., & Gool, L. V. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2052239 · Cited by 162 · see citing works
Martini, N., Menicucci, D., Sebastiani, L., Bedini, R., Pingitore, A., Vanello, N., Milanesi, M., Landini, L., & Gemignani, A. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.060 · Cited by 148 · see citing works
Christensen, J. C., Estepp, J. R., Wilson, G. F., & Russell, C. A. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.07.091 · Cited by 144 · see citing works
Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals
AlZoubi, O., D’Mello, S. K., & Calvo, R. A. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.4 · Cited by 131 · see citing works
Electroencephalogram-based emotion assessment system using ontology and data mining techniques
Chen, J., Hu, B., Moore, P. T., Zhang, X., & Ma, X. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2015.01.007 · Cited by 126 · see citing works
A Novel Semi-Supervised Deep Learning Framework for Affective State Recognition on EEG Signals
Jia, X., Li, K., Li, X., & Zhang, A. (2014) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe.2014.26 · Cited by 100 · see citing works
Mehmood, R. M., & Lee, H. J. (2016) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2016.04.009 · Cited by 99 · see citing works
Zhang, J., Yin, Z., & Wang, R. (2014) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2014.2366914 · Cited by 97 · see citing works
Self-citationMultimodal video classification with stacked contractive autoencoders
Liu, Y., Xiao-qing, F., & Zhou, Z. (2015) · Signal Processing · doi:10.1016/j.sigpro.2015.01.001 · Cited by 96 · see citing works
A real-time classification algorithm for EEG-based BCI driven by self-induced emotions
Iacoviello, D., Petracca, A., Spezialetti, M., & Placidi, G. (2015) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2015.08.011 · Cited by 92 · see citing works
Towards wireless emotional valence detection from EEG
Brown, L., Grundlehner, B., & Penders, J. (2011) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2011.6090412 · Cited by 90 · see citing works
Khezri, M., Firoozabadi, M., & Sharafat, A. R. (2015) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2015.07.006 · Cited by 89 · see citing works
Modeling Physiological Data with Deep Belief Networks
Wang, D., & Shang, Y. (2013) · International Journal of Information and Education Technology · doi:10.7763/ijiet.2013.v3.326 · Cited by 87 · see citing works
Analysis of physiological for emotion recognition with the IRS model
Li, C., Xu, C., & Feng, Z. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.07.112 · Cited by 78 · see citing works
Multimodal information improves the rapid detection of mental fatigue
Laurent, F., Valderrama, M. A., Besserve, M., Guillard, M., Lachaux, J.-P., Martinerie, J. M., & Florence, G. (2013) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2013.01.007 · Cited by 78 · see citing works
Jensen, R., & Shen, Q. (2008) · doi:10.1002/9780470377888.ch4 · Cited by 78 · see citing works
EEG Based Emotion Identification Using Unsupervised Deep Feature Learning
Li, X., Zhang, P., Song, D., Yu, G., Hou, Y., & Hu, B. X. (2015) · Open Research Online - ORO (The Open University) · Cited by 67 · see citing works
Yin, Z., & Zhang, J. (2013) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2013.09.007 · Cited by 66 · see citing works
Self-citationYin, Z., & Zhang, J. (2014) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2014.04.011 · Cited by 65 · see citing works
Self-citationAffective state recognition from EEG with deep belief networks
Li, K., Li, X., Zhang, Y., & Zhang, A. (2013) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2013.6732507 · Cited by 65 · see citing works
Understanding Affective Content of Music Videos through Learned Representations
Acar, E., Hopfgartner, F., & Albayrak, Ş. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-04114-8_26 · Cited by 64 · see citing works
Emotion recognition based on 3D fuzzy visual and EEG features in movie clips
Lee, G., Kwon, M., Sri, S. K., & Lee, M.-H. (2014) · Neurocomputing · doi:10.1016/j.neucom.2014.04.008 · Cited by 57 · see citing works
Test‐retest reliability of voluntary emotion regulation
Lee, H., Shackman, A. J., Jackson, D. C., & Davidson, R. J. (2009) · Psychophysiology · doi:10.1111/j.1469-8986.2009.00830.x · Cited by 54 · see citing works
Recognition of emotions induced by music videos using DT-CWPT
Naser, D. S., & Saha, G. (2013) · doi:10.1109/indiancmit.2013.6529408 · Cited by 50 · see citing works
Zhang, J., Yin, Z., & Wang, R. (2016) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2016.2561927 · Cited by 42 · see citing works
Self-citationEEG-based Valence Level Recognition for Real-Time Applications
Liu, Y., & Sourina, O. (2012) · International Conference on Cyberworlds (CW) · doi:10.1109/cw.2012.15 · Cited by 40 · see citing works
Brunner, C., Billinger, M., Vidaurre, C., & Neuper, C. (2011) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-011-0828-x · Cited by 39 · see citing works
Shin, Y., Lee, S., Ahn, M., Cho, H., Jun, S. C., & Lee, H.-N. (2015) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2015.08.017 · Cited by 32 · see citing works
Detection of temporal changes in psychophysiological data using statistical process control methods
Cannon, J., Krokhmal, P. A., Chen, Y., & Murphey, R. A. (2011) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2011.01.003 · Cited by 29 · see citing works
Konstantinidis, E. I., Frantzidis, C. A., Pappas, C., & Bamidis, P. D. (2012) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2012.03.008 · Cited by 28 · see citing works
Randomly dividing homologous samples leads to overinflated accuracies for emotion recognition
Liu, S., Zhang, D., Xu, M., Qi, H., He, F., Zhao, X. W., Zhou, P., Zhang, L., & Ming, D. (2015) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2015.02.023 · Cited by 26 · see citing works
(2009) · Encyclopedia of Biometrics · doi:10.1007/978-0-387-73003-5_722 · Cited by 13 · see citing works
Kim, J., & André, E. (2008) · IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) · doi:10.1109/mfi.2008.4648119 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A review of classification algorithms for EEG-based brain–computer interfaces: A 10 year update
Lotte, F., Bougrain, L., Cichocki, A., Clerc, M., Congedo, M., Rakotomamonjy, A., & Yger, F. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aab2f2 · Cited by 2,239 · see citing works
Ensemble deep learning: A review
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105151 · Cited by 2,282 · see citing works
Dong, X., Yu, Z., Cao, W., Shi, Y., & Ma, Q. (2020) · Frontiers of Computer Science · doi:10.1007/s11704-019-8208-z · Cited by 1,928 · see citing works
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806 · see citing works
Self-citationQiu, S., Zhao, H., Jiang, N., Wang, Z., Liu, L., An, Y., Zhao, H., Miao, X., Liu, R., & Fortino, G. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.006 · Cited by 671 · see citing works
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 657 · see citing works
Self-citationEEG Based Emotion Recognition: A Tutorial and Review
Li, X., Zhang, Y., Tiwari, P., Song, D., Hu, B., Yang, M., Zhao, Z., Kumar, N., & Marttinen, P. (2023) · ACM Computing Surveys · doi:10.1145/3524499 · Cited by 508 · see citing works
Hong, S., Zhou, Y., Shang, J., Xiao, C., & Sun, J. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103801 · Cited by 468 · see citing works
Liu, W., Qiu, J., Zheng, W., & Lu, B. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3071170 · Cited by 460 · see citing works
Houssein, E. H., Hammad, A. S. A., & Ali, A. A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07292-4 · Cited by 449 · see citing works
Emotion Recognition from Multi-Channel EEG through Parallel Convolutional Recurrent Neural Network
Yang, Y., Wu, Q., Qiu, M., Wang, Y., & Chen, X. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489331 · Cited by 358 · see citing works
Self-citationHuman and artificial intelligence collaboration for socially shared regulation in learning
Järvelä, S., Nguyen, A., & Hadwin, A. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13325 · Cited by 342 · see citing works
Emotion Recognition from Multiband EEG Signals Using CapsNet
Chao, H., Dong, L., Liu, Y., & Lu, B. (2019) · Sensors · doi:10.3390/s19092212 · Cited by 287 · see citing works
Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E., & Arunkumar, N. (2019) · IEEE Access · doi:10.1109/access.2018.2883213 · Cited by 289 · see citing works
A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Investigating EEG-based functional connectivity patterns for multimodal emotion recognition
Wu, X., Zheng, W., Li, Z., & Lu, B. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac49a7 · Cited by 244 · see citing works
Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing
Siddharth, S., Jung, T., & B, T. J. S. A. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2916015 · Cited by 237 · see citing works
Emotion Recognition using Multimodal Residual LSTM Network
Ma, J., Tang, H., Zheng, W.-L., & Lu, B.-L. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350871 · Cited by 230 · see citing works
Deep learning and alternative learning strategies for retrospective real-world clinical data
Chen, D. J., Liu, S., Kingsbury, P., Sohn, S., Storlie, C. B., Habermann, E. B., Naessens, J. M., Larson, D. W., & Liu, H. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0122-0 · Cited by 223 · see citing works
Jiang, Y., Li, W., Hossain, M. S., Chen, M., Alelaiwi, A. A., & Al-Hammadi, M. H. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.06.019 · Cited by 219 · see citing works
Majid Mehmood, R., Du, R., & Lee, H. J. (2017) · IEEE Access · doi:10.1109/access.2017.2724555 · Cited by 213 · see citing works
Comparison of different feature extraction methods for EEG-based emotion recognition
Nawaz, R., Cheah, K. H., Nisar, H., & Yap, V. V. (2020) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2020.04.005 · Cited by 213 · see citing works
Kumar, P., Chauhan, S., & Awasthi, L. K. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09986-x · Cited by 220 · see citing works
Kwon, Y.-H., Shin, S.-B., & Kim, S. (2018) · Sensors · doi:10.3390/s18051383 · Cited by 206 · see citing works
Continuous Convolutional Neural Network with 3D Input for EEG-Based Emotion Recognition
Yang, Y., Wu, Q., Fu, Y., & Chen, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-04239-4_39 · Cited by 197 · see citing works
Deep Learning in the Biomedical Applications: Recent and Future Status
Zemouri, R. M., Noureddine Zerhouni, A., & Racoceanu, D. (2019) · Applied Sciences · doi:10.3390/app9081526 · Cited by 194 · see citing works
Emotion recognition from unimodal to multimodal analysis: A review
Ezzameli, K., & Mahersia, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101847 · Cited by 195 · see citing works
Subaşı, A., Tuncer, T., Doğan, Ş., Tanko, D., & Sakoğlu, Ü. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.102648 · Cited by 176 · see citing works
A survey on methods and challenges in EEG based authentication
Jalaly Bidgoly, A., Jalaly Bidgoly, H., & Arezoumand, Z. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101788 · Cited by 173 · see citing works
Awais, M., Raza, M., Singh, N. K., Bashir, K., Manzoor, U., Islam, S. U., & Rodrigues, J. J. P. C. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3044031 · Cited by 166 · see citing works
Multimodal Data Fusion in Learning Analytics: A Systematic Review
Mu, S., Cui, M., & Huang, X. (2020) · Sensors · doi:10.3390/s20236856 · Cited by 157 · see citing works
Yin, Z., Wang, Y., Liu, L., Zhang, W., & Zhang, J. (2017) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2017.00019 · Cited by 155 · see citing works
Self-citationJeong, S., Ko, J., & Yeom, J. (2021) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.149726 · Cited by 154 · see citing works
Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J. (2020) · Brain Sciences · doi:10.3390/brainsci10100687 · Cited by 157 · see citing works
Rane, N. L., Choudhary, S. P., & Rane, J. (2024) · Studies in medical and health sciences. · doi:10.48185/smhs.v1i2.1225 · Cited by 150 · see citing works
Multimodal Emotion Recognition Using Deep Neural Networks
Tang, H., Liu, W., Zheng, W.-L., & Lu, B.-L. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70093-9_86 · Cited by 143 · see citing works
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
Emotion Modelling for Social Robotics Applications: A Review
Cavallo, F., Semeraro, F., Fiorini, L., Magyar, G., Sinčák, P., & Dario, P. (2018) · Journal of Bionic Engineering · doi:10.1007/s42235-018-0015-y · Cited by 139 · see citing works
Subject independent emotion recognition using EEG signals employing attention driven neural networks
Arjun, Rajpoot, A. S., & Panicker, M. R. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103547 · Cited by 137 · see citing works
Advances in Sentiment Analysis - Techniques, Applications, and Challenges
Li, J. (2023) · Artificial intelligence · doi:10.5772/intechopen.111293 · Cited by 137 · see citing works
Status of deep learning for EEG-based brain–computer interface applications
Hossain, K. M., Islam, M. A., Hossain, S., Nijholt, A., & Ahad, M. A. R. (2023) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.1006763 · Cited by 131 · see citing works
EEG-Based Emotion Recognition Using Quadratic Time-Frequency Distribution
Alazrai, R., Homoud, R., Alwanni, H., & Daoud, M. I. (2018) · Sensors · doi:10.3390/s18082739 · Cited by 129 · see citing works
Salankar, N., Mishra, P., & Garg, L. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.102389 · Cited by 130 · see citing works
Deep Learning in EEG: Advance of the Last Ten-Year Critical Period
Gong, S., Xing, K., Cichocki, A., & Li, J. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3079712 · Cited by 133 · see citing works
A new fractal pattern feature generation function based emotion recognition method using EEG
Tuncer, T., Doğan, Ş., & Subaşı, A. (2021) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2021.110671 · Cited by 126 · see citing works
EEG emotion recognition using improved graph neural network with channel selection
Lin, X., Chen, J., Ma, W., Tang, W., & Wang, Y. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107380 · Cited by 119 · see citing works
Self-citationEEG-Based Multimodal Emotion Recognition: A Machine Learning Perspective
Liu, H., Lou, T., Zhang, Y., Wu, Y., Xiao, Y., Jensen, C. S., & Zhang, D. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3369130 · Cited by 115 · see citing works
Locally robust EEG feature selection for individual-independent emotion recognition
Yin, Z., Liu, L., Chen, J., Zhao, B., & Wang, Y. (2020) · Expert systems with applications · doi:10.1016/j.eswa.2020.113768 · Cited by 108 · see citing works
Self-citationLi, H., Huang, J., & Ji, S. (2019) · Sensors · doi:10.3390/s19092034 · Cited by 111 · see citing works
A Review on EEG Signals Based Emotion Recognition
Soroush, M. Z., Maghooli, K., Setarehdan, S. K., & Nasrabadi, A. M. (2017) · International Clinical Neuroscience Journal · doi:10.15171/icnj.2017.01 · Cited by 108 · see citing works
Spatial-frequency convolutional self-attention network for EEG emotion recognition
Li, D., Xie, L. H., Chai, B., Wang, Z., & Yang, H. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.108740 · Cited by 108 · see citing works
An unsupervised EEG decoding system for human emotion recognition
Liang, Z., Oba, S., & Ishii, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.04.003 · Cited by 107 · see citing works
Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review
Alzahab, N. A., Apollonio, L., Iorio, A. D., Alshalak, M., Iarlori, S., Ferracuti, F., Monteriù, A., & Porcaro, C. (2021) · Brain Sciences · doi:10.3390/brainsci11010075 · Cited by 106 · see citing works
Hao, C., & Dong, L. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3020828 · Cited by 103 · see citing works
EEG-based emotion recognition with deep convolutional neural networks
Özdemir, M. A., Değirmenci, M., İzci, E., & Akan, A. (2020) · Biomedizinische Technik/Biomedical Engineering · doi:10.1515/bmt-2019-0306 · Cited by 101 · see citing works
Perdómo, O., Ríos, H. A., Rodriguez, F. J., Otálora, S., Mériaudeau, F., Müller, H., & Gonzalez, F. A. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.06.016 · Cited by 94 · see citing works
Exploring Deep Physiological Models for Nociceptive Pain Recognition
Thiam, P., Bellmann, P., Kestler, H. A., & Schwenker, F. (2019) · Sensors · doi:10.3390/s19204503 · Cited by 94 · see citing works
Li, Z., Zhang, G., Wang, L., Wei, J., & Dang, J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/acb79e · Cited by 88 · see citing works
Deep Learning on 1-D Biosignals: a Taxonomy-based Survey
Ganapathy, N., Swaminathan, R., & Deserno, T. (2018) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1667083 · Cited by 83 · see citing works
Visual-audio emotion recognition based on multi-task and ensemble learning with multiple features
Hao, M., Cao, W., Liu, Z., Wu, M., & Xiao, P. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.048 · Cited by 83 · see citing works
Tang, J., Ma, Z., Gan, K., Zhang, J., & Yin, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102129 · Cited by 82 · see citing works
Self-citationIntroducing Attention Mechanism for EEG Signals: Emotion Recognition with Vision Transformers
Arjun, A., Rajpoot, A. S., & Panicker, M. R. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9629837 · Cited by 82 · see citing works
Spatial-Temporal Feature Fusion Neural Network for EEG-Based Emotion Recognition
Wang, Z., Wang, Y., Zhang, J., Hu, C., Yin, Z., & Song, Y. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3165280 · Cited by 80 · see citing works
Self-citationAffective State Recognition in Livestock—Artificial Intelligence Approaches
Neethirajan, S. R. (2022) · Animals · doi:10.3390/ani12060759 · Cited by 78 · see citing works
Shi, H. Z., Hai, Z., Liu, Y., Gao, W., & Dou, S. (2019) · Sensors · doi:10.3390/s19122651 · Cited by 76 · see citing works
Lin, W., Li, C., & Sun, S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71589-6_33 · Cited by 72 · see citing works
EEG-based emotion recognition systems; comprehensive study
Hamzah, H. A., & Abdalla, K. K. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e31485 · Cited by 82 · see citing works
Ahonen, L., Cowley, B. U., Hellas, A., & Puolamäki, K. (2018) · Scientific Reports · doi:10.1038/s41598-018-21518-3 · Cited by 70 · see citing works
EEG Feature Extraction and Data Augmentation in Emotion Recognition
Kalashami, M. P., Pedram, M. M., & Sadr, H. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/7028517 · Cited by 69 · see citing works
Multimodal Emotion Recognition Using Deep Canonical Correlation AnalysisPreprint
Liu, W., Qiu, J., Zheng, W., & Lu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05349 · Cited by 83 · see citing works
Li, X., Zhao, Z., Song, D., Zhang, Y., Pan, J., Wu, L., Huo, J., Niu, C., & Wang, D. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00087 · Cited by 67 · see citing works
Nita, S., Bitam, S., Heidet, M., & Mellouk, A. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103580 · Cited by 67 · see citing works
Afzal, S., Khan, H. A., Piran, M. J., & Lee, J. W. (2024) · IEEE Access · doi:10.1109/access.2024.3422480 · Cited by 75 · see citing works
Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis
Qiu, J., Liu, W., & Lu, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-04221-9_20 · Cited by 69 · see citing works
Multimodal Emotion Recognition Based on Ensemble Convolutional Neural Network
Huang, H., Hu, Z., Wang, W., & Wu, M. (2019) · IEEE Access · doi:10.1109/access.2019.2962085 · Cited by 66 · see citing works
Survey of emotion recognition methods using EEG information
Yu, C., & Wang, M. (2022) · Cognitive Robotics · doi:10.1016/j.cogr.2022.06.001 · Cited by 58 · see citing works
Hao, C., Zhi, H., Dong, L., & Liu, Y. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/9750904 · Cited by 65 · see citing works
Single-Channel Selection for EEG-Based Emotion Recognition Using Brain Rhythm Sequencing
Li, J. W., Barma, S., Mak, P. U., Chen, F., Li, C., Li, M. T., Vai, M. I., & Pun, S. H. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3148109 · Cited by 63 · see citing works
Yan, H., Guo, K., Xing, X., & Xu, X. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3394873 · Cited by 63 · see citing works
Jin, W., Zhu, X., Qian, L., Wu, C., Yang, F., Zhan, D., Kang, Z., Luo, K., Meng, D., & Xu, G. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1431815 · Cited by 65 · see citing works
Siddiqui, M. F. H., & Javaid, A. Y. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030046 · Cited by 62 · see citing works
Hao, C., & Liu, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2974009 · Cited by 60 · see citing works
Oh, S., Lee, J., & Kim, D. K. (2020) · Sensors · doi:10.3390/s20030866 · Cited by 59 · see citing works
Zhang, T., Liu, M., Tian, Y., & Al-Nabhan, N. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3038631 · Cited by 59 · see citing works
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70 · see citing works
Dzieżyc, M., Gjoreski, M., Kazienko, P., Saganowski, S., & Gams, M. (2020) · Sensors · doi:10.3390/s20226535 · Cited by 58 · see citing works
Bokade, R., Navato, A. P., Ouyang, R., Jin, X., Chou, C.-A., Ostadabbas, S., & Mueller, A. V. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113885 · Cited by 59 · see citing works
Zhao, Y., Cao, X., Lin, J., Yu, D., & Cao, X. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3093923 · Cited by 57 · see citing works
Stress Reduction Using Bilateral Stimulation in Virtual Reality
Kamińska, D., Smółka, K., Zwoliński, G., Wiak, S., Merecz‐Kot, D., & Anbarjafari, G. (2020) · IEEE Access · doi:10.1109/access.2020.3035540 · Cited by 56 · see citing works
Hemanth, D. J., Anitha, J., & Sơn, L. H. (2018) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2018.04.006 · Cited by 53 · see citing works
Ning, X., Wang, J., Lin, Y., Cai, X., Chen, H., Gou, H., Li, X., & Jia, Z. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3338676 · Cited by 53 · see citing works
Liu, M., Zhou, M., Zhang, T., & Xiong, N. N. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106071 · Cited by 51 · see citing works
Emotion recognition using EEG phase space dynamics and Poincare intersections
Soroush, M. Z., Maghooli, K., Setarehdan, S., & Nasrabadi, A. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.101918 · Cited by 45 · see citing works
EEG-based affective state recognition from human brain signals by using Hjorth-activity
Mehmood, R. M., Bilal, M., Vimal, S., & Lee, S. (2022) · Measurement · doi:10.1016/j.measurement.2022.111738 · Cited by 50 · see citing works
Yin, Z., Zhao, M., Zhang, W., Wang, Y., Wang, Y., & Zhang, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.02.061 · Cited by 48 · see citing works
Self-citationSymbolic Analysis of Brain Dynamics Detects Negative Stress
García-Martínez, B., Martínez‐Rodrigo, A., Zangróniz, R., Pastor, J. M., & Alcaraz, R. (2017) · Entropy · doi:10.3390/e19050196 · Cited by 47 · see citing works
Choi, E. J., & Kim, D. K. (2018) · Healthcare Informatics Research · doi:10.4258/hir.2018.24.4.309 · Cited by 47 · see citing works
Emotion recognition through EEG phase space dynamics and Dempster-Shafer theory
Soroush, M. Z., Maghooli, K., Setarehdan, S. K., & Nasrabadi, A. M. (2019) · Medical Hypotheses · doi:10.1016/j.mehy.2019.03.025 · Cited by 47 · see citing works
Goshvarpour, A., Abbasi, A., & Goshvarpour, A. (2017) · Australasian Physical & Engineering Sciences in Medicine · doi:10.1007/s13246-017-0571-1 · Cited by 46 · see citing works
- Authors
- Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J.
- Affiliations
- University of Shanghai for Science and Technology (CN)
- East China University of Science and Technology (CN)
- Corresponding author
- Yin, Zhong
- Wang, Yongxiong
- Venue
- Computer Methods and Programs in Biomedicine
- Publisher
- Elsevier BV
- Published
- 2017-03
- Volume
- 140
- Pages
- 93-110
- ISSN
- 0169-2607 (ISSN-L)
- OpenAlex
- W2565944610
- Cited by
- 416
- References
- 59
- Citations by year
- 2026: 17 · 2025: 41 · 2024: 42 · 2023: 50 · 2022: 73 · 2021: 40 · 2020: 52 · 2019: 48 · 2018: 36 · 2017: 16
- FWCI
- 47.46
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Computer Methods and Programs in Biomedicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- emotion recognition, multimodal physiological signals, stacked autoencoder, ensemble classifiers, feature fusion, graph neural networks, valence and arousal, DEAP database
- MeSH terms
- Emotions, Humans, Learning, Models, Psychological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
