Emotion Recognition from Multiband EEG Signals Using CapsNet
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Emotion recognition based on multi-channel electroencephalograph (EEG) signals is becoming increasingly attractive. However, the conventional methods ignore the spatial characteristics of EEG signals, which also contain salient information related to emotion states. In this paper, a deep learning framework based on a multiband feature matrix (MFM) and a capsule network (CapsNet) is proposed. In the framework, the frequency domain, spatial characteristics, and frequency band characteristics of the multi-channel EEG signals are combined to construct the MFM. Then, the CapsNet model is introduced to recognize emotion states according to the input MFM. Experiments conducted on the dataset for emotion analysis using EEG, physiological, and video signals (DEAP) indicate that the proposed method outperforms most of the common models. The experimental results demonstrate that the three characteristics contained in the MFM were complementary and the capsule network was more suitable for mining and utilizing the three correlation characteristics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 2×
“To compensate for the shortcomings of CNN, a network structure called the capsule network has been proposed [45].”
BackgroundMethodDEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 2×
“Methods Each trial was divided into two classes on each indicator.”
BackgroundMethodDynamic Routing Between Capsules
Sabour, S., Frosst, N., & Hinton, G. E. (2017) · arXiv · doi:10.48550/arxiv.1710.09829 · Cited by 4,167 · cited 4×
“To compensate for the shortcomings of CNN, a network structure called the capsule network has been proposed [43].”
BackgroundInfluentialAdolphs, R., Tranel, D., Damasio, H., & Damasio, A. (1994) · Nature · doi:10.1038/372669a0 · Cited by 2,360 · cited 1×
“Current emotion recognition research is implemented mainly according to non-physiological signals such as facial expressions [1], speech [2], body movement [3], and physiological signals like skin resistance (SR), electrocardiogram (ECG), functional magnetic resonance imaging (fMRI), electroencephalograph (EEG) [4], a…”
BackgroundEEG analysis based on time domain properties
Hjorth, B. (1970) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(70)90143-4 · Cited by 1,504
Feature Extraction and Selection for Emotion Recognition from EEG
Jenke, R., Peer, A., & Buss, M. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2339834 · Cited by 1,081
EEG-Based Emotion Recognition in Music Listening
Lin, Y.-P., Wang, C.-H., Jung, T.-P., Wu, T.-L., Jeng, S.-K., Duann, J.-R., & Chen, J.-H. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2048568 · Cited by 1,021
Emotion Recognition From EEG Using Higher Order Crossings
Petrantonakis, P. C., & Hadjileontiadis, L. J. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2034649 · Cited by 687 · cited 1×
“In addition, other studies introduced Hjorth features (Mobility, Complexity, and Activity) [17], non-stationary index (NSI) [18] and higher-order crossing features (HOCs) [19,20] for EEG emotion recognition.”
BackgroundJirayucharoensak, S., Pan-Ngum, S., & Israsena, P. (2014) · The Scientific World Journal · doi:10.1155/2014/627892 · Cited by 486
Classification of human emotion from EEG using discrete wavelet transform
Murugappan, M., Ramachandran, N., & Sazali, Y. (2010) · Journal of Biomedical Science and Engineering · doi:10.4236/jbise.2010.34054 · Cited by 436
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.12.005 · Cited by 416 · cited 2×
“Fusion of the power features from different frequency bands has been used in emotion recognition studies [4,6].”
BackgroundEEG-based emotion classification using deep belief networks
Zheng, W.-L., Zhu, J.-Y., Peng, Y., & Lu, B.-L. (2014) · 2014 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890166 · Cited by 367
Self-citationReal-Time Movie-Induced Discrete Emotion Recognition from EEG Signals
Liu, Y.-J., Yu, M., Zhao, G., Song, J., Ge, Y., & Shi, Y. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2660485 · Cited by 356
Self-citationEmotion Recognition Using Multimodal Deep Learning
Liu, W., Zheng, W.-L., & Lu, B.-L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46672-9_58 · Cited by 301
Self-citationEmotion Recognition from EEG Signals Using Multidimensional Information in EMD Domain
Zhuang, N., Zeng, Y., Tong, L., Zhang, C., Zhang, H., & Yan, B. (2017) · BioMed Research International · doi:10.1155/2017/8317357 · Cited by 283 · cited 6×
“As shown in Tabl 4, the proposed method achieved the highest recognition accuracy in arousal and valence dimensions, with the exception of the results of reference [42].”
BackgroundMethodResultInfluentialPetrantonakis, P. C., & Hadjileontiadis, L. J. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.7 · Cited by 260
Frantzidis, C. A., Bratsas, C., Papadelis, C. L., Konstantinidis, E., Pappas, C., & Bamidis, P. D. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2010.2041553 · Cited by 247
Multimodal emotion recognition using EEG and eye tracking data
Zheng, W.-L., Dong, B.-N., & Lu, B.-L. (2014) · 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc.2014.6944757 · Cited by 230 · cited 1×
“Since the FFT cannot reflect temporal information in the frequency data, the short-time Fourier transform (STFT) is used to extract time-frequency domain features [8,24,25].”
MethodSelf-citationEmotion recognition from multichannel EEG signals using K-nearest neighbor classification
Li, M., Xu, H., Liu, X., & Lu, S. (2018) · Technology and Health Care · doi:10.3233/thc-174836 · Cited by 230
Kwon, Y.-H., Shin, S.-B., & Kim, S.-D. (2018) · Sensors · doi:10.3390/s18051383 · Cited by 206 · cited 3×
“[23] proposed a method in which time-frequency feature maps of each channel as well as other physiological features were fed into a convolutional neural network (CNN) to recognize emotion states.”
MethodInfluentialEmotion recognition based on the sample entropy of EEG
Jie, X., Cao, R., & Li, L. (2014) · Bio-Medical Materials and Engineering · doi:10.3233/bme-130919 · Cited by 205
Li, Y., Huang, J., Zhou, H., & Zhong, N. (2017) · Applied Sciences · doi:10.3390/app7101060 · Cited by 185
Voice emotion recognition by cochlear-implanted children and their normally-hearing peers
Chatterjee, M., Zion, D. J., Deroche, M. L., Burianek, B. A., Limb, C. J., Goren, A. P., Kulkarni, A. M., & Christensen, J. A. (2015) · Hearing Research · doi:10.1016/j.heares.2014.10.003 · Cited by 182
Yin, Z., Wang, Y., Liu, L., Zhang, W., & Zhang, J. (2017) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2017.00019 · Cited by 155
Affect representation and recognition in 3D continuous valence–arousal–dominance space
Verma, G. K., & Tiwary, U. S. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-015-3119-y · Cited by 148
Real-Time Fractal-Based Valence Level Recognition from EEG
Liu, Y., & Sourina, O. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38803-3_6 · Cited by 139
Self-citationClassification of Human Emotions from EEG Signals using Statistical Features and Neural Network
Chai, T.-Y., San, W. S., Seong, T. C., & Rizon, M. (2009) · International Journal of Integrated Engineering · Cited by 127
Evaluating Classifiers for Emotion Recognition Using EEG
Sohaib, A. T., Qureshi, S., Hagelbäck, J., Hilborn, O., & Jerčić, P. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39454-6_53 · Cited by 125 · cited 2×
“Many machine-learning techniques have been applied to identify emotion states, such as support vector machine (SVM) [28], k-nearest neighbor (k-NN) [30], and artificial neural network (ANN) [31].”
MethodRobust EEG emotion classification using segment level decision fusion
Rozgic, V., Vitaladevuni, S. N., & Prasad, R. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6637858 · Cited by 116 · cited 1×
“Welch’s method [26] was also used to calculate the power spectral density (PSD) of different frequency bands in some studies [27,28].”
MethodFast CapsNet for Lung Cancer Screening
Mobiny, A., & Van Nguyen, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00934-2_82 · Cited by 110
EEG Correlates of Different Emotional States Elicited during Watching Music Videos
Kroupi, E., Yazdani, A., & Ebrahimi, T. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_58 · Cited by 101 · cited 1×
“In addition, other studies introduced Hjorth features (Mobility, Complexity, and Activity) [17], non-stationary index (NSI) [18] and higher-order crossing features (HOCs) [19,20] for EEG emotion recognition.”
BackgroundAffective classification using Bayesian classifier and supervised learning
Chung, S. Y., & Yoon, H. J. (2012) · International Conference on Control, Automation and Systems (ICCAS) · Cited by 93
Classification of emotions induced by music videos and correlation with participants’ rating
Daimi, S. N., & Saha, G. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2014.03.050 · Cited by 92
EEG Based Emotion Identification Using Unsupervised Deep Feature Learning
Li, X., Zhang, P., Song, D., Yu, G., Hou, Y., & Hu, B. X. (2015) · Open Research Online - ORO (The Open University) · Cited by 85
Ross, P. D., Polson, L., & Grosbras, M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0044815 · Cited by 84
Modeling Physiological Data with Deep Belief Networks.
Wang, D., & Shang, Y. (2013) · PubMed · doi:10.7763/ijiet.2013.v3.326 · Cited by 75 · cited 2×
“Chung and Yoon [41] Power spectral features Bayes classifier weighted-log-pos terior function 10-fold cross-validation 0.”
BackgroundMethodContinuous Music-Emotion Recognition Based on Electroencephalogram
Thammasan, N., Moriyama, K., Fukui, K., & Numao, M. (2016) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2015edp7251 · Cited by 75
Hao, C., Zhi, H., Dong, L., & Liu, Y. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/9750904 · Cited by 72
Self-citationAn evaluation of feature extraction in EEG-based emotion prediction with support vector machines
Wichakam, I., & Vateekul, P. (2014) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse.2014.6841851 · Cited by 65 · cited 9×
“In the 1D-CNN model, the input were 128-dimensional vectors (32 The average recognition accuracies of all subjects using CapsNet model A in the three dimensions of arousal, valence, and dominance were 0.6828, 0.6673, and 0.6725, respectively, whereas the average accuracies of other models were 0.6667, 0.6460, and 0.66…”
MethodInfluentialRecognition of emotions induced by music videos using DT-CWPT
Daimi, S. N., & Saha, G. (2013) · doi:10.1109/indiancmit.2013.6529408 · Cited by 57 · cited 5×
“Meanwhile, we also propose a deep learning model based on a capsule network, which was used to judge emotional state.”
MethodResultInfluentialMultimodal Emotion Recognition Using Multimodal Deep Learning
Liu, W., Zheng, W., & Lu, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08225 · Cited by 49
Self-citationGender and emotion recognition with implicit user signals
Bilalpur, M., Kia, S. M., Chawla, M., Chua, T., & Subramanian, R. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3136790 · Cited by 30
Three class emotions recognition based on deep learning using staked autoencoder
Yang, B., Han, X., & Tang, J. (2017) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei.2017.8302098 · Cited by 20
Decoding affect in videos employing the MEG brain signal
Abadi, M. K., Kia, S. M., Subramanian, R., Avesani, P., & Sebe, N. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553809 · Cited by 14 · cited 1×
“Current emotion recognition research is implemented mainly according to non-physiological signals such as facial expressions [1], speech [2], body movement [3], and physiological signals like skin resistance (SR), electrocardiogram (ECG), functional magnetic resonance imaging (fMRI), electroencephalograph (EEG) [4], a…”
BackgroundGender and Emotion Recognition with Implicit User SignalsPreprint
Bilalpur, M., Kia, S. M., Chawla, M., Chua, T., & Subramanian, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.08735 · Cited by 5
Research works printed but not linked: 1
- Morris, 1995, SAM: The Self-Assessment Manikin an Efficient Cross-Cultural Measurement of Emotional Response, Advertising Research., 35, 63
Cited by
Human Emotion Recognition: Review of Sensors and Methods
Dzedzickis, A., Kaklauskas, A., & Bucinskas, V. (2020) · Sensors · doi:10.3390/s20030592 · Cited by 582
“High/low arousal, valence and dominance The DEAP dataset was used [59]”
MethodEEG Based Emotion Recognition: A Tutorial and Review
Li, X., Zhang, Y., Tiwari, P., Song, D., Hu, B., Yang, M., Zhao, Z., Kumar, N., & Marttinen, P. (2023) · ACM Computing Surveys · doi:10.1145/3524499 · Cited by 508
“[133] DEAP a framework based on multi-band feature matrix (MFM) and capsule network (CapsNet) 2 classes (Valence: 0.”
BackgroundMethodInfluentialKhare, S. K., Blanes-Vidal, V., Nadimi, E. S., & Acharya, U. R. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102019 · Cited by 469
Kwabena Patrick, M., Felix Adekoya, A., Abra Mighty, A., & Edward, B. Y. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2019.09.014 · Cited by 282
“…(Wang et al., 2018a), hand-written and text recognition (Mandal et al., 2019; Zhao et al., 2018a; Kim et al., 2018a) intent detection (Xia et al., 2018; Zhang et al., 2018a), mood and emotion detection (Wang et al., 2018c; Rathnayaka et al., 2018; Guo et al., 2018; Chao et al., 2019) and so on.”
Vistorte, A. O. R., Deroncele-Acosta, A., Ayala, J. L. M., Barrasa, A., López-Granero, C., & Martí-González, M. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1387089 · Cited by 281
“Chao et al. (2019) introduced a deep learning framework that employs a multiband feature matrix and a CapsNet model to improve emotion recognition from multi-channel EEG signals, outperforming common models.”
Method
Metadata
- Authors
- Chao, H., Dong, L., Liu, Y., & Lu, B.
- Affiliations
- School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo 454000, China
- Henan Polytechnic University (CN)
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2019-05-13
- Volume
- 19
- Issue
- 9
- Pages
- 2212
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC6540345
- OpenAlex
- W2944401411
- Cited by
- 287
- References
- 45
- Citations by year
- 2026: 16 · 2025: 25 · 2024: 34 · 2023: 51 · 2022: 65 · 2021: 47 · 2020: 40 · 2019: 9
- FWCI
- 47.26
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Foundation for University Key Teacher by Henan Province (2015GGJS068)
- Fundamental Research Funds for the Universities of Henan Province (NSFRF1616)
- Key Scientific Research Projects of Universities in Henan (19A520004)
- Foundation for Scientific and Technological Project of Henan Province (172102210279)
- National Nature Science Foundation of China (61502150 and 61403128)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Keywords
- emotion recognition, capsule networks, frequency-domain features, DEAP dataset, physiological signals
- MeSH terms
- Machine Learning, Cerebral Cortex, Electrodes, Electroencephalography, Emotions, Humans, Signal Processing, Computer-Assisted, Entropy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.