CoChat

Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces

Open access
He, Zhipeng, Li, Zina, Yang, Fuzhou, Wang, Lei, Li, Jingcong, Zhou, Chengju, Pan, Jiahui · 2020 · Brain Sciences
Cited by 151

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.

Abstract

With the continuous development of portable noninvasive human sensor technologies such as brain–computer interfaces (BCI), multimodal emotion recognition has attracted increasing attention in the area of affective computing. This paper primarily discusses the progress of research into multimodal emotion recognition based on BCI and reviews three types of multimodal affective BCI (aBCI): aBCI based on a combination of behavior and brain signals, aBCI based on various hybrid neurophysiology modalities and aBCI based on heterogeneous sensory stimuli. For each type of aBCI, we further review several representative multimodal aBCI systems, including their design principles, paradigms, algorithms, experimental results and corresponding advantages. Finally, we identify several important issues and research directions for multimodal emotion recognition based on BCI.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/brainsci10100687

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 19 references

Research works printed but not linked: 113

  1. Mühl, C., Nijholt, A., Allison, B., Dunne, S., and Heylen, D. (2011, January 9–12). Affective brain-computer interfaces (aBCI 2011). Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Memphis, TN, USA. · DOI
  2. Allison, 2014, A survey of affective brain computer interfaces: Principles, state-of-the-art, and challenges, Brain-Comput. Interfaces, 1, 66, 10.1080/2326263X.2014.912881 · DOI · Paper page
  3. Van den Broek, E.L. (2012). Affective computing: A reverence for a century of research. Cognitive Behavioural Systems, Springer. · DOI
  4. Ekman, P.E., and Davidson, R.J. (1994). The Nature of Emotion: Fundamental Questions, Oxford University Press.
  5. Russell, 1980, A circumplex model of affect, J. Personal. Soc. Psychol., 39, 1161, 10.1037/h0077714 · DOI · Paper page
  6. Niemic, 2004, Studies of Emotion: A Theoretical and Empirical Review of Psychophysiological Studies of Emotion, Psychophy, 1, 15
  7. Chanel, 2009, Short-term emotion assessment in a recall paradigm, Int. J. Hum.-Comput. Stud., 67, 607, 10.1016/j.ijhcs.2009.03.005 · DOI · Paper page
  8. Huang, H., Xie, Q., Pan, J., He, Y., Wen, Z., Yu, R., and Li, Y. (2019). An EEG-based brain computer interface for emotion recognition and its application in patients with Disorder of Consciousness. IEEE Trans. Affect. Comput.
  9. Wang, 2017, Enhancing clinical communication assessments using an audiovisual BCI for patients with disorders of consciousness, J. Neural Eng., 14, 046024, 10.1088/1741-2552/aa6c31 · DOI
  10. Kory, 2015, A review and meta-analysis of multimodal affect detection systems, ACM Comput. Surv. (CSUR), 47, 1
  11. Ebrahimi, 2003, Guest editorial braincomputer interface technology: A review of the second international meeting, IEEE Signal Process. Mag., 20, 14, 10.1109/MSP.2003.1166626 · DOI
  12. Phan, 2010, Tensor decompositions for feature extraction and classification of high dimensional datasets, Nonlinear Theory Its Appl. IEICE, 1, 37, 10.1587/nolta.1.37 · DOI · Paper page
  13. Congedo, 2017, Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review, Brain-Comput. Interfaces, 4, 155, 10.1080/2326263X.2017.1297192 · DOI · Paper page
  14. Soleymani, 2011, Multimodal emotion recognition in response to videos, IEEE Trans. Affect. Comput., 3, 211, 10.1109/T-AFFC.2011.37 · DOI · Paper page
  15. Liu, W., Zheng, W.-L., and Lu, B.-L. (2016, January 16–21). Emotion recognition using multimodal deep learning. Proceedings of the International Conference on Neural Information Processing, Kyoto, Japan. · DOI · Paper page
  16. Minotto, 2015, Multimodal multi-channel on-line speaker diarization using sensor fusion through SVM, IEEE Trans. Multimed., 17, 1694, 10.1109/TMM.2015.2463722 · DOI
  17. Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12–15). Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain. · DOI · Paper page
  18. Haghighat, 2016, Discriminant correlation analysis: Real-time feature level fusion for multimodal biometric recognition, IEEE Trans. Inf. Forensics Secur., 11, 1984, 10.1109/TIFS.2016.2569061 · DOI · Paper page
  19. Zhalehpour, 2016, BAUM-1: A spontaneous audio-visual face database of affective and mental states, IEEE Trans. Affect. Comput., 8, 300, 10.1109/TAFFC.2016.2553038 · DOI · Paper page
  20. Wu, 2016, A novel lip descriptor for audio-visual keyword spotting based on adaptive decision fusion, IEEE Trans. Multimed., 18, 326, 10.1109/TMM.2016.2520091 · DOI
  21. Koelstra, 2013, Fusion of facial expressions and EEG for implicit affective tagging, Image Vis. Comput., 31, 164, 10.1016/j.imavis.2012.10.002 · DOI · Paper page
  22. Soleymani, M., Asghari-Esfeden, S., Pantic, M., and Fu, Y. (2014, January 14–18). Continuous emotion detection using EEG signals and facial expressions. Proceedings of the 2014 IEEE International Conference on Multimedia and Expo (ICME), Chengdu, China. · DOI · Paper page
  23. Freund, Y., and Schapire, R.E. (1996, January 3–6). Experiments with a new boosting algorithm. Proceedings of the ICML, Bari, Italy.
  24. Ponti, M.P. (2011, January 28–30). Combining classifiers: From the creation of ensembles to the decision fusion. Proceedings of the 2011 24th SIBGRAPI Conference on Graphics, Patterns, and Images Tutorials, Alagoas, Brazil. · DOI
  25. Chang, Z., Liao, X., Liu, Y., and Wang, W. (2011, January 19–21). Research of decision fusion for multi-source remote-sensing satellite information based on SVMs and DS evidence theory. Proceedings of the Fourth International Workshop on Advanced Computational Intelligence, Wuhan, China. · doi:10.1109/iwaci.2011.6160042
  26. Nefian, 2002, Dynamic Bayesian networks for audio-visual speech recognition, EURASIP J. Adv. Signal Process., 2002, 783042, 10.1155/S1110865702206083 · DOI · Paper page
  27. Murofushi, 1989, An interpretation of fuzzy measures and the Choquet integral as an integral with respect to a fuzzy measure, Fuzzy Sets Syst., 29, 201, 10.1016/0165-0114(89)90194-2 · DOI
  28. Savran, A., Ciftci, K., Chanel, G., Mota, J., Hong Viet, L., Sankur, B., Akarun, L., Caplier, A., and Rombaut, M. (August, January 17). Emotion detection in the loop from brain signals and facial images. Proceedings of the eNTERFACE 2006 Workshop, Dubrovnik, Croatia.
  29. Correa, J.A.M., Abadi, M.K., Sebe, N., and Patras, I. (2018). Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE Trans. Affect. Comput.
  30. Conneau, A.-C., Hajlaoui, A., Chetouani, M., and Essid, S. (September, January 28). EMOEEG: A new multimodal dataset for dynamic EEG-based emotion recognition with audiovisual elicitation. Proceedings of the 2017 25th European Signal Processing Conference (EUSIPCO), Kos, Greece. · DOI
  31. Subramanian, 2016, ASCERTAIN: Emotion and personality recognition using commercial sensors, IEEE Trans. Affect. Comput., 9, 147, 10.1109/TAFFC.2016.2625250 · DOI · Paper page
  32. Song, 2019, MPED: A multi-modal physiological emotion database for discrete emotion recognition, IEEE Access, 7, 12177, 10.1109/ACCESS.2019.2891579 · DOI · Paper page
  33. Liu, W., Zheng, W.-L., and Lu, B.-L. (2016). Multimodal emotion recognition using multimodal deep learning. arXiv. · DOI · Paper page
  34. Bulling, 2010, Eye movement analysis for activity recognition using electrooculography, IEEE Trans. Pattern Anal. Mach. Intell., 33, 741, 10.1109/TPAMI.2010.86 · DOI · Paper page
  35. Bakshi, 2014, A survey on face detection methods and feature extraction techniques of face recognition, Int. J. Emerg. Trends Technol. Comput. Sci., 3, 233
  36. Jacobs, 2008, The coupling of emotion and cognition in the eye: Introducing the pupil old/new effect, Psychophysiology, 45, 130, 10.1111/j.1469-8986.2007.00606.x · DOI · Paper page
  37. Wang, 2015, A circuit for pupil orienting responses: Implications for cognitive modulation of pupil size, Curr. Opin. Neurobiol., 33, 134, 10.1016/j.conb.2015.03.018 · DOI · Paper page
  38. Chang, 2011, LIBSVM: A library for support vector machines, ACM Trans. Intell. Syst. Technol. (TIST), 2, 1, 10.1145/1961189.1961199 · DOI · Paper page
  39. Wu, X., Zheng, W.-L., and Lu, B.-L. (2020). Investigating EEG-Based Functional Connectivity Patterns for Multimodal Emotion Recognition. arXiv. · DOI
  40. Xu, X., Quan, C., and Ren, F. (2015, January 2–5). Facial expression recognition based on Gabor Wavelet transform and Histogram of Oriented Gradients. Proceedings of the 2015 IEEE International Conference on Mechatronics and Automation (ICMA), Beijing, China. · DOI
  41. Huang, X., He, Q., Hong, X., Zhao, G., and Pietikainen, M. (2014, January 12–16). Improved spatiotemporal local monogenic binary pattern for emotion recognition in the wild. Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey. · DOI
  42. Saeed, A., Al-Hamadi, A., and Niese, R. (2013, January 13–15). The effectiveness of using geometrical features for facial expression recognition. Proceedings of the 2013 IEEE International Conference on Cybernetics (CYBCO), Lausanne, Switzerland. · DOI
  43. Ekman, R. (1997). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press. · DOI · Paper page
  44. Sokolov, S., Velchev, Y., Radeva, S., and Radev, D. (2017, January 25–26). Human Emotion Estimation from EEG and Face Using Statistical Features and SVM. Proceedings of the Fourth International Conference on Computer Science and Information Technology, Geneva, Switzerland. · DOI
  45. Chang, C.Y., Tsai, J.S., Wang, C.J., and Chung, P.C. (April, January 30). Emotion recognition with consideration of facial expression and physiological signals. Proceedings of the IEEE Conference on Computational Intelligence in Bioinformatics & Computational Biology, Nashville, TN, USA.
  46. Chunawale, A., and Bedekar, D. (2020, January 20). Human Emotion Recognition using Physiological Signals: A Survey. Proceedings of the 2nd International Conference on Communication & Information Processing (ICCIP), Tokyo, Japan. · DOI
  47. Vijayakumar, S., Flynn, R., and Murray, N. (2020, January 11–12). A comparative study of machine learning techniques for emotion recognition from peripheral physiological signals. Proceedings of the ISSC 2020 31st Irish Signals and System Conference, Letterkenny, Ireland. · DOI
  48. Liu, Y., and Sourina, O. (2014). Real-Time Subject-Dependent EEG-Based Emotion Recognition Algorithm. Transactions on Computational Science XXIII, Springer. · DOI
  49. Bo, 1970, EEG analysis based on time domain properties, Electroencephalogr. Clin. Neurophysiol., 29, 306, 10.1016/0013-4694(70)90143-4 · DOI · Paper page
  50. Thammasan, 2016, Continuous Music-Emotion Recognition Based on Electroencephalogram, IEICE Trans. Inf. Syst., 99, 1234, 10.1587/transinf.2015EDP7251 · DOI
  51. Shi, L.C., Jiao, Y.Y., and Lu, B.L. (2013, January 3–7). Differential entropy feature for EEG-based vigilance estimation. Proceedings of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan.
  52. Zheng, 2017, Identifying stable patterns over time for emotion recognition from EEG, IEEE Trans. Affect. Comput., 10, 417, 10.1109/TAFFC.2017.2712143 · DOI · Paper page
  53. Akin, 2002, Comparison of Wavelet Transform and FFT Methods in the Analysis of EEG Signals, J. Med. Syst., 26, 241, 10.1023/A:1015075101937 · doi:10.1023/a:1015075101937
  54. 2005, Comparison of STFT and wavelet transform methods in determining epileptic seizure activity in EEG signals for real-time application, Comput. Biol. Med., 35, 603, 10.1016/j.compbiomed.2004.05.001 · DOI · Paper page
  55. Hadjidimitriou, 2012, Toward an EEG-Based Recognition of Music Liking Using Time-Frequency Analysis, IEEE Trans. Biomed. Eng., 59, 3498, 10.1109/TBME.2012.2217495 · DOI · Paper page
  56. Liao, 2020, Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP Conf. Ser. Mater. Sci. Eng., 782, 032005, 10.1088/1757-899X/782/3/032005 · DOI
  57. 2018, Electroencephalography Based Fusion Two-Dimensional (2D)-Convolution Neural Networks (CNN) Model for Emotion Recognition System, Sensors, 18, 1383, 10.3390/s18051383 · DOI · Paper page
  58. Rahimi, 2017, High-Dimensional Computing as a Nanoscalable Paradigm, IEEE Trans. Circuits Syst. I Regul. Pap., 64, 2508, 10.1109/TCSI.2017.2705051 · DOI · Paper page
  59. Montagna, F., Rahimi, A., Benatti, S., Rossi, D., and Benini, L. (2018, January 24–28). PULP-HD: Accelerating Brain-Inspired High-Dimensional Computing on a Parallel Ultra-Low Power Platform. Proceedings of the 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), San Francisco, CA, USA. · DOI
  60. Rahimi, A., Kanerva, P., Millán, J.D.R., and Rabaey, J.M. (2017, January 15–17). Hyperdimensional Computing for Noninvasive Brain–Computer Interfaces: Blind and One-Shot Classification of EEG Error-Related Potentials. Proceedings of the 10th EAI International Conference on Bio-Inspired Information and Communications Technologies (Formerly BIONETICS), Hoboken, NJ, USA. · DOI
  61. Rahimi, 2019, Efficient Biosignal Processing Using Hyperdimensional Computing: Network Templates for Combined Learning and Classification of ExG Signals, Proc. IEEE, 107, 123, 10.1109/JPROC.2018.2871163 · DOI · Paper page
  62. Chang, E.-J., Rahimi, A., Benini, L., and Wu, A.-Y.A. (2019, January 18–20). Hyperdimensional Computing-based Multimodality Emotion Recognition with Physiological Signals. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), Hsinchu, Taiwan. · DOI · Paper page
  63. Huppert, 2006, A temporal comparison of BOLD, ASL, and NIRS hemodynamic responses to motor stimuli in adult humans, Neuroimage, 29, 368, 10.1016/j.neuroimage.2005.08.065 · DOI · Paper page
  64. Jasdzewski, 2003, Differences in the hemodynamic response to event-related motor and visual paradigms as measured by near-infrared spectroscopy, Neuroimage, 20, 479, 10.1016/S1053-8119(03)00311-2 · DOI · Paper page
  65. Malonek, 1996, Interactions between electrical activity and cortical microcirculation revealed by imaging spectroscopy: Implications for functional brain mapping, Science, 272, 551, 10.1126/science.272.5261.551 · DOI · Paper page
  66. Ayaz, 2011, Using MazeSuite and functional near infrared spectroscopy to study learning in spatial navigation, J. Vis. Exp., 56, e3443
  67. Liao, 2002, Estimating the delay of the fMRI response, NeuroImage, 16, 593, 10.1006/nimg.2002.1096 · DOI
  68. Buxton, 2004, Modeling the hemodynamic response to brain activation, Neuroimage, 23, S220, 10.1016/j.neuroimage.2004.07.013 · DOI · Paper page
  69. Duckett, 2011, Invasive acute hemodynamic response to guide left ventricular lead implantation predicts chronic remodeling in patients undergoing cardiac resynchronization therapy, J. Am. Coll. Cardiol., 58, 1128, 10.1016/j.jacc.2011.04.042 · DOI
  70. Sun, Y., Ayaz, H., and Akansu, A.N. (2015, January 14–16). Neural correlates of affective context in facial expression analysis: A simultaneous EEG-fNIRS study. Proceedings of the 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Orlando, FL, USA. · DOI
  71. Morioka, 2014, Decoding spatial attention by using cortical currents estimated from electroencephalography with near-infrared spectroscopy prior information, Neuroimage, 90, 128, 10.1016/j.neuroimage.2013.12.035 · DOI
  72. Ahn, 2016, Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data, Front. Hum. Neurosci., 10, 219, 10.3389/fnhum.2016.00219 · DOI · Paper page
  73. Fazli, 2012, Enhanced performance by a hybrid NIRS–EEG brain computer interface, Neuroimage, 59, 519, 10.1016/j.neuroimage.2011.07.084 · DOI · Paper page
  74. Tomita, 2014, Bimodal BCI using simultaneously NIRS and EEG, IEEE Trans. Biomed. Eng., 61, 1274, 10.1109/TBME.2014.2300492 · DOI · Paper page
  75. Pistoia, 2015, EEG-detected olfactory imagery to reveal covert consciousness in minimally conscious state, Brain Inj., 29, 1729, 10.3109/02699052.2015.1075251 · doi:10.3109/02699052.2015.1075251
  76. Zander, 2011, Towards passive brain–computer interfaces: Applying brain–computer interface technology to human–machine systems in general, J. Neural Eng., 8, 025005, 10.1088/1741-2560/8/2/025005 · DOI · Paper page
  77. Lang, 1997, International affective picture system (IAPS): Technical manual and affective ratings, NIMH Center Study Emot. Atten., 1, 39
  78. Pan, J., Li, Y., and Wang, J. (2016, January 24–29). An EEG-based brain-computer interface for emotion recognition. Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada.
  79. Morris, 1995, Observations: SAM: The Self-Assessment Manikin; an efficient cross-cultural measurement of emotional response, J. Advert. Res., 35, 63
  80. Kim, 2008, Emotion recognition based on physiological changes in music listening, IEEE Trans. Pattern Anal. Mach. Intell., 30, 2067, 10.1109/TPAMI.2008.26 · DOI · Paper page
  81. Iacoviello, 2015, A real-time classification algorithm for EEG-based BCI driven by self-induced emotions, Comput. Methods Programs Biomed., 122, 293, 10.1016/j.cmpb.2015.08.011 · DOI · Paper page
  82. An, X., Höhne, J., Ming, D., and Blankertz, B. (2014). Exploring combinations of auditory and visual stimuli for gaze-independent brain-computer interfaces. PLoS ONE, 9. · DOI
  83. Wang, 2015, A novel audiovisual brain-computer interface and its application in awareness detection, Sci. Rep., 5, 1
  84. Gilleade, K., Dix, A., and Allanson, J. (2005, January 16–20). Affective videogames and modes of affective gaming: Assist me, challenge me, emote me. Proceedings of the International Conference on Changing Views, Vancouver, BC, Canada.
  85. Pan, 2020, Prognosis for patients with cognitive motor dissociation identified by brain-computer interface, Brain, 143, 1177, 10.1093/brain/awaa026 · DOI
  86. George, L., Lotte, F., Abad, R.V., and Lécuyer, A. (September, January 30). Using scalp electrical biosignals to control an object by concentration and relaxation tasks: Design and evaluation. Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA. · DOI
  87. Hjelm, 2003, Research+ Design: The Making of Brainball, Interactions, 10, 26, 10.1145/604575.604576 · doi:10.1145/604575.604576
  88. Brousseau, B., Rose, J., and Eizenman, M. (2020). Hybrid Eye-Tracking on a Smartphone with CNN Feature Extraction and an Infrared 3D Model. Sensors, 20. · doi:10.3390/s20020543
  89. Chen, 2016, Smart clothing: Connecting human with clouds and big data for sustainable health monitoring, Mob. Netw. Appl., 21, 825, 10.1007/s11036-016-0745-1 · DOI · Paper page
  90. Lim, J.Z., Mountstephens, J., and Teo, J. (2020). Emotion Recognition Using Eye-Tracking: Taxonomy, Review and Current Challenges. Sensors, 20. · DOI · Paper page
  91. Wabnitz, 2016, M3BA: A mobile, modular, multimodal biosignal acquisition architecture for miniaturized EEG-NIRS-based hybrid BCI and monitoring, IEEE Trans. Biomed. Eng., 64, 1199
  92. Scouten, 2017, Automated EEG artifact elimination by applying machine learning algorithms to ICA-based features, J. Neural Eng., 14, 046004, 10.1088/1741-2552/aa69d1 · DOI
  93. 2015, EEG artifact removal—State-of-the-art and guidelines, J. Neural Eng., 12, 031001, 10.1088/1741-2560/12/3/031001 · DOI · Paper page
  94. Islam, 2016, Methods for artifact detection and removal from scalp EEG: A review, Neurophysiol. Clin. Clin. Neurophysiol., 46, 287, 10.1016/j.neucli.2016.07.002 · DOI · Paper page
  95. Jung, 2000, Removing electroencephalographic artifacts by blind source separation, Psychophysiology, 37, 163, 10.1111/1469-8986.3720163 · doi:10.1111/1469-8986.3720163 · Paper page
  96. Dong, 2015, Characterizing nonlinear relationships in functional imaging data using eigenspace maximal information canonical correlation analysis (emiCCA), Neuroimage, 109, 388, 10.1016/j.neuroimage.2015.01.006 · DOI
  97. Makeig, S., Bell, A.J., Jung, T.-P., and Sejnowski, T.J. (1996, January 2–5). Independent component analysis of electroencephalographic data. Proceedings of the Advances in Neural Information Processing Systems, Denver, CO, USA.
  98. Li, M., and Lu, B.-L. (2009, January 2–6). Emotion classification based on gamma-band EEG. Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA.
  99. Zheng, 2015, Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks, IEEE Trans. Auton. Ment. Dev., 7, 162, 10.1109/TAMD.2015.2431497 · DOI · Paper page
  100. Samara, A., Menezes, M.L.R., and Galway, L. (2016, January 14–16). Feature extraction for emotion recognition and modelling using neurophysiological data. Proceedings of the 2016 15th International Conference on Ubiquitous Computing and Communications and 2016 International Symposium on Cyberspace and Security (IUCC-CSS), Granada, Spain. · DOI
  101. Li, Z., Qiu, L., Li, R., He, Z., Xiao, J., Liang, Y., Wang, F., and Pan, J. (2020). Enhancing BCI-Based Emotion Recognition Using an Improved Particle Swarm Optimization for Feature Selection. Sensors, 20. · DOI
  102. Gharavian, 2012, Speech emotion recognition using FCBF feature selection method and GA-optimized fuzzy ARTMAP neural network, Neural Comput. Appl., 21, 2115, 10.1007/s00521-011-0643-1 · DOI
  103. Nawaz, 2020, Comparison of different feature extraction methods for EEG-based emotion recognition, Biocybern. Biomed. Eng., 40, 910, 10.1016/j.bbe.2020.04.005 · DOI · Paper page
  104. Kang, J.-S., Kavuri, S., and Lee, M. (2019). ICA-evolution based data augmentation with ensemble deep neural networks using time and frequency kernels for emotion recognition from EEG-data. IEEE Trans. Affect. Comput.
  105. Aithal, 2015, Emotion Detection from Distorted Speech Signal using PCA-Based Technique, Emotion, 2, 14
  106. Du, C., Du, C., Wang, H., Li, J., Zheng, W.L., Lu, B.L., and He, H. (2018, January 22–26). Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data. Proceedings of the ACM Multimedia Conference, Seoul, Korea. · DOI · Paper page
  107. Sangineto, E., Zen, G., Ricci, E., and Sebe, N. (2014, January 3–7). We are not all equal: Personalizing models for facial expression analysis with transductive parameter transfer. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA. · DOI
  108. Li, 2020, Domain Adaptation for EEG Emotion Recognition Based on Latent Representation Similarity, IEEE Trans. Cogn. Dev. Syst., 12, 344, 10.1109/TCDS.2019.2949306 · DOI · Paper page
  109. Joshi, 2013, Multimodal assistive technologies for depression diagnosis and monitoring, J. Multimodal User Interfaces, 7, 217, 10.1007/s12193-013-0123-2 · DOI · Paper page
  110. Gonzalez, H.A., Yoo, J., and Elfadel, I.A.M. (2019, January 23–27). EEG-based Emotion Detection Using Unsupervised Transfer Learning. Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany. · DOI
  111. Elatlassi, R. (2018). Modeling Student Engagement in Online Learning Environments Using Real-Time Biometric Measures: Electroencephalography (EEG) and Eye-Tracking. [Master’s Thesis, Oregon State University].
  112. Park, C., Shahrdar, S., and Nojoumian, M. (2018, January 8–11). EEG-based classification of emotional state using an autonomous vehicle simulator. Proceedings of the 2018 IEEE 10th Sensor Array and Multichannel Signal Processing Workshop (SAM), Sheffield, UK. · DOI
  113. Sourina, O., Wang, Q., Liu, Y., and Nguyen, M.K. (2011, January 26–29). A Real-time Fractal-based Brain State Recognition from EEG and its Applications. Proceedings of the BIOSIGNALS 2011—The International Conference on Bio-Inspired Systems and Signal Processing, Rome, Italy.

Cited by

See all 151

Metadata

Authors
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J.
Venue
Brain Sciences
Publisher
MDPI AG
Published
2020-09-29
Volume
10
Issue
10
Pages
687
ISSN
2076-3425 (ISSN-L)
OpenAlex
W3089475900
Cited by
151
References
19
FWCI
13.65
Type
journal article
Language
en
Open access
gold
Funders
  • Natural Science Foundation of Guangdong Province (2019A1515011375)
  • National Natural Science Foundation of China (61876067)
  • Research and Development Plan in Key Areas of Guangzhou Science and Technology Plan Project (202007030005)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces. https://app.cochat.ai/papers/10.3390/brainsci10100687
The paper
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J. (2020). Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces. Brain Sciences. https://doi.org/10.3390/brainsci10100687

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.