Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
With the continuous development of portable noninvasive human sensor technologies such as brain–computer interfaces (BCI), multimodal emotion recognition has attracted increasing attention in the area of affective computing. This paper primarily discusses the progress of research into multimodal emotion recognition based on BCI and reviews three types of multimodal affective BCI (aBCI): aBCI based on a combination of behavior and brain signals, aBCI based on various hybrid neurophysiology modalities and aBCI based on heterogeneous sensory stimuli. For each type of aBCI, we further review several representative multimodal aBCI systems, including their design principles, paradigms, algorithms, experimental results and corresponding advantages. Finally, we identify several important issues and research directions for multimodal emotion recognition based on BCI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“Eye movement tracking signals can provide a variety of eye movement indicators and eye movement tracking signals can reflect user subconscious behaviors and provide important clues to the context of the subjects’ current activity [48].”
BackgroundBrain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
The pupil as a measure of emotional arousal and autonomic activation
Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J. (2008) · Psychophysiology · doi:10.1111/j.1469-8986.2008.00654.x · Cited by 1,919
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1×
“we believe that multimodal aBCI systems based on EEG signals can greatly improve the results of emotion recognition [14,15].”
BackgroundA Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Moving magnetoencephalography towards real-world applications with a wearable system
Boto, E., Holmes, N., Leggett, J., Roberts, G., Shah, V., Meyer, S. S., Muñoz, L. D., Mullinger, K. J., Tierney, T. M., Bestmann, S., et al. (2018) · Nature · doi:10.1038/nature26147 · Cited by 1,377 · cited 1×
“A team of researchers [107] at the University of Nottingham in the UK has developed a magnetoencephalography system that can be worn like a helmet that allows the scanned person to move freely and naturally during the scanning process.”
BackgroundEmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.12.005 · Cited by 416 · cited 2×
“[21] DEAP (EEG + PPS) EEG: power differences, statistical PPS: statistical feature, frequency–domain feature.”
Background
Research works printed but not linked: 113
- Mühl, C., Nijholt, A., Allison, B., Dunne, S., and Heylen, D. (2011, January 9–12). Affective brain-computer interfaces (aBCI 2011). Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Memphis, TN, USA. · DOI
- Allison, 2014, A survey of affective brain computer interfaces: Principles, state-of-the-art, and challenges, Brain-Comput. Interfaces, 1, 66, 10.1080/2326263X.2014.912881 · DOI · Paper page
- Van den Broek, E.L. (2012). Affective computing: A reverence for a century of research. Cognitive Behavioural Systems, Springer. · DOI
- Ekman, P.E., and Davidson, R.J. (1994). The Nature of Emotion: Fundamental Questions, Oxford University Press.
- Russell, 1980, A circumplex model of affect, J. Personal. Soc. Psychol., 39, 1161, 10.1037/h0077714 · DOI · Paper page
- Niemic, 2004, Studies of Emotion: A Theoretical and Empirical Review of Psychophysiological Studies of Emotion, Psychophy, 1, 15
- Chanel, 2009, Short-term emotion assessment in a recall paradigm, Int. J. Hum.-Comput. Stud., 67, 607, 10.1016/j.ijhcs.2009.03.005 · DOI · Paper page
- Huang, H., Xie, Q., Pan, J., He, Y., Wen, Z., Yu, R., and Li, Y. (2019). An EEG-based brain computer interface for emotion recognition and its application in patients with Disorder of Consciousness. IEEE Trans. Affect. Comput.
- Wang, 2017, Enhancing clinical communication assessments using an audiovisual BCI for patients with disorders of consciousness, J. Neural Eng., 14, 046024, 10.1088/1741-2552/aa6c31 · DOI
- Kory, 2015, A review and meta-analysis of multimodal affect detection systems, ACM Comput. Surv. (CSUR), 47, 1
- Ebrahimi, 2003, Guest editorial braincomputer interface technology: A review of the second international meeting, IEEE Signal Process. Mag., 20, 14, 10.1109/MSP.2003.1166626 · DOI
- Phan, 2010, Tensor decompositions for feature extraction and classification of high dimensional datasets, Nonlinear Theory Its Appl. IEICE, 1, 37, 10.1587/nolta.1.37 · DOI · Paper page
- Congedo, 2017, Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review, Brain-Comput. Interfaces, 4, 155, 10.1080/2326263X.2017.1297192 · DOI · Paper page
- Soleymani, 2011, Multimodal emotion recognition in response to videos, IEEE Trans. Affect. Comput., 3, 211, 10.1109/T-AFFC.2011.37 · DOI · Paper page
- Liu, W., Zheng, W.-L., and Lu, B.-L. (2016, January 16–21). Emotion recognition using multimodal deep learning. Proceedings of the International Conference on Neural Information Processing, Kyoto, Japan. · DOI · Paper page
- Minotto, 2015, Multimodal multi-channel on-line speaker diarization using sensor fusion through SVM, IEEE Trans. Multimed., 17, 1694, 10.1109/TMM.2015.2463722 · DOI
- Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12–15). Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain. · DOI · Paper page
- Haghighat, 2016, Discriminant correlation analysis: Real-time feature level fusion for multimodal biometric recognition, IEEE Trans. Inf. Forensics Secur., 11, 1984, 10.1109/TIFS.2016.2569061 · DOI · Paper page
- Zhalehpour, 2016, BAUM-1: A spontaneous audio-visual face database of affective and mental states, IEEE Trans. Affect. Comput., 8, 300, 10.1109/TAFFC.2016.2553038 · DOI · Paper page
- Wu, 2016, A novel lip descriptor for audio-visual keyword spotting based on adaptive decision fusion, IEEE Trans. Multimed., 18, 326, 10.1109/TMM.2016.2520091 · DOI
- Koelstra, 2013, Fusion of facial expressions and EEG for implicit affective tagging, Image Vis. Comput., 31, 164, 10.1016/j.imavis.2012.10.002 · DOI · Paper page
- Soleymani, M., Asghari-Esfeden, S., Pantic, M., and Fu, Y. (2014, January 14–18). Continuous emotion detection using EEG signals and facial expressions. Proceedings of the 2014 IEEE International Conference on Multimedia and Expo (ICME), Chengdu, China. · DOI · Paper page
- Freund, Y., and Schapire, R.E. (1996, January 3–6). Experiments with a new boosting algorithm. Proceedings of the ICML, Bari, Italy.
- Ponti, M.P. (2011, January 28–30). Combining classifiers: From the creation of ensembles to the decision fusion. Proceedings of the 2011 24th SIBGRAPI Conference on Graphics, Patterns, and Images Tutorials, Alagoas, Brazil. · DOI
- Chang, Z., Liao, X., Liu, Y., and Wang, W. (2011, January 19–21). Research of decision fusion for multi-source remote-sensing satellite information based on SVMs and DS evidence theory. Proceedings of the Fourth International Workshop on Advanced Computational Intelligence, Wuhan, China. · doi:10.1109/iwaci.2011.6160042
- Nefian, 2002, Dynamic Bayesian networks for audio-visual speech recognition, EURASIP J. Adv. Signal Process., 2002, 783042, 10.1155/S1110865702206083 · DOI · Paper page
- Murofushi, 1989, An interpretation of fuzzy measures and the Choquet integral as an integral with respect to a fuzzy measure, Fuzzy Sets Syst., 29, 201, 10.1016/0165-0114(89)90194-2 · DOI
- Savran, A., Ciftci, K., Chanel, G., Mota, J., Hong Viet, L., Sankur, B., Akarun, L., Caplier, A., and Rombaut, M. (August, January 17). Emotion detection in the loop from brain signals and facial images. Proceedings of the eNTERFACE 2006 Workshop, Dubrovnik, Croatia.
- Correa, J.A.M., Abadi, M.K., Sebe, N., and Patras, I. (2018). Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE Trans. Affect. Comput.
- Conneau, A.-C., Hajlaoui, A., Chetouani, M., and Essid, S. (September, January 28). EMOEEG: A new multimodal dataset for dynamic EEG-based emotion recognition with audiovisual elicitation. Proceedings of the 2017 25th European Signal Processing Conference (EUSIPCO), Kos, Greece. · DOI
- Subramanian, 2016, ASCERTAIN: Emotion and personality recognition using commercial sensors, IEEE Trans. Affect. Comput., 9, 147, 10.1109/TAFFC.2016.2625250 · DOI · Paper page
- Song, 2019, MPED: A multi-modal physiological emotion database for discrete emotion recognition, IEEE Access, 7, 12177, 10.1109/ACCESS.2019.2891579 · DOI · Paper page
- Liu, W., Zheng, W.-L., and Lu, B.-L. (2016). Multimodal emotion recognition using multimodal deep learning. arXiv. · DOI · Paper page
- Bulling, 2010, Eye movement analysis for activity recognition using electrooculography, IEEE Trans. Pattern Anal. Mach. Intell., 33, 741, 10.1109/TPAMI.2010.86 · DOI · Paper page
- Bakshi, 2014, A survey on face detection methods and feature extraction techniques of face recognition, Int. J. Emerg. Trends Technol. Comput. Sci., 3, 233
- Jacobs, 2008, The coupling of emotion and cognition in the eye: Introducing the pupil old/new effect, Psychophysiology, 45, 130, 10.1111/j.1469-8986.2007.00606.x · DOI · Paper page
- Wang, 2015, A circuit for pupil orienting responses: Implications for cognitive modulation of pupil size, Curr. Opin. Neurobiol., 33, 134, 10.1016/j.conb.2015.03.018 · DOI · Paper page
- Chang, 2011, LIBSVM: A library for support vector machines, ACM Trans. Intell. Syst. Technol. (TIST), 2, 1, 10.1145/1961189.1961199 · DOI · Paper page
- Wu, X., Zheng, W.-L., and Lu, B.-L. (2020). Investigating EEG-Based Functional Connectivity Patterns for Multimodal Emotion Recognition. arXiv. · DOI
- Xu, X., Quan, C., and Ren, F. (2015, January 2–5). Facial expression recognition based on Gabor Wavelet transform and Histogram of Oriented Gradients. Proceedings of the 2015 IEEE International Conference on Mechatronics and Automation (ICMA), Beijing, China. · DOI
- Huang, X., He, Q., Hong, X., Zhao, G., and Pietikainen, M. (2014, January 12–16). Improved spatiotemporal local monogenic binary pattern for emotion recognition in the wild. Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey. · DOI
- Saeed, A., Al-Hamadi, A., and Niese, R. (2013, January 13–15). The effectiveness of using geometrical features for facial expression recognition. Proceedings of the 2013 IEEE International Conference on Cybernetics (CYBCO), Lausanne, Switzerland. · DOI
- Ekman, R. (1997). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press. · DOI · Paper page
- Sokolov, S., Velchev, Y., Radeva, S., and Radev, D. (2017, January 25–26). Human Emotion Estimation from EEG and Face Using Statistical Features and SVM. Proceedings of the Fourth International Conference on Computer Science and Information Technology, Geneva, Switzerland. · DOI
- Chang, C.Y., Tsai, J.S., Wang, C.J., and Chung, P.C. (April, January 30). Emotion recognition with consideration of facial expression and physiological signals. Proceedings of the IEEE Conference on Computational Intelligence in Bioinformatics & Computational Biology, Nashville, TN, USA.
- Chunawale, A., and Bedekar, D. (2020, January 20). Human Emotion Recognition using Physiological Signals: A Survey. Proceedings of the 2nd International Conference on Communication & Information Processing (ICCIP), Tokyo, Japan. · DOI
- Vijayakumar, S., Flynn, R., and Murray, N. (2020, January 11–12). A comparative study of machine learning techniques for emotion recognition from peripheral physiological signals. Proceedings of the ISSC 2020 31st Irish Signals and System Conference, Letterkenny, Ireland. · DOI
- Liu, Y., and Sourina, O. (2014). Real-Time Subject-Dependent EEG-Based Emotion Recognition Algorithm. Transactions on Computational Science XXIII, Springer. · DOI
- Bo, 1970, EEG analysis based on time domain properties, Electroencephalogr. Clin. Neurophysiol., 29, 306, 10.1016/0013-4694(70)90143-4 · DOI · Paper page
- Thammasan, 2016, Continuous Music-Emotion Recognition Based on Electroencephalogram, IEICE Trans. Inf. Syst., 99, 1234, 10.1587/transinf.2015EDP7251 · DOI
- Shi, L.C., Jiao, Y.Y., and Lu, B.L. (2013, January 3–7). Differential entropy feature for EEG-based vigilance estimation. Proceedings of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan.
- Zheng, 2017, Identifying stable patterns over time for emotion recognition from EEG, IEEE Trans. Affect. Comput., 10, 417, 10.1109/TAFFC.2017.2712143 · DOI · Paper page
- Akin, 2002, Comparison of Wavelet Transform and FFT Methods in the Analysis of EEG Signals, J. Med. Syst., 26, 241, 10.1023/A:1015075101937 · doi:10.1023/a:1015075101937
- 2005, Comparison of STFT and wavelet transform methods in determining epileptic seizure activity in EEG signals for real-time application, Comput. Biol. Med., 35, 603, 10.1016/j.compbiomed.2004.05.001 · DOI · Paper page
- Hadjidimitriou, 2012, Toward an EEG-Based Recognition of Music Liking Using Time-Frequency Analysis, IEEE Trans. Biomed. Eng., 59, 3498, 10.1109/TBME.2012.2217495 · DOI · Paper page
- Liao, 2020, Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP Conf. Ser. Mater. Sci. Eng., 782, 032005, 10.1088/1757-899X/782/3/032005 · DOI
- 2018, Electroencephalography Based Fusion Two-Dimensional (2D)-Convolution Neural Networks (CNN) Model for Emotion Recognition System, Sensors, 18, 1383, 10.3390/s18051383 · DOI · Paper page
- Rahimi, 2017, High-Dimensional Computing as a Nanoscalable Paradigm, IEEE Trans. Circuits Syst. I Regul. Pap., 64, 2508, 10.1109/TCSI.2017.2705051 · DOI · Paper page
- Montagna, F., Rahimi, A., Benatti, S., Rossi, D., and Benini, L. (2018, January 24–28). PULP-HD: Accelerating Brain-Inspired High-Dimensional Computing on a Parallel Ultra-Low Power Platform. Proceedings of the 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), San Francisco, CA, USA. · DOI
- Rahimi, A., Kanerva, P., Millán, J.D.R., and Rabaey, J.M. (2017, January 15–17). Hyperdimensional Computing for Noninvasive Brain–Computer Interfaces: Blind and One-Shot Classification of EEG Error-Related Potentials. Proceedings of the 10th EAI International Conference on Bio-Inspired Information and Communications Technologies (Formerly BIONETICS), Hoboken, NJ, USA. · DOI
- Rahimi, 2019, Efficient Biosignal Processing Using Hyperdimensional Computing: Network Templates for Combined Learning and Classification of ExG Signals, Proc. IEEE, 107, 123, 10.1109/JPROC.2018.2871163 · DOI · Paper page
- Chang, E.-J., Rahimi, A., Benini, L., and Wu, A.-Y.A. (2019, January 18–20). Hyperdimensional Computing-based Multimodality Emotion Recognition with Physiological Signals. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), Hsinchu, Taiwan. · DOI · Paper page
- Huppert, 2006, A temporal comparison of BOLD, ASL, and NIRS hemodynamic responses to motor stimuli in adult humans, Neuroimage, 29, 368, 10.1016/j.neuroimage.2005.08.065 · DOI · Paper page
- Jasdzewski, 2003, Differences in the hemodynamic response to event-related motor and visual paradigms as measured by near-infrared spectroscopy, Neuroimage, 20, 479, 10.1016/S1053-8119(03)00311-2 · DOI · Paper page
- Malonek, 1996, Interactions between electrical activity and cortical microcirculation revealed by imaging spectroscopy: Implications for functional brain mapping, Science, 272, 551, 10.1126/science.272.5261.551 · DOI · Paper page
- Ayaz, 2011, Using MazeSuite and functional near infrared spectroscopy to study learning in spatial navigation, J. Vis. Exp., 56, e3443
- Liao, 2002, Estimating the delay of the fMRI response, NeuroImage, 16, 593, 10.1006/nimg.2002.1096 · DOI
- Buxton, 2004, Modeling the hemodynamic response to brain activation, Neuroimage, 23, S220, 10.1016/j.neuroimage.2004.07.013 · DOI · Paper page
- Duckett, 2011, Invasive acute hemodynamic response to guide left ventricular lead implantation predicts chronic remodeling in patients undergoing cardiac resynchronization therapy, J. Am. Coll. Cardiol., 58, 1128, 10.1016/j.jacc.2011.04.042 · DOI
- Sun, Y., Ayaz, H., and Akansu, A.N. (2015, January 14–16). Neural correlates of affective context in facial expression analysis: A simultaneous EEG-fNIRS study. Proceedings of the 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Orlando, FL, USA. · DOI
- Morioka, 2014, Decoding spatial attention by using cortical currents estimated from electroencephalography with near-infrared spectroscopy prior information, Neuroimage, 90, 128, 10.1016/j.neuroimage.2013.12.035 · DOI
- Ahn, 2016, Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data, Front. Hum. Neurosci., 10, 219, 10.3389/fnhum.2016.00219 · DOI · Paper page
- Fazli, 2012, Enhanced performance by a hybrid NIRS–EEG brain computer interface, Neuroimage, 59, 519, 10.1016/j.neuroimage.2011.07.084 · DOI · Paper page
- Tomita, 2014, Bimodal BCI using simultaneously NIRS and EEG, IEEE Trans. Biomed. Eng., 61, 1274, 10.1109/TBME.2014.2300492 · DOI · Paper page
- Pistoia, 2015, EEG-detected olfactory imagery to reveal covert consciousness in minimally conscious state, Brain Inj., 29, 1729, 10.3109/02699052.2015.1075251 · doi:10.3109/02699052.2015.1075251
- Zander, 2011, Towards passive brain–computer interfaces: Applying brain–computer interface technology to human–machine systems in general, J. Neural Eng., 8, 025005, 10.1088/1741-2560/8/2/025005 · DOI · Paper page
- Lang, 1997, International affective picture system (IAPS): Technical manual and affective ratings, NIMH Center Study Emot. Atten., 1, 39
- Pan, J., Li, Y., and Wang, J. (2016, January 24–29). An EEG-based brain-computer interface for emotion recognition. Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada.
- Morris, 1995, Observations: SAM: The Self-Assessment Manikin; an efficient cross-cultural measurement of emotional response, J. Advert. Res., 35, 63
- Kim, 2008, Emotion recognition based on physiological changes in music listening, IEEE Trans. Pattern Anal. Mach. Intell., 30, 2067, 10.1109/TPAMI.2008.26 · DOI · Paper page
- Iacoviello, 2015, A real-time classification algorithm for EEG-based BCI driven by self-induced emotions, Comput. Methods Programs Biomed., 122, 293, 10.1016/j.cmpb.2015.08.011 · DOI · Paper page
- An, X., Höhne, J., Ming, D., and Blankertz, B. (2014). Exploring combinations of auditory and visual stimuli for gaze-independent brain-computer interfaces. PLoS ONE, 9. · DOI
- Wang, 2015, A novel audiovisual brain-computer interface and its application in awareness detection, Sci. Rep., 5, 1
- Gilleade, K., Dix, A., and Allanson, J. (2005, January 16–20). Affective videogames and modes of affective gaming: Assist me, challenge me, emote me. Proceedings of the International Conference on Changing Views, Vancouver, BC, Canada.
- Pan, 2020, Prognosis for patients with cognitive motor dissociation identified by brain-computer interface, Brain, 143, 1177, 10.1093/brain/awaa026 · DOI
- George, L., Lotte, F., Abad, R.V., and Lécuyer, A. (September, January 30). Using scalp electrical biosignals to control an object by concentration and relaxation tasks: Design and evaluation. Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA. · DOI
- Hjelm, 2003, Research+ Design: The Making of Brainball, Interactions, 10, 26, 10.1145/604575.604576 · doi:10.1145/604575.604576
- Brousseau, B., Rose, J., and Eizenman, M. (2020). Hybrid Eye-Tracking on a Smartphone with CNN Feature Extraction and an Infrared 3D Model. Sensors, 20. · doi:10.3390/s20020543
- Chen, 2016, Smart clothing: Connecting human with clouds and big data for sustainable health monitoring, Mob. Netw. Appl., 21, 825, 10.1007/s11036-016-0745-1 · DOI · Paper page
- Lim, J.Z., Mountstephens, J., and Teo, J. (2020). Emotion Recognition Using Eye-Tracking: Taxonomy, Review and Current Challenges. Sensors, 20. · DOI · Paper page
- Wabnitz, 2016, M3BA: A mobile, modular, multimodal biosignal acquisition architecture for miniaturized EEG-NIRS-based hybrid BCI and monitoring, IEEE Trans. Biomed. Eng., 64, 1199
- Scouten, 2017, Automated EEG artifact elimination by applying machine learning algorithms to ICA-based features, J. Neural Eng., 14, 046004, 10.1088/1741-2552/aa69d1 · DOI
- 2015, EEG artifact removal—State-of-the-art and guidelines, J. Neural Eng., 12, 031001, 10.1088/1741-2560/12/3/031001 · DOI · Paper page
- Islam, 2016, Methods for artifact detection and removal from scalp EEG: A review, Neurophysiol. Clin. Clin. Neurophysiol., 46, 287, 10.1016/j.neucli.2016.07.002 · DOI · Paper page
- Jung, 2000, Removing electroencephalographic artifacts by blind source separation, Psychophysiology, 37, 163, 10.1111/1469-8986.3720163 · doi:10.1111/1469-8986.3720163 · Paper page
- Dong, 2015, Characterizing nonlinear relationships in functional imaging data using eigenspace maximal information canonical correlation analysis (emiCCA), Neuroimage, 109, 388, 10.1016/j.neuroimage.2015.01.006 · DOI
- Makeig, S., Bell, A.J., Jung, T.-P., and Sejnowski, T.J. (1996, January 2–5). Independent component analysis of electroencephalographic data. Proceedings of the Advances in Neural Information Processing Systems, Denver, CO, USA.
- Li, M., and Lu, B.-L. (2009, January 2–6). Emotion classification based on gamma-band EEG. Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA.
- Zheng, 2015, Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks, IEEE Trans. Auton. Ment. Dev., 7, 162, 10.1109/TAMD.2015.2431497 · DOI · Paper page
- Samara, A., Menezes, M.L.R., and Galway, L. (2016, January 14–16). Feature extraction for emotion recognition and modelling using neurophysiological data. Proceedings of the 2016 15th International Conference on Ubiquitous Computing and Communications and 2016 International Symposium on Cyberspace and Security (IUCC-CSS), Granada, Spain. · DOI
- Li, Z., Qiu, L., Li, R., He, Z., Xiao, J., Liang, Y., Wang, F., and Pan, J. (2020). Enhancing BCI-Based Emotion Recognition Using an Improved Particle Swarm Optimization for Feature Selection. Sensors, 20. · DOI
- Gharavian, 2012, Speech emotion recognition using FCBF feature selection method and GA-optimized fuzzy ARTMAP neural network, Neural Comput. Appl., 21, 2115, 10.1007/s00521-011-0643-1 · DOI
- Nawaz, 2020, Comparison of different feature extraction methods for EEG-based emotion recognition, Biocybern. Biomed. Eng., 40, 910, 10.1016/j.bbe.2020.04.005 · DOI · Paper page
- Kang, J.-S., Kavuri, S., and Lee, M. (2019). ICA-evolution based data augmentation with ensemble deep neural networks using time and frequency kernels for emotion recognition from EEG-data. IEEE Trans. Affect. Comput.
- Aithal, 2015, Emotion Detection from Distorted Speech Signal using PCA-Based Technique, Emotion, 2, 14
- Du, C., Du, C., Wang, H., Li, J., Zheng, W.L., Lu, B.L., and He, H. (2018, January 22–26). Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data. Proceedings of the ACM Multimedia Conference, Seoul, Korea. · DOI · Paper page
- Sangineto, E., Zen, G., Ricci, E., and Sebe, N. (2014, January 3–7). We are not all equal: Personalizing models for facial expression analysis with transductive parameter transfer. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA. · DOI
- Li, 2020, Domain Adaptation for EEG Emotion Recognition Based on Latent Representation Similarity, IEEE Trans. Cogn. Dev. Syst., 12, 344, 10.1109/TCDS.2019.2949306 · DOI · Paper page
- Joshi, 2013, Multimodal assistive technologies for depression diagnosis and monitoring, J. Multimodal User Interfaces, 7, 217, 10.1007/s12193-013-0123-2 · DOI · Paper page
- Gonzalez, H.A., Yoo, J., and Elfadel, I.A.M. (2019, January 23–27). EEG-based Emotion Detection Using Unsupervised Transfer Learning. Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany. · DOI
- Elatlassi, R. (2018). Modeling Student Engagement in Online Learning Environments Using Real-Time Biometric Measures: Electroencephalography (EEG) and Eye-Tracking. [Master’s Thesis, Oregon State University].
- Park, C., Shahrdar, S., and Nojoumian, M. (2018, January 8–11). EEG-based classification of emotional state using an autonomous vehicle simulator. Proceedings of the 2018 IEEE 10th Sensor Array and Multichannel Signal Processing Workshop (SAM), Sheffield, UK. · DOI
- Sourina, O., Wang, Q., Liu, Y., and Nguyen, M.K. (2011, January 26–29). A Real-time Fractal-based Brain State Recognition from EEG and its Applications. Proceedings of the BIOSIGNALS 2011—The International Conference on Bio-Inspired Systems and Signal Processing, Rome, Italy.
Cited by
Houssein, E. H., Hammad, A., & Ali, A. A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07292-4 · Cited by 426
Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3240280 · Cited by 121
Self-citationA multimodal emotion recognition method based on facial expressions and electroencephalography
Tan, Y., Sun, Z., Duan, F., Solé-Casals, J., & Caiafa, C. F. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103029 · Cited by 88
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Muhammad, F., Hussain, M., & Aboalsamh, H. (2023) · Diagnostics · doi:10.3390/diagnostics13050977 · Cited by 36
“The methods based on physiological signals attain high accuracy in recognizing emotions because these signals can accurately reflect the emotional mood of the subject [27].”
Metadata
- Authors
- He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J.
- Venue
- Brain Sciences
- Publisher
- MDPI AG
- Published
- 2020-09-29
- Volume
- 10
- Issue
- 10
- Pages
- 687
- ISSN
- 2076-3425 (ISSN-L)
- OpenAlex
- W3089475900
- Cited by
- 151
- References
- 19
- FWCI
- 13.65
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Natural Science Foundation of Guangdong Province (2019A1515011375)
- National Natural Science Foundation of China (61876067)
- Research and Development Plan in Key Areas of Guangzhou Science and Technology Plan Project (202007030005)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces. https://app.cochat.ai/papers/10.3390/brainsci10100687
- The paper
- He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J. (2020). Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces. Brain Sciences. https://doi.org/10.3390/brainsci10100687
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.