Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals
Abstract
In this study, we proposed a new adaptive method for fusing multiple emotional modalities to improve the performance of the emotion recognition system. Three-channel forehead biosignals along with peripheral physiological measurements (blood volume pressure, skin conductance, and interbeat intervals) were utilized as emotional modalities. Six basic emotions, i.e., anger, sadness, fear, disgust, happiness, and surprise were elicited by displaying preselected video clips for each of the 25 participants in the experiment; the physiological signals were collected simultaneously. In our multimodal emotion recognition system, recorded signals with the formation of several classification units identified the emotions independently. Then the results were fused using the adaptive weighted linear model to produce the final result. Each classification unit is assigned a weight that is determined dynamically by considering the performance of the units during the testing phase and the training phase results. This dynamic weighting scheme enables the emotion recognition system to adapt itself to each new user. The results showed that the suggested method outperformed conventional fusion of the features and classification units using the majority voting method. In addition, a considerable improvement, compared to the systems that used the static weighting schemes for fusing classification units, was also shown. Using support vector machine (SVM) and k-nearest neighbors (KNN) classifiers, the overall classification accuracies of 84.7% and 80% were obtained in identifying the emotions, respectively. In addition, applying the forehead or physiological signals in the proposed scheme indicates that designing a reliable emotion recognition system is feasible without the need for additional emotional modalities.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous studies found general recognition rates from 57.77% to 85.46% for different emotional states.
“Previous studies have provided evidence for general recognition rate from 57,77% to 85,46% for different emotional states”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [64]
A larger set of analyzed signals and more recognized emotions provides better results.
“that more higher set of analyzed signals and more recognized emotions provides better results [64][65][66].”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [64]
Cites
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · see citing works
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515 · see citing works
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382 · see citing works
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,312 · see citing works
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · see citing works
A Practical Guide to Support Vector Classication
Hsu, C., Chang, C.-C., & Lin, C. (2008) · Cited by 5,070 · see citing works
Floating search methods in feature selection
Pudil, P., Novovičová, J., & Kittler, J. (1994) · Pattern Recognition Letters · doi:10.1016/0167-8655(94)90127-9 · Cited by 3,129 · see citing works
The emotion probe. Studies of motivation and attention.
Lang, P. J. (1995) · American Psychologist · doi:10.1037/0003-066x.50.5.372 · Cited by 2,846 · see citing works
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797 · see citing works
The emotion probe: Studies of motivation and attention.
Lang, P. J. (1995) · American Psychologist · doi:10.1037//0003-066x.50.5.372 · Cited by 2,752 · see citing works
Autonomic nervous system activity distinguishes among emotions
Ekman, P., Levenson, R. W., & Friesen, W. V. (1983) · Science · doi:10.1126/science.6612338 · Cited by 2,692 · see citing works
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · see citing works
Emotion elicitation using films
Gross, J. J., & Levenson, R. W. (1995) · Cognition & Emotion · doi:10.1080/02699939508408966 · Cited by 2,179 · see citing works
BioelectromagnetismPrinciples and Applications of Bioelectric and Biomagnetic Fields
Malmivuo, J., & Plonsey, R. (1995) · doi:10.1093/acprof:oso/9780195058239.001.0001 · Cited by 1,834 · see citing works
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763 · see citing works
Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I.
Davidson, R. J., Ekman, P., Saron, C. D., Senulis, J. A., & Friesen, W. V. (1990) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.58.2.330 · Cited by 1,638 · see citing works
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279 · see citing works
The Handbook of emotion elicitation and assessment
(2008) · Choice Reviews Online · doi:10.5860/choice.46-1769 · Cited by 1,127 · see citing works
Emotion recognition based on physiological changes in music listening
Kim, J., & André, E. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.26 · Cited by 1,079 · see citing works
EEG-Based Emotion Recognition in Music Listening
Lin, Y., Wang, C., Jung, T., Wu, T.-L., Jeng, S., Duann, J., & Chen, J. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2048568 · Cited by 1,022 · see citing works
Affective computing: challenges
Picard, R. W. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00052-1 · Cited by 978 · see citing works
The psychophysiology of emotion.
Larsen, J. T., Berntson, G. G., Poehlmann, K. M., Ito, T. A., & Cacioppo, J. T. (1993) · Cited by 957 · see citing works
Emotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944 · see citing works
Emotion Elicitation Using Films
Rottenberg, J. A., Ray, R. D., & Gross, J. J. (2007) · Oxford University Press eBooks · doi:10.1093/oso/9780195169157.003.0002 · Cited by 862 · see citing works
Single-layer learning revisited: a stepwise procedure for building and training a neural network
Knerr, S., Personnaz, L., & Dreyfus, G. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_5 · Cited by 837 · see citing works
A theoretical study on six classifier fusion strategies
Kuncheva, L. I. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982906 · Cited by 724 · see citing works
Emotion Recognition From EEG Using Higher Order Crossings
Petrantonakis, P. C., & Hadjileontiadis, L. J. (2009) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2034649 · Cited by 687 · see citing works
Control of Multifunctional Prosthetic Hands by Processing the Electromyographic Signal
Zecca, M., Micera, S., Carrozza, M. C., & Dario, P. (2002) · Critical Reviews? in Biomedical Engineering · doi:10.1615/critrevbiomedeng.v30.i456.80 · Cited by 606 · see citing works
Control of Multifunctional Prosthetic Hands by Processing the Electromyographic Signal
Zecca, M., Micera, S., Carrozza, M. C., & Dario, P. (2017) · Critical Reviews in Biomedical Engineering · doi:10.1615/critrevbiomedeng.v45.i1-6.150 · Cited by 569 · see citing works
User Modeling for Adaptive News Access
Billsus, D., & Pazzani, M. J. (2000) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1026501525781 · Cited by 491 · see citing works
Ecg pattern analysis for emotion detection
Agrafioti, F., Hatzinakos, D., & Anderson, A. K. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.28 · Cited by 476 · see citing works
Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System
Haag, A., Goronzy, S., Schaich, P., & Williams, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24842-2_4 · Cited by 451 · see citing works
The production and recognition of emotions in speech: features and algorithms
Pierre-Yves, O. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(02)00141-6 · Cited by 439 · see citing works
Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty
Chanel, G., Rebetez, C., Bétrancourt, M., & Pun, T. (2011) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2011.2116000 · Cited by 404 · see citing works
Emotion recognition from physiological signals using wireless sensors for presence technologies
Nasoz, F., Alvarez, K., Lisetti, C., & Finkelstein, N. (2004) · Cognition Technology & Work · doi:10.1007/s10111-003-0143-x · Cited by 369 · see citing works
Short-term emotion assessment in a recall paradigm
Chanel, G., Kierkels, J. J. M., Soleymani, M., & Pun, T. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.005 · Cited by 367 · see citing works
D’Mello, S. K., & Graesser, A. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9074-4 · Cited by 323 · see citing works
Automated Posture Analysis for Detecting Learner's Interest Level
Mota, S., & Picard, R. W. (2003) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2003.10047 · Cited by 315 · see citing works
Neurocomputing : algorithms, architectures and applications
Fogelman‐Soulié, F., & Hérault, J. (1990) · Springer eBooks · Cited by 275 · see citing works
Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech
Castellano, G., Kessous, L., & Caridakis, G. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85099-1_8 · Cited by 267 · see citing works
Wu, C., & Liang, W.-B. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.16 · Cited by 265 · see citing works
Frantzidιs, C. A., Bratsas, C., Papadelis, C. L., Konstantinidis, E. I., Pappas, C., & Bamidis, P. D. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2010.2041553 · Cited by 250 · see citing works
Petrantonakis, P. C., & Hadjileontiadis, L. J. (2011) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2011.2157933 · Cited by 226 · see citing works
A Framework for Automatic Human Emotion Classification Using Emotion Profiles
Mower, E., Mataric, M. J., & Narayanan, S. (2011) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2076804 · Cited by 215 · see citing works
Toward a Minimal Representation of Affective Gestures
Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M., & Scherer, K. R. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.7 · Cited by 214 · see citing works
Affect and Emotion in Human-Computer Interaction
Hutchison, D., Weikum, G., Vardi, M. Y., Tygar, D., Terzopoulos, D., Sudan, M., Steffen, B., Peter, C., Pandu Rangan, C., Nierstrasz, O., et al. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85099-1 · Cited by 186 · see citing works
Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data
Wagner, J., Andre, E., Lingenfelser, F., & Kim, J. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.12 · Cited by 156 · see citing works
Bimodal Emotion Recognition using Speech and Physiological Changes
Kim, J. (2007) · doi:10.5772/4754 · Cited by 145 · see citing works
AUTOMATIC DETECTION OF LEARNER'S AFFECT FROM GROSS BODY LANGUAGE
D’Mello, S. K., & Graesser, A. C. (2009) · Applied Artificial Intelligence · doi:10.1080/08839510802631745 · Cited by 143 · see citing works
Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals
AlZoubi, O., D’Mello, S. K., & Calvo, R. A. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.4 · Cited by 131 · see citing works
Experimental evaluation of expert fusion strategies
Alkoot, F. M., & Kittler, J. (1999) · Pattern Recognition Letters · doi:10.1016/s0167-8655(99)00107-5 · Cited by 118 · see citing works
ERNN: A Biologically Inspired Feedforward Neural Network to Discriminate Emotion From EEG Signal
Khosrowabadi, R., Quek, C. H., Ang, K. K., & Wahab, A. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2280271 · Cited by 106 · see citing works
Robust Speech Recognition and Understanding
Grimm, M., & Kroschel, K. (2007) · doi:10.5772/35 · Cited by 98 · see citing works
Test‐retest reliability of voluntary emotion regulation
Lee, H., Shackman, A. J., Jackson, D. C., & Davidson, R. J. (2009) · Psychophysiology · doi:10.1111/j.1469-8986.2009.00830.x · Cited by 54 · see citing works
Rezazadeh, I. M., Firoozabadi, M., Hu, H., & Golpayegani, S. M. R. H. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.3 · Cited by 50 · see citing works
Self-citationEmotion classification during music listening from forehead biosignals
Naji, M., Firoozabadi, M., & Azadfallah, P. (2013) · Signal, Image and Video Processing · doi:10.1007/s11760-013-0591-6 · Cited by 46 · see citing works
Self-citationNaji, M., Firoozabadi, M., & Azadfallah, P. (2014) · Cognitive Computation · doi:10.1007/s12559-013-9239-7 · Cited by 46 · see citing works
Self-citationAdaptive classifier integration for robust pattern recognition
Chibelushi, C. C., Deravi, F., & Mason, J. S. (1999) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/3477.809043 · Cited by 44 · see citing works
Toward multimodal fusion of affective cues
Paleari, M., & Lisetti, C. (2006) · doi:10.1145/1178745.1178762 · Cited by 43 · see citing works
Recognizing Emotions From an Ensemble of Features
Tariq, U., Lin, K.-H., Li, Z., Zhou, X., Wang, Z., Le, V., Huang, T. S., Lv, X., & Han, T. X. (2012) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2194701 · Cited by 36 · see citing works
EOG signal processing and analysis for controlling computer by eye movements
Estrany, B., Fuster, P., García, A., et al. (2009) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/1579114.1579132 · Cited by 20 · see citing works
An adaptive classifier fusion method for analysis of knee-joint vibroarthrographic signals
Wu, Y., & Krishnan, S. (2009) · IEEE International Conference on Computational Intelligence for Measurement Systems and Applications (CIMSA) · doi:10.1109/cimsa.2009.5069945 · Cited by 15 · see citing works
Kim, J., & André, E. (2008) · IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) · doi:10.1109/mfi.2008.4648119 · Cited by 12 · see citing works
Alignment Method for Spectrograms of DNA Sequences
Bucur, A., van Leeuwen, J., Dimitrova, N., et al. (2009) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2033052 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Qiu, S., Zhao, H., Jiang, N., Wang, Z., Liu, L., An, Y., Zhao, H., Miao, X., Liu, R., & Fortino, G. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.006 · Cited by 671 · see citing works
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.12.005 · Cited by 416 · see citing works
Survey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · see citing works
Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals
Wei, W., Jia, Q., Feng, Y., & Chen, G. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/5296523 · Cited by 82 · see citing works
Goshvarpour, A., Abbasi, A., & Goshvarpour, A. (2017) · Australasian Physical & Engineering Sciences in Medicine · doi:10.1007/s13246-017-0571-1 · Cited by 46 · see citing works
- Authors
- Khezri, M., Firoozabadi, M., & Sharafat, A. R.
- Affiliations
- Tarbiat Modares University (IR)
- Corresponding author
- Khezri, Mahdi
- Venue
- Computer Methods and Programs in Biomedicine
- Publisher
- Elsevier BV
- Published
- 2015-11
- Volume
- 122
- Issue
- 2
- Pages
- 149-164
- ISSN
- 0169-2607 (ISSN-L)
- OpenAlex
- W2130175312
- Semantic Scholar
- 761fffe0bd9128cfdc2ac4d21c0e84c2506994a9
- Cited by
- 89
- References
- 65
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 7 · 2023: 9 · 2022: 10 · 2021: 6 · 2020: 10 · 2019: 11 · 2018: 10 · 2017: 13 · 2016: 6 · 2015: 2
- FWCI
- 15.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Computer Methods and Programs in Biomedicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Keywords
- emotion recognition, multimodal fusion, adaptive weighted fusion, physiological signals, skin conductance, RR interval, basic emotions
- MeSH terms
- Adult, Algorithms, Blood Pressure, Emotions, Female, Forehead, Galvanic Skin Response, Heart Rate, Humans, Male, Pattern Recognition, Automated, Sensitivity and Specificity, Reproducibility of Results, Young Adult, Support Vector Machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
