Multimodal emotion recognition by combining physiological signals and facial expressions: A preliminary study
Abstract
Lately, multimodal approaches for automatic emotion recognition have gained significant scientific interest. In this paper, emotion recognition by combining physiological signals and facial expressions was studied. Heart rate variability parameters, respiration frequency, and facial expressions were used to classify person's emotions while watching pictures with emotional content. Three classes were used for both valence and arousal. The preliminary results show that, over the proposed channels, detecting arousal seem to be easier compared to valence. While the classification performance of 54.5% was attained with arousal, only 38.0% of the samples were classified correctly in terms of valence. In future, additional modalities as well as feature selection will be utilized to improve the results.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Combining heart rate variability, respiration frequency, and facial expressions yielded relatively low accuracy of 54.5%.
“Accuracy for combining heart rate variability, respiration frequency, and facial expressions was relatively low: 54.5% [59].”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [59]
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · see citing works
Heart Rate Variability: Standards of Measurement, Physiological Interpretation, and Clinical Use
Electrophysiology, T. F. O. T. E. S. O. C. T. N. A. (1996) · Circulation · doi:10.1161/01.cir.93.5.1043 · Cited by 17,159 · see citing works
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,592 · see citing works
(2005) · CTIT technical reports series · Cited by 5,307 · see citing works
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Physiological signals based human emotion Recognition: a review
Jerritta, S., Murugappan, M., Nagarajan, R., & Khairunizam, W. (2011) · IEEE International Colloquium on Signal Processing & Its Applications (CSPA) · doi:10.1109/cspa.2011.5759912 · Cited by 349 · see citing works
Emotion representation, analysis and synthesis in continuous space: A survey
Güneş, H., Schuller, B. W., Pantić, M., & Cowie, R. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771357 · Cited by 328 · see citing works
Estimation of parameters for a linear difference equation with application to EEG analysis
Zetterberg, L. H. (1969) · Mathematical Biosciences · doi:10.1016/0025-5564(69)90044-3 · Cited by 293 · see citing works
Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A. C., Nass, C., & John, O. P. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.10.011 · Cited by 276 · see citing works
2D Cascaded AdaBoost for Eye Localization
Niu, Z., Shan, S., Yan, S., Chen, X., & Gao, W. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.1194 · Cited by 125 · see citing works
Arapakis, I., Konstas, I., & Jose, J. M. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631336 · Cited by 78 · see citing works
Emotion recognition with consideration of facial expression and physiological signals
Chang, C., Tsai, J.-S., Wang, C., & Chung, P. (2009) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb.2009.4925739 · Cited by 51 · see citing works
Flagged first, then by citations
1 of the 13 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Survey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · see citing works
Multi-modal emotion analysis from facial expressions and electroencephalogram
Huang, X., Kortelainen, J., Zhao, G., Li, X., Moilanen, A., Seppänen, T., & Pietikäinen, M. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.09.015 · Cited by 117 · see citing works
Self-citationA Survey on Human Emotion Recognition Approaches, Databases and Applications
Vinola, C., & Vimaladevi, K. (2015) · ELCVIA Electronic Letters on Computer Vision and Image Analysis · doi:10.5565/rev/elcvia.795 · Cited by 72 · see citing works
DRER: Deep Learning–Based Driver’s Real Emotion Recognizer
Oh, G., Ryu, J., Jeong, E., Yang, J. H., Hwang, S., Lee, S., & Lim, S. (2021) · Sensors · doi:10.3390/s21062166 · Cited by 55 · see citing works
Siddiqui, M. F. H., Dhakal, P., Yang, X., & Javaid, A. Y. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6060047 · Cited by 46 · see citing works
High-Performance Modelling and Simulation for Big Data Applications
Kołodziej, J., & González–Vélez, H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-16272-6 · Cited by 37 · see citing works
Gaussian process dynamical models for multimodal affect recognition
García, H. F., Álvarez, M. A., & Orozco, A. A. (2016) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2016.7590834 · Cited by 34 · see citing works
Emotion recognition with facial expressions and physiological signals
Zhong, B., Qin, Z., Yang, S., Chen, J., Mudrick, N. V., Taub, M., Azevedo, R., & Lobaton, E. J. (2017) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2017.8285365 · Cited by 34 · see citing works
EEG-based recognition of video-induced emotions: Selecting subject-independent feature set
Kortelainen, J., & Seppänen, T. (2013) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2013.6610493 · Cited by 33 · see citing works
Self-citationFeature selection for multimodal emotion recognition in the arousal-valence space
Torres-Valencia, C. A., Orozco, A. A., & Álvarez, M. A. (2013) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2013.6610504 · Cited by 31 · see citing works
Ouzar, Y., Bousefsaf, F., Djeldjli, D., & Maaoui, C. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00275 · Cited by 31 · see citing works
Emotion recognition techniques using physiological signals and video games –Systematic review–
Callejas-Cuervo, M., Martínez-Tejada, L. A., & Alarcón-Aldana, A. C. (2017) · Revista Facultad de Ingeniería · doi:10.19053/01211129.v26.n46.2017.7310 · Cited by 29 · see citing works
Estimating Stress in Online Meetings by Remote Physiological Signal and Behavioral Features
Sun, Z., Vedernikov, A., Kykyri, V., Pohjola, M. J., Nokia, M. S., & Li, X. (2022) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3544793.3563406 · Cited by 23 · see citing works
Self-citationDetecting naturalistic expression of emotions using physiological signals while playing video games
AlZoubi, O., AlMakhadmeh, B., Yassein, M. B., & Mardini, W. E. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03367-7 · Cited by 18 · see citing works
Hierarchical fusion of visual and physiological signals for emotion recognition
Fang, Y., Rong, R., & Huang, J. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-021-00774-z · Cited by 17 · see citing works
Is She Truly Enjoying the Conversation?
Katada, S., Okada, S., Hirano, Y., & Komatani, K. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418844 · Cited by 15 · see citing works
Lei, J., Sala, J., & Jasra, S. K. (2017) · Cited by 14 · see citing works
Torres-Valencia, C. A., Álvarez, M. A., & Orozco, A. A. (2014) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2014.6943754 · Cited by 13 · see citing works
Personal Identification Based on Content-Independent EEG Signal Analysis
Li, Y., Zhao, Y., Tan, T., Liu, N., & Fang, Y. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-69923-3_58 · Cited by 12 · see citing works
Discerning Affect From Touch and Gaze During Interaction With a Robot Pet
Cang, X. L., Bucci, P., Rantala, J., & MacLean, K. E. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3094894 · Cited by 10 · see citing works
The Novel EfficientNet Architecture-Based System and Algorithm to Predict Complex Human Emotions
Khomidov, M., & Lee, J.-H. (2024) · Algorithms · doi:10.3390/a17070285 · Cited by 6 · see citing works
Continuous Monitoring of Emotions by a Multimodal Cooperative Sensor System
Mencattini, A., Ringeval, F., Schuller, B. W., Martinelli, E., & Natale, C. D. (2015) · Procedia Engineering · doi:10.1016/j.proeng.2015.08.716 · Cited by 5 · see citing works
Human Fear Analysis using Signal and Image Processing
Sarkar, S. B. (2019) · International Conference on Computing and Communications Technologies (ICCCT) · doi:10.1109/iccct2.2019.8824815 · Cited by 3 · see citing works
Architecture of an emotion recognition and video games system to identify personality traits
Callejas-Cuervo, M., Martínez-Tejada, L. A., & Botero-Fagua, J. A. (2017) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-981-10-4086-3_11 · Cited by 3 · see citing works
Zhang, T., Cao, Z., Li, W., & Lv, Z. (2026) · Pediatric Investigation · doi:10.1002/ped4.70038 · Cited by 1 · see citing works
Glavey, E. M. (2020) · Journal of International Crisis and Risk Communication Research · Cited by 1 · see citing works
Unconstrained Face Recognition Using Infrared Images
Butt, A. R., Rahman, Z. U., Haq, A. U., Ahmed, B., & Manzoor, S. (2024) · International Journal of Image and Graphics · doi:10.1142/s0219467825500561 · Cited by 1 · see citing works
Valencia, T., & Alejandro, C. (2013) · Americanae (AECID Library)
Advances in Emotional Picture Classification
Zhang, Y. (2014) · Advances in information quality and management · doi:10.4018/978-1-4666-5888-2.ch048
Bimodal Emotion Recognition using Kernel Canonical Correlation Analysis and Multiple Kernel Learning
Yan, J., & Qiu, W. (2021) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei53629.2021.9624428
- Authors
- Kortelainen, J., Tiinanen, S., Huang, X., Li, X., Laukka, S., Pietikainen, M., & Seppanen, T.
- Affiliations
- University of Oulu (FI)
- Venue
- 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society
- Publisher
- IEEE
- Published
- 2012-08
- Volume
- 2012
- Pages
- 5238-5241
- ISSN
- 2375-7477 (ISSN-L)
- OpenAlex
- W2154129660
- Cited by
- 30
- References
- 13
- Citations by year
- 2026: 1 · 2024: 2 · 2022: 3 · 2021: 5 · 2020: 2 · 2019: 3 · 2017: 5 · 2016: 2 · 2015: 2 · 2014: 2 · 2013: 3
- FWCI
- 4.33
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- multimodal emotion recognition, physiological signals, facial expressions, heart rate variability, respiratory rate, valence, arousal, emotion classification, emotional pictures
- MeSH terms
- Algorithms, Electrocardiography, Emotions, Facial Expression, Heart Rate, Humans, Image Interpretation, Computer-Assisted, Pattern Recognition, Automated, Pilot Projects, Psychometrics, Sensitivity and Specificity, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review