CoChat
Sign Up
Cited by 69Emotion and Mood RecognitionFace and Expression RecognitionHeart Rate Variability and Autonomic Control

Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

Open access
Saffaryazdi, Nastaran, Wasim, Syed Talal, Dileep, Kuldeep, Nia, Alireza Farrokhi, Nanayakkara, Suranga, Broadbent, Elizabeth, Billinghurst, Mark·2022·Frontiers in Psychology
DOI 10.3389/fpsyg.2022.864047PMC PMC9275379
Open this paper in CoChat — free accountOpen PDF

Abstract

Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyg.2022.864047

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Combining multimodal features (EEG and facial expressions) can enhance performance.

    “Recently, there has been a growing trend of emotion recognition models that integrate EEG data with recorded facial feature videos, as combining multimodal features can enhance performance.”

    From Combination of facial expressions and EEG for multimodal emotion recognition (2025) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 107

  1. Alarcao S. M. Fonseca M. J. (2017). Emotions recognition using EEG signals: a survey. IEEE Trans. Affect. Comput. 10, 374– 393. 10.1109/TAFFC.2017.2714671 · DOI · Paper page
  2. Allaert B. Bilasco I. M. Djeraba C. (2018). Advanced local motion patterns for macro and micro facial expression recognition. arXiv preprint arXiv:1805.01951. 10.48550/arXiv.1805.01951 · DOI
  3. Aranha R. V. Corrêa C. G. Nunes F. L. (2019). Adapting software with affective computing: a systematic review. IEEE Trans. Affect. Comput. 12, 883– 899. 10.1109/TAFFC.2019.2902379 · DOI
  4. Aydın S. (2020). Deep learning classification of neuro-emotional phase domain complexity levels induced by affective video film clips. IEEE J. Biomed. Health Inform. 24, 1695– 1702. 10.1109/JBHI.2019.2959843 31841425 · DOI
  5. Aydın S. Demirtaş S. Yetkin S. (2018). Cortical correlations in wavelet domain for estimation of emotional dysfunctions. Neural Comput. Appl. 30, 1085– 1094. 10.1007/s00521-016-2731-8 · DOI
  6. Bartlett M. S. Littlewort G. Fasel I. Movellan J. R. (2003). Real time face detection and facial expression recognition: Development and applications to human computer interaction, in 2003 Conference on Computer Vision and Pattern Recognition Workshop (Madison, WI: IEEE).
  7. Bazgir O. Mohammadi Z. Habibi S. A. H. (2018). Emotion recognition with machine learning using EEG signals, in 2018 25th National and 3rd International Iranian Conference on Biomedical Engineering (ICBME) (Qom: IEEE), 1– 5. 10.1109/ICBME.2018.8703559 · DOI
  8. Ben X. Ren Y. Zhang J. Wang S.-J. Kpalma K. Meng W.. (2021). Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 10.1109/TPAMI.2021.3067464 33739920 · DOI
  9. Benlamine M. S. Chaouachi M. Frasson C. Dufresne A. (2016). Physiology-based recognition of facial micro-expressions using EEG and identification of the relevant sensors by emotion, in International Conference on Physiological Computing Systems - PhyCS, Vol. 2 (Lisbon: SciTePress), 130– 137. 10.5220/0006002701300137 · DOI
  10. Bressan M. Vitria J. (2003). Nonparametric discriminant analysis and nearest neighbor classification. Pattern Recogn. Lett. 24, 2743– 2749. 10.1016/S0167-8655(03)00117-X · DOI
  11. Busso C. Bulut M. Lee C.-C. Kazemzadeh A. Mower E. Kim S.. (2008). Iemocap: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42: 335. 10.1007/s10579-008-9076-6 · DOI · Paper page
  12. Chaparro V. Gomez A. Salgado A. Quintero O. L. Lopez N. Villa L. F. (2018). Emotion recognition from EEG and facial expressions: a multimodal approach, in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Honolulu, HI: IEEE), 530– 533. 10.1109/EMBC.2018.8512407 30440451 · DOI · Paper page
  13. Chin H. Molefi L. W. Yi M. Y. (2020). Empathy is all you need: how a conversational agent should respond to verbal abuse, in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (New York, NY: Association for Computing Machinery), 1– 13. 10.1145/3313831.3376461 · DOI · Paper page
  14. Claesen M. De Moor B. (2015). Hyperparameter search in machine learning. arXiv preprint arXiv:1502.02127. 10.48550/arXiv.1502.02127 · DOI
  15. Criminisi A. Shotton J. Konukoglu E. (2011). Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Technical Report, Microsoft Research Cambridge, MSRTR-2011-114, 5, 12. 10.1561/9781601985415 · DOI
  16. Das P. Khasnobish A. Tibarewala D. (2016). Emotion recognition employing ECG and GSR signals as markers of ANS, in 2016 Conference on Advances in Signal Processing (CASP) (Pune: IEEE), 37– 42. 10.1109/CASP.2016.7746134 · DOI
  17. Davison A. K. Yap M. H. Lansley C. (2015). Micro-facial movement detection using individualised baselines and histogram-based descriptors, in 2015 IEEE International Conference on Systems, Man, and Cybernetics (Hong Kong: IEEE), 1864– 1869. 10.1109/SMC.2015.326 · DOI
  18. Deng L. Wang X. Jiang F. Doss R. (2021). Eeg-based emotion recognition via capsule network with channel-wise attention and LSTM models. CCF Trans. Pervas. Comput. Interact. 3, 425– 435. 10.1007/s42486-021-00078-y · DOI
  19. Doma V. Pirouz M. (2020). A comparative analysis of machine learning methods for emotion recognition using EEG and peripheral physiological signals. J. Big Data 7, 1– 21. 10.1186/s40537-020-00289-7 · DOI
  20. Eerola T. Vuoskoski J. K. (2011). A comparison of the discrete and dimensional models of emotion in music. Psychol. Mus. 39, 18– 49. 10.1177/0305735610362821 · DOI
  21. Ekman P. (2003). Darwin, deception, and facial expression. Ann. N.Y. Acad. Sci. 1000, 205– 221. 10.1196/annals.1280.010 14766633 · DOI
  22. Ekman P. Friesen W. V. (1971). Constants across cultures in the face and emotion. J. Pers. Soc. psychol. 17: 124. 10.1037/h0030377 5542557 · DOI · Paper page
  23. Ekman P. Rosenberg E. L. (1997). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford: Oxford University Press.
  24. Ganapathy N. Veeranki Y. R. Kumar H. Swaminathan R. (2021). Emotion recognition using electrodermal activity signals and multiscale deep convolutional neural network. J. Med. Syst. 45, 1– 10. 10.1007/s10916-020-01676-6 33660087 · DOI
  25. Goh K. M. Ng C. H. Lim L. L. Sheikh U. U. (2020). Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis. Comput. 36, 445– 468. 10.1007/s00371-018-1607-6 · DOI
  26. Goodfellow I. J. Erhan D. Carrier P. L. Courville A. Mirza M. Hamner B.. (2013). Challenges in representation learning: a report on three machine learning contests, in International Conference on Neural Information Processing (Berlin; Heidelberg: Springer), 117– 124. 10.1007/978-3-642-42051-116 · DOI
  27. Gross J. J. Levenson R. W. (1995). Emotion elicitation using films. Cogn. Emot. 9, 87– 108. 10.1080/02699939508408966 · DOI · Paper page
  28. Guermazi R. Ben Abdallah T. Hammami M. (2021). Facial micro-expression recognition based on accordion spatio-temporal representation and random forests. J. Vis. Commun. Image Represent. 79: 103183. 10.1016/j.jvcir.2021.103183 · DOI
  29. Hashmi M. F. Ashish B. K. K. Sharma V. Keskar A. G. Bokde N. D. Yoon J. H.. (2021). LARNet: Real-time detection of facial micro expression using lossless attention residual network. Sensors 21: 1098. 10.3390/s21041098 33562767 PMC7914525 · DOI
  30. Hassouneh A. Mutawa A. Murugappan M. (2020). Development of a real-time emotion recognition system using facial expressions and EEG based on machine learning and deep neural network methods. Inform. Med. Unlock. 20: 100372. 10.1016/j.imu.2020.100372 · DOI
  31. Hossain M. Z. Gedeon T. (2019). Observers' physiological measures in response to videos can be used to detect genuine smiles. Int. J. Hum. Comput. Stud. 122, 232– 241. 10.1016/j.ijhcs.2018.10.003 · DOI
  32. Hu J. Huang Y. Hu X. Xu Y. (2021). Enhancing the perceived emotional intelligence of conver-sational agents through acoustic cues, in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 1– 7. 10.1145/3411763.3451660 · DOI
  33. Huang H. Xie Q. Pan J. He Y. Wen Z. Yu R.. (2021). An eeg-based brain computer interface for emotion recognition and its application in patients with disorder of consciousness. IEEE Trans. Affect. Comput. 12, 832– 842. 10.1109/TAFFC.2019.2901456 · DOI
  34. Huang X. Kortelainen J. Zhao G. Li X. Moilanen A. Seppänen T.. (2016). Multi-modal emotion analysis from facial expressions and electroencephalogram. Comput. Vis. Image Understand. 147, 114– 124. 10.1016/j.cviu.2015.09.01532041316 · DOI
  35. Islam M. R. Moni M. A. Islam M. M. Rashed-Al-Mahfuz M. Islam M. S. Hasan M. K.. (2021). Emotion recognition from EEG signal focusing on deep learning and shallow learning techniques. IEEE Access 9, 94601– 94624. 10.1109/ACCESS.2021.3091487 · DOI
  36. Jiang X. Bian G.-B. Tian Z. (2019). Removal of artifacts from EEG signals: a review. Sensors 19: 987. 10.3390/s19050987 30813520 PMC6427454 · DOI
  37. Jiang Y. Li W. Hossain M. S. Chen M. Alelaiwi A. Al-Hammadi M. (2020). A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Inform. Fus. 53, 209– 221. 10.1016/j.inffus.2019.06.019 · DOI
  38. Keelawat P. Thammasan N. Kijsirikul B. Numao M. (2019). Subject-independent emotion recognition during music listening based on EEG using deep convolutional neural networks, in 2019 IEEE 15th International Colloquium on Signal Processing & Its Applications (CSPA) (Penang: IEEE), 21– 26. 10.1109/CSPA.2019.8696054 · DOI
  39. Khalfallah J. Slama J. B. H. (2015). Facial expression recognition for intelligent tutoring systems in remote laboratories platform. Proc. Comput. Sci. 73, 274– 281. 10.1016/j.procs.2015.12.030 · DOI
  40. Kingma D. P. Ba J. (2014). Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. 10.48550/arXiv.1412.6980 · DOI · Paper page
  41. Koelstra S. Muhl C. Soleymani M. Lee J.-S. Yazdani A. Ebrahimi T.. (2011). Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3, 18– 31. 10.1109/T-AFFC.2011.15 · DOI · Paper page
  42. Koelstra S. Patras I. (2013). Fusion of facial expressions and EEG for implicit affective tagging. Image Vis. Comput. 31, 164– 174. 10.1016/j.imavis.2012.10.002 · DOI
  43. Kołodziej M. Tarnowski P. Majkowski A. Rak R. (2019). Electrodermal activity measurements for detection of emotional arousal. Bull. Pol. Acad. Sci.. 67: 4. 10.24425/bpasts.2019.130190 · DOI
  44. Kwon Y.-H. Shin S.-B. Kim S.-D. (2018). Electroencephalography based fusion two-dimensional (2d)-convolution neural networks (CNN) model for emotion recognition system. Sensors 18: 1383. 10.3390/s18051383 29710869 PMC5982398 · DOI
  45. Lan Z. Sourina O. Wang L. Scherer R. Müller-Putz G. R. (2018). Domain adaptation techniques for EEG-based emotion recognition: a comparative study on two public datasets. IEEE Trans. Cogn. Dev. Syst. 11, 85– 94. 10.1109/TCDS.2018.2826840 · DOI
  46. Lee M. S. Lee Y. K. Pae D. S. Lim M. T. Kim D. W. Kang T. K. (2019). Fast emotion recognition based on single pulse PPG signal with convolutional neural network. Appl. Sci. 9: 3355. 10.3390/app9163355 · DOI
  47. Li C. Wang B. Zhang S. Liu Y. Song R. Cheng J. Chen X. (2022). Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism. Comput. Biol. Med. 143: 105303. 10.1016/j.compbiomed.2022.105303 35217341 · DOI
  48. Li C. Zhang Z. Song R. Cheng J. Liu Y. Chen X. (2021). EEG-based emotion recognition via neural architecture search. IEEE Trans. Affect. Comput. 10.1109/TAFFC.2021.3130387 · DOI
  49. Li S. Deng W. (2020). Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 10.1109/TAFFC.2020.2981446 · DOI
  50. Li X. Pfister T. Huang X. Zhao G. Pietikäinen M. (2013). A spontaneous micro-expression database: inducement, collection and baseline, in 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (Shanghai: IEEE), 1– 6. 10.1109/FG.2013.6553717 · DOI
  51. Li X. Song D. Zhang P. Zhang Y. Hou Y. Hu B. (2018). Exploring EEG features in cross-subject emotion recognition. Front. Neurosci. 12: 162. 10.3389/fnins.2018.00162 29615853 PMC5867345 · DOI
  52. Lichtenstein A. Oehme A. Kupschick S. Jürgensohn T. (2008). Comparing two emotion models for deriving affective states from physiological data, in Affect and Emotion in Human-Computer Interaction, eds Christian P. Russell B. (Berlin; Heidelberg: Springer), 35– 50. 10.1007/978-3-540-85099-1_4 · DOI
  53. Liong S.-T. See J. Wong K. Le Ngo A. C. Oh Y.-H. Phan R. (2015). Automatic apex frame spotting in micro-expression database, in 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR) (Kuala Lumpur: IEEE), 665– 669. 10.1109/ACPR.2015.7486586 · DOI
  54. Ma J. Tang H. Zheng W.-L. Lu B.-L. (2019). Emotion recognition using multimodal residual LSTM network, in Proceedings of the 27th ACM International Conference on Multimedia (Nice), 176– 183. 10.1145/3343031.3350871 · DOI · Paper page
  55. Maia C. L. B. Furtado E. S. (2019). An approach to analyze user's emotion in HCI experiments using psychophysiological measures. IEEE Access 7, 36471– 36480. 10.1109/ACCESS.2019.2904977 · DOI
  56. Marcos-Pablos S. Gonzàlez-Pablos E. Martín-Lorenzo C. Flores L. A. Gòmez-García-Bermejo J. Zalama E. (2016). Corrigendum: virtual avatar for emotion recognition in patients with schizophrenia: a pilot study. Front. Hum. Neurosci. 10: 554. 10.3389/fnhum.2016.00554 27807413 PMC5089980 · DOI
  57. Maria E. Matthias L. Sten H. (2019). Emotion recognition from physiological signal analysis: a review. Electron. Notes Theor. Comput. Sci. 343, 35– 55. 10.1016/j.entcs.2019.04.00934388471 · DOI
  58. Oh Y.-H. See J. Le Ngo A. C. Phan R. C.-W. Baskaran V. M. (2018). A survey of automatic facial micro-expression analysis: databases, methods, and challenges. Front. Psychol. 9: 1128. 10.3389/fpsyg.2018.01128 30042706 PMC6049018 · DOI
  59. Pan H. Xie L. Wang Z. Liu B. Yang M. Tao J. (2021). Review of micro-expression spotting and recognition in video sequences. Virt. Real. Intell. Hardw. 3, 1– 17. 10.1016/j.vrih.2020.10.003 · DOI
  60. Pandey P. Seeja K. (2019a). Emotional state recognition with EEG signals using subject independent approach, in Data Science and Big Data Analytics, eds Durgesh Kumar M. Xin-She Y. Aynur U. (Singapore: Springer), 117– 124. 10.1007/978-981-10-7641-1_10 · DOI
  61. Pandey P. Seeja K. (2019b). Subject independent emotion recognition from EEG using VMD and deep learning. J. King Saud Univ. Comput. Inform. Sci. 34, 1730– 1738. 10.1016/j.jksuci.2019.11.003 · DOI
  62. Pantic M. Valstar M. Rademaker R. Maat L. (2005). Web-based database for facial expression analysis, in 2005 IEEE International Conference on Multimedia and Expo (Amsterdam: IEEE), 5. 10.1109/ICME.2005.1521424 · DOI
  63. Perez-Rosero M. S. Rezaei B. Akcakaya M. Ostadabbas S. (2017). Decoding emotional experiences through physiological signal processing, in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 881– 885. 10.1109/ICASSP.2017.7952282 · DOI
  64. Pfister T. Li X. Zhao G. Pietikäinen M. (2011). Recognising spontaneous facial micro-expressions, in 2011 International Conference on Computer Vision (Barcelona: IEEE), 1449– 1456. 10.1109/ICCV.2011.6126401 · DOI · Paper page
  65. Piumsomboon T. Lee Y. Lee G. A. Dey A. Billinghurst M. (2017). Empathic mixed reality: sharing what you feel and interacting with what you see, in 2017 International Symposium on Ubiquitous Virtual Reality (ISUVR), 38– 41. 10.1109/ISUVR.2017.20 · DOI
  66. Qu F. Wang S.-J. Yan W.-J. Fu X. (2016). CAS (me) 2: a database of spontaneous macro-expressions and micro-expressions, in International Conference on Human-Computer Interaction (Toronto, ON: Springer), 48– 59. 10.1007/978-3-319-39513-5_5 · DOI
  67. Quigley K. S. Lindquist K. A. Barrett L. F. (2014). Inducing and measuring emotion and affect: Tips, tricks, and secrets, in Handbook of Research Methods in Social and Personality Psychology, eds Reis H. T. Judd C. M. (Cambridge University Press), 220– 252.
  68. Raheel A. Majid M. Anwar S. M. (2019). Facial expression recognition based on electroence-phalography, in 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) (Sukkur: IEEE), 1– 5. 10.1109/ICOMET.2019.8673408 · DOI
  69. Rayatdoost S. Soleymani M. (2018). Cross-corpus EEG-based emotion recognition, in 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP) (Aalborg: IEEE), 1– 6. 10.1109/MLSP.2018.8517037 · DOI
  70. Reddy S. P. T. Karri S. T. Dubey S. R. Mukherjee S. (2019). Spontaneous facial micro-expression recognition using 3d spatiotemporal convolutional neural networks, in 2019 International Joint Conference on Neural Networks (IJCNN), 1– 8. 10.1109/IJCNN.2019.8852419 · DOI
  71. Redmon J. Farhadi A. (2018). Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767. 10.48550/arXiv.1804.02767 · DOI · Paper page
  72. Roberts N. A. Tsai J. L. Coan J. A. (2007). Emotion elicitation using dyadic interaction tasks, in Handbook of Emotion Elicitation and Assessment, eds Coan J. A. Allen J. B. (New York, NY: Oxford University Press), 106– 123.
  73. Samadiani N. Huang G. Cai B. Luo W. Chi C.-H. Xiang Y.. (2019). A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors 19: 1863. 10.3390/s19081863 31003522 PMC6514576 · DOI
  74. Schaefer A. Nils F. Sanchez X. Philippot P. (2010). Assessing the effectiveness of a large database of emotion-eliciting films: a new tool for emotion researchers. Cogn. Emot. 24, 1153– 1172. 10.1080/02699930903274322 · DOI · Paper page
  75. Setyohadi D. B. Kusrohmaniah S. Gunawan S. B. Pranowo P. Prabuwono A. S. (2018). Galvanic skin response data classification for emotion detection. Int. J. Electr. Comput. Eng. 8: 4004. 10.11591/ijece.v8i5.pp4004-4014 · DOI
  76. Sharma R. Pachori R. B. Sircar P. (2020). Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomed. Signal Process. Control 58: 101867. 10.1016/j.bspc.2020.101867 · DOI
  77. Shu L. Xie J. Yang M. Li Z. Li Z. Liao D.. (2018). A review of emotion recognition using physiological signals. Sensors 18: 2074. 10.3390/s18072074 29958457 PMC6069143 · DOI · Paper page
  78. Siedlecka E. Denson T. F. (2019). Experimental methods for inducing basic emotions: a qualitative review. Emot. Rev. 11, 87– 97. 10.1177/1754073917749016 · DOI
  79. Simonyan K. Zisserman A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. 10.48550/arXiv.1409.1556 · DOI · Paper page
  80. Soleymani M. Asghari-Esfeden S. Fu Y. Pantic M. (2015). Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affect. Comput. 7, 17– 28. 10.1109/TAFFC.2015.2436926 · DOI
  81. Soleymani M. Lichtenauer J. Pun T. Pantic M. (2011). A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3, 42– 55. 10.1109/T-AFFC.2011.2533679348 · DOI · Paper page
  82. Staudemeyer R. C. Morris E. R. (2019). Understanding LSTM-a tutorial into long short-term memory recurrent neural networks. arXiv preprint arXiv:1909.09586. 10.48550/arXiv.1909.09586 · DOI
  83. Sun Y. Ayaz H. Akansu A. N. (2020). Multimodal affective state assessment using fNIRS+ EEG and spontaneous facial expression. Brain Sci. 10: 85. 10.3390/brainsci10020085 32041316 PMC7071625 · DOI
  84. Sun Y. Wong A. K. Kamel M. S. (2009). Classification of imbalanced data: a review. Int. J. Pattern Recogn. Artif. Intell. 23, 687– 719. 10.1142/S0218001409007326 · DOI
  85. Takalkar M. Xu M. Wu Q. Chaczko Z. (2018). A survey: facial micro-expression recognition. Multim. Tools Appl. 77, 19301– 19325. 10.1007/s11042-017-5317-2 · DOI
  86. Tarnowski P. Kołodziej M. Majkowski A. Rak R. J. (2018). Combined analysis of GSR and EEG signals for emotion recognition, in 2018 International Interdisciplinary PhD Workshop (IIPhDW) (Świnoujście: IEEE), 137– 141. 10.1109/IIPHDW.2018.8388342 · DOI
  87. Topic A. Russo M. (2021). Emotion recognition based on EEG feature maps through deep learning network. Eng. Sci. Technol. Int. J. 24, 1442– 1454. 10.1016/j.jestch.2021.03.012 · DOI
  88. Tran T.-K. Vo Q.-N. Hong X. Li X. Zhao G. (2020). Micro-expression spotting: a new benchmark. arXiv preprint arXiv:2007.12421. 10.1016/j.neucom.2021.02.022 · DOI
  89. Udovičić G. Derek J. Russo M. Sikora M. (2017). Wearable emotion recognition system based on GSR and PPG signals, in Proceedings of the 2nd International Workshop on Multimedia for Personal Health and Health Care (Mountain View, CA), 53– 59. 10.1145/3132635.3132641 · DOI
  90. Van Quang N. Chun J. Tokuyama T. (2019). Capsulenet for micro-expression recognition, in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), 1– 7. 10.1109/FG.2019.8756544 · DOI
  91. Verma G. K. Tiwary U. S. (2014). Multimodal fusion framework: a multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage 102, 162– 172. 10.1016/j.neuroimage.2013.11.007 24269801 · DOI · Paper page
  92. Wagh K. P. Vasanth K. (2019). Electroencephalograph (EEG) based emotion recognition system: a review, in Innovations in Electronics and Communication Engineering (Singapore: Springer), 37– 59. 10.1007/978-981-10-8204-7_5 · DOI
  93. Weber R. Li J. Soladie C. Seguier R. (2018). A survey on databases of facial macro-expression and micro-expression, in International Joint Conference on Computer Vision, Imaging and Computer Graphics (Funchal-Madeira: Springer), 298– 325.
  94. Wei W. Jia Q. Feng Y. Chen G. (2018). Emotion recognition based on weighted fusion strategy of multichannel physiological signals. Comput. Intell. Neurosci. 2018: 5296523. 10.1155/2018/5296523 30073024 PMC6057426 · DOI · Paper page
  95. Wioleta S. (2013). Using physiological signals for emotion recognition, in 2013 6th International Conference on Human System Interactions (HSI) (Sopot: IEEE), 556– 561. 10.1109/HSI.2013.6577880 · DOI
  96. Xia Z. Hong X. Gao X. Feng X. Zhao G. (2019). Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans. Multim. 22, 626– 640. 10.1109/TMM.2019.2931351 · DOI
  97. Xia Z. Peng W. Khor H.-Q. Feng X. Zhao G. (2020). Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans. Image Process. 29, 8590– 8605. 10.1109/TIP.2020.3018222 32845838 · DOI
  98. Yan W.-J. Li X. Wang S.-J. Zhao G. Liu Y.-J. Chen Y.-H.. (2014). CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9: e86041. 10.1371/journal.pone.0086041 24475068 PMC3903513 · DOI · Paper page
  99. Yan W.-J. Wu Q. Liang J. Chen Y.-H. Fu X. (2013). How fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverb. Behav. 37, 217– 230. 10.1007/s10919-013-0159-8 · DOI
  100. Yang S. Luo P. Loy C.-C. Tang X. (2016). Wider face: a face detection benchmark, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, NV: IEEE), 5525– 5533. 10.1109/CVPR.2016.596 · DOI
  101. Yang W. Rifqi M. Marsala C. Pinna A. (2018). Physiological-based emotion detection and recognition in a video game context, in 2018 International Joint Conference on Neural Networks (IJCNN) (Rio de Janeiro: IEEE), 1– 8. 10.1109/IJCNN.2018.8489125 · DOI
  102. Yazdani A. Lee J.-S. Vesin J.-M. Ebrahimi T. (2012). Affect recognition based on physiological changes during the watching of music videos. ACM Trans. Interact. Intell. Syst. 2: 7. 10.1145/2133366.2133373 · DOI
  103. Zeng Z. Pantic M. Roisman G. I. Huang T. S. (2008). A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39– 58. 10.1109/TPAMI.2008.52 19029545 · DOI · Paper page
  104. Zepf S. Hernandez J. Schmitt A. Minker W. Picard R. W. (2020). Driver emotion recognition for intelligent vehicles: a survey. ACM Comput. Surveys 53, 1– 30. 10.1145/3388790 · DOI
  105. Zhao G. Li X. (2019). Automatic micro-expression analysis: open challenges. Front. Psychol. 2019: 1833. 10.3389/fpsyg.2019.01833 31447752 PMC6692451 · DOI
  106. Zheng W.-L. Liu W. Lu Y. Lu B.-L. Cichocki A. (2018). Emotionmeter: a multimodal framework for recognizing human emotions. IEEE Trans. Cybern. 49, 1110– 1122. 10.1109/TCYB.2018.2797176 29994384 · DOI · Paper page
  107. Zhu Q. Lu G. Yan J. (2020). Valence-arousal model based emotion recognition using EEG, peri-pheral physiological signals and facial expression, in Proceedings of the 4th International Conference on Machine Learning and Soft Computing (Haiphong City: ACM), 81– 85. 10.1145/3380688.3380694 · DOI

Cited by

Sorted by citations

Authors
Saffaryazdi, N., Wasim, S. T., Dileep, K., Nia, A. F., Nanayakkara, S., Broadbent, E., & Billinghurst, M.
Affiliations
University of Auckland (NZ)
Corresponding author
Saffaryazdi, Nastaran
ORCID
Venue
Frontiers in Psychology
Publisher
Frontiers Media SA
Published
2022-06-28
Volume
13
Pages
864047-864047
ISSN
1664-1078 (ISSN-L)
DOI
10.3389/fpsyg.2022.864047
PMC
PMC9275379
OpenAlex
W4283655854
Cited by
69
References
12
Citations by year
2026: 15 · 2025: 30 · 2024: 13 · 2023: 8 · 2022: 3
FWCI
12.42
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Psychology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
Keywords
emotion recognition, facial micro-expressions, EEG, skin conductance, photoplethysmography, multimodal emotion recognition, DEAP dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review