CoChat
Sign Up
Cited by 77Emotion and Mood RecognitionEEG and Brain-Computer InterfacesGaze Tracking and Assistive Technology

A Survey on Psycho-Physiological Analysis & Measurement Methods in Multimodal Systems

Open access
Baig, Muhammad Zeeshan, Kavakli, Manolya·2019·Multimodal Technologies and Interaction
DOI 10.3390/mti3020037
Open this paper in CoChat — free accountOpen PDF

Abstract

Psycho-physiological analysis has gained greater attention in the last few decades in various fields including multimodal systems. Researchers use psychophysiological feedback devices such as skin conductance (SC), Electroencephalography (EEG) and Electrocardiography (ECG) to detect the affective states of the users during task performance. Psycho-physiological feedback has been successful in detection of the cognitive states of users in human-computer interaction (HCI). Recently, in game studies, psycho-physiological feedback has been used to capture the user experience and the effect of interaction on human psychology. This paper reviews several psycho-physiological, cognitive, and affective assessment studies and focuses on the use of psychophysiological signals in estimating the user’s cognitive and emotional states in multimodal systems. In this paper, we review the measurement techniques and methods that have been used to record psycho-physiological signals as well as the cognitive and emotional states in a variety of conditions. The aim of this review is to conduct a detailed study to identify, describe and analyze the key psycho-physiological parameters that relate to different mental and emotional states in order to provide an insight into key approaches. Furthermore, the advantages and limitations of these approaches are also highlighted in this paper. The findings state that the classification accuracy of >90% has been achieved in classifying emotions with EEG signals. A strong correlation between self-reported data, HCI experience, and psychophysiological data has been observed in a wide range of domains including games, human-robot interaction, mobile interaction, and simulations. An increase in β and γ -band activity have been observed in high intense games and simulations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/mti3020037

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Physiological signals as a direct measure of sympathetic or parasympathetic nervous system activation following psychological or physiological arousal could inform about the inner state of subjects facing diverse contexts.

    “As a direct measure of sympathetic or parasympathetic nervous system activation following psychological or physiological arousal (Baig & Kavakli, 2019;Taylor & Epstein, 1967), they could inform us about the inner state of the subjects facing diverse contexts.”

    From The Peripersonal Space in a social world (2021) · cites this paper as Baig & Kavakli, 2019

    Section 3.3 (page unavailable) says autonomic nervous system activity can be derived from heart rate and relates changes in heart rate to sympathetic and parasympathetic activity. Sections 2 and 4 say physiological signals can help estimate cognitive and emotional states across contexts. The paper does not establish that physiological signals generally provide a direct measure of sympathetic or parasympathetic activation; it describes deriving that activity from a particular signal. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 98

  1. Cacioppo, J.T.; Tassinary, L.G.; Berntson, G.G. Psychophysiological science: Interdisciplinary approaches to classic questions about the mind. In Handbook of Psychophysiology; Cambridge University Press: New York, NY, USA, 2007; pp. 1-16.
  2. Poels, K.; De Kort, Y.; Ijsselsteijn, W. It is always a lot of fun!: Exploring dimensions of digital game experience using focus group methodology. In Proceedings of the 2007 Conference on Future Play, Toronto, ON, Canada, 14-17 November 2007; ACM: New York, NY, USA, 2007; pp. 83-89.
  3. Jaimes, A.; Sebe, N. Multimodal human-computer interaction: A survey. Comput. Vis. Image Understand. 2007, 108, 116-134. [CrossRef] · DOI · Paper page
  4. Maxwell, S.E.; Delaney, H.D. Designing Experiments and Analyzing Data: A Model Comparison Perspective; Psychology Press: New York, NY, USA, 2004; Volume 1.
  5. Field, A. Discovering Statistics Using IBM SPSS Statistics; Sage Publications: London, UK, 2013.
  6. Gratch, J.; Marsella, S. A domain-independent framework for modeling emotion. Cogn. Syst. Res. 2004, 5, 269-306. [CrossRef] · DOI
  7. Cacioppo, J.T.; Tassinary, L.G.; Berntson, G. Handbook of Psychophysiology; Cambridge University Press: New York, NY, USA, 2007.
  8. Glynn, L.M.; Christenfeld, N.; Gerin, W. The role of rumination in recovery from reactivity: Cardiovascular consequences of emotional states. Psychosom. Med. 2002, 64, 714-726. [PubMed]
  9. Ekman, P. Facial expression and emotion. Am. Psychol. 1993, 48, 384. [CrossRef] [PubMed] · DOI · Paper page
  10. Holmqvist, K.; Nyström, M.; Andersson, R.; Dewhurst, R.; Jarodzka, H.; Van de Weijer, J. Eye Tracking: A Comprehensive Guide to Methods and Measures; OUP Oxford: Oxford, UK, 2011.
  11. McKee, M.G. Biofeedback: An overview in the context of heart-brain medicine. Clevel. Clin. J. Med. 2008, 75, S31. [CrossRef] · DOI
  12. Sterelny, K. Thought in a Hostile World: The Evolution of Human Cognition; Wiley-Blackwell: London, UK, 2003.
  13. Albulbul, A. Evaluating major electrode types for idle biological signal measurements for modern medical technology. Bioengineering 2016, 3, 20. [CrossRef] [PubMed] · DOI
  14. Zhang, X.; Li, J.; Liu, Y.; Zhang, Z.; Wang, Z.; Luo, D.; Zhou, X.; Zhu, M.; Salman, W.; Hu, G.; et al. Design of a fatigue detection system for high-speed trains based on driver vigilance using a wireless wearable EEG. Sensors 2017, 17, 486. [CrossRef] [PubMed] · DOI
  15. Jabaudon, D.; Sztajzel, J.; Sievert, K.; Landis, T.; Sztajzel, R. Usefulness of ambulatory 7-day ECG monitoring for the detection of atrial fibrillation and flutter after acute stroke and transient ischemic attack. Stroke 2004, 35, 1647-1651. [CrossRef] [PubMed] · DOI
  16. Park, C.; Chou, P.H.; Bai, Y.; Matthews, R.; Hibbs, A. An ultra-wearable, wireless, low power ECG monitoring system. In Proceedings of the 2006 IEEE Biomedical Circuits and Systems Conference, London, UK, 29 November-1 December 2006; pp. 241-244.
  17. Grozea, C.; Voinescu, C.D.; Fazli, S. Bristle-sensors-Low-cost flexible passive dry EEG electrodes for neurofeedback and BCI applications. J. Neural Eng. 2011, 8, 025008. [CrossRef] [PubMed] · DOI
  18. Amores, J.; Benavides, X.; Maes, P. Psychicvr: Increasing mindfulness by using virtual reality and brain computer interfaces. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 7-12 May 2016; p. 2.
  19. Alam, M.; Azad, A. Development of biomedical data acquisition system in Hard Real-Time Linux environment. In Proceedings of the 2012 International Conference on Biomedical Engineering (ICoBE), Penang, Malaysia, 27-28 February 2012; pp. 436-440.
  20. Wong, C.; Zhang, Z.Q.; Lo, B.; Yang, G.Z. Wearable sensing for solid biomechanics: A review. IEEE Sens. J. 2015, 15, 2747-2760.
  21. Ishimaru, S.; Kunze, K.; Uema, Y.; Kise, K.; Inami, M.; Tanaka, K. Smarter eyewear: Using commercial EOG glasses for activity recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, Seattle, WA, USA, 13-17 September 2014; pp. 239-242.
  22. Garbarino, M.; Lai, M.; Bender, D.; Picard, R.W.; Tognetti, S. Empatica E3-A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In Proceedings of the 2014 EAI 4th International Conference on Wireless Mobile Communication and Healthcare (Mobihealth), Athens, Greece, 3-5 November 2014; pp. 39-42.
  23. Burns, A.; Greene, B.R.; McGrath, M.J.; O'Shea, T.J.; Kuris, B.; Ayer, S.M.; Stroiescu, F.; Cionca, V. SHIMMER TM -A wireless sensor platform for noninvasive biomedical research. IEEE Sens. J. 2010, 10, 1527-1534. [CrossRef] · DOI
  24. Panigrahy, S.K.; Jena, S.K.; Turuk, A.K. Study and Analysis of Human Stress Detection using Galvanic Skin Response (GSR) Sensor inWired and Wireless Environments. Res. J. Pharmacy Technol. 2017, 10, 545-550. [CrossRef] · DOI
  25. McKenzie, J.; Osgood, D. Validation of a new telemetric core temperature monitor. J. Ther. Biol. 2004, 29, 605-611. [CrossRef] · DOI
  26. Bonarini, A.; Matteucci, M.; Tognetti, S. Chest expansion reconstruction from respiration sound by using artificial neural networks. In Proceedings of the 4th IET International Conference on Advances in Medical, Signal and Information Processing-MEDSIP 2008, Santa Margherita Ligure, Italy, 14-16 July 2008.
  27. Jones, V.; Van Halteren, A.; Dokovsky, N.; Koprinkov, G.; Peuscher, J.; Bults, R.; Konstantas, D.; Herzog, R. Mobihealth: Mobile services for health professionals. In M-Health; Springer: Boston, MA, USA, 2006; pp. 237-246.
  28. McSharry, P.E.; Clifford, G.D.; Tarassenko, L.; Smith, L.A. A dynamical model for generating synthetic electrocardiogram signals. IEEE Trans. Biomed. Eng. 2003, 50, 289-294. [CrossRef] [PubMed] · DOI
  29. Allen, J. Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 2007, 28, R1. [CrossRef] · DOI · Paper page
  30. Barreto, A.; Zhai, J.; Adjouadi, M. Non-intrusive physiological monitoring for automated stress detection in human-computer interaction. In International Workshop on Human-Computer Interaction; Springer: Berlin/Heidelberg, Germany, 2007; pp. 29-38.
  31. Healey, J. Physiological sensing of emotion. In The Oxford Handbook of Affective Computing; Oxford University Press: Oxford, UK, 2014; pp. 204-216.
  32. van Ravenswaaij-Arts, C.M.; Kollee, L.A.; Hopman, J.C.; Stoelinga, G.B.; van Geijn, H.P. Heart rate variability. Ann. Internal Med. 1993, 118, 436-447. [CrossRef] · DOI
  33. Rowe, D.W.; Sibert, J.; Irwin, D. Heart rate variability: Indicator of user state as an aid to human-computer interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Los Angeles, CA, USA, 18-23 April 1998; ACM Press/Addison-Wesley Publishing Co.: New York, NY, USA, 1998; pp. 480-487.
  34. Akselrod, S.; Gordon, D.; Ubel, F.A.; Shannon, D.C.; Berger, A.; Cohen, R.J. Power spectrum analysis of heart rate fluctuation: a quantitative probe of beat-to-beat cardiovascular control. Science 1981, 213, 220-222. [CrossRef] [PubMed] · DOI · Paper page
  35. Itoh, H.; Takeda, K.; Nakamura, K.; Fujita, H.; Uchida, A.; Kuwahara, T.; Takenaka, K.; Tanaka, M.; Nakata, T.; Sasaki, S.; et al. Young borderline hypertensives are hyperreactive to mental arithmetic stress: Spectral analysis of RR intervals. J. Auton. Nerv. Syst. 1995, 54, 155-162. [CrossRef] · DOI
  36. Cacioppo, J.T.; Berntson, G.G.; Larsen, J.T.; Poehlmann, K.M.; Ito, T.A. The psychophysiology of emotion. In Handbook of Emotions; Guilford Press: New York, NY, USA, 1993; pp. 119-142.
  37. Jung, C.G. Studies in Word-Association; Taylor & Francis: Abingdon, UK, 1969.
  38. Healey, J. Gsr sock: A new e-textile sensor prototype. In Proceedings of the 2011 15th Annual International Symposium on Wearable Computers, San Francisco, CA, USA, 12-15 June 2011; pp. 113-114.
  39. Picard, R.W.; Healey, J. Affective wearables. Pers. Technol. 1997, 1, 231-240. [CrossRef] · DOI
  40. Healey, J. Recording affect in the field: Towards methods and metrics for improving ground truth labels. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Memphis, TN, USA, 9-12 October 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 107-116.
  41. Coan, J.A.; Allen, J.J. Frontal EEG asymmetry as a moderator and mediator of emotion. Biol. Psychol. 2004, 67, 7-50. [CrossRef] [PubMed] · DOI · Paper page
  42. Baig, M.Z.; Javed, E.; Ayaz, Y.; Afzal, W.; Gillani, S.O.; Naveed, M.; Jamil, M. Classification of left/right hand movement from EEG signal by intelligent algorithms. In Proceedings of the 2014 IEEE Symposium on Computer Applications and Industrial Electronics (ISCAIE), Penang, Malaysia, 7-8 April 2014; pp. 163-168.
  43. Baig, M.Z.; Aslam, N.; Shum, H.P.; Zhang, L. Differential evolution algorithm as a tool for optimal feature subset selection in motor imagery EEG. Expert Syst. Appl. 2017, 90, 184-195. [CrossRef] · DOI
  44. Spapé, M.M.; Filetti, M.; Eugster, M.J.; Jacucci, G.; Ravaja, N. Human computer interaction meets psychophysiology: A critical perspective. In Proceedings of the International Workshop on Symbiotic Interaction, Berlin, Germany, 7-8 October 2015; Springer: Berlin/Heidelberg, Germany, 2015; pp. 145-158.
  45. Ganglbauer, E.; Schrammel, J.; Schwarz, S.; Tscheligi, M. Applying Psychophysiological Methods for Measuring User Experience: Possibilities, Challenges and Feasibility. In Workshop on User Experience Evaluation Methods in Product Development; CiteSeerX: University Park, PA, USA, 2009.
  46. Horlings, R.; Datcu, D.; Rothkrantz, L.J. Emotion recognition using brain activity. In Proceedings of the 9th International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing, Gabrovo, Bulgaria, 12-13 June 2008; p. 6.
  47. Ekman, P.E.; Davidson, R.J. The Nature of Emotion: Fundamental Questions; Oxford University Press: Oxford, UK, 1994.
  48. Russel, J.A.; Weiss, A.; Mendelsohn, G.A. Affect grid: A single-item scale of pleasure and arousal. J. Pers. Soc. Psychol. 1989, 57, 493-502. [CrossRef] · DOI
  49. Lang, P.J. The emotion probe: Studies of motivation and attention. Am. Psychol. 1995, 50, 372. [CrossRef] · DOI
  50. Lang, P.J.; Bradley, M.M.; Cuthbert, B.N. International Affective Picture System (IAPS): Technical Manual and Affective Ratings; NIMH Center for the Study of Emotion and Attention: Gainesville, FL, USA, 1997; pp. 39-58.
  51. Partala, T.; Surakka, V. The effects of affective interventions in human-computer interaction. Interact. Comput. 2004, 16, 295-309. [CrossRef] · DOI
  52. Norman, D. Emotion & design: Attractive things work better. Interactions 2002, 9, 36-42.
  53. Haag, A.; Goronzy, S.; Schaich, P.; Williams, J. Emotion recognition using bio-sensors: First steps towards an automatic system. In Proceedings of the Tutorial and Research Workshop on Affective Dialogue Systems, Kloster Irsee, Germany, 14-16 June 2004; Springer: Berlin/Heidelberg, Germany, 2004, pp. 36-48.
  54. Klein, J.; Moon, Y.; Picard, R.W. This computer responds to user frustration: Theory, design, and results. Interact. Comput. 2002, 14, 119-140. [CrossRef] · DOI
  55. Partala, T.; Surakka, V. Pupil size variation as an indication of affective processing. Int. J. Hum.-Comput. Stud. 2003, 59, 185-198. [CrossRef] · DOI · Paper page
  56. Frijda, N.H. The Emotions: Studies in Emotion and Social Interaction; Cambridge University Press: Cambridge, UK, 1986.
  57. Mandryk, R.L.; Atkins, M.S. A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. Int. J. Hum.-Comput. Stud. 2007, 65, 329-347. [CrossRef] · DOI
  58. Juma, L. The Role of Secondary Emotions in Action Selection and Its Effects on the Believability of a Character. Master's Thesis, Aalborg University Copenhagen, København, Denmark, 2008 .
  59. Costa, T.; Rognoni, E.; Galati, D. EEG phase synchronization during emotional response to positive and negative film stimuli. Neurosci. Lett. 2006, 406, 159-164. [CrossRef] [PubMed] · DOI
  60. Li, M.; Lu, B.L. Emotion classification based on gamma-band EEG. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2009 (EMBC 2009), Minneapolis, MN, USA, 3-6 September 2009; pp. 1223-1226.
  61. Wang, X.W.; Nie, D.; Lu, B.L. EEG-based emotion recognition using frequency domain features and support vector machines. In Proceedings of the International Conference on Neural Information Processing, Shanghai, China, 13-17 November 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 734-743.
  62. Thompson, T.; Steffert, T.; Ros, T.; Leach, J.; Gruzelier, J. EEG applications for sport and performance. Methods 2008, 45, 279-288. [CrossRef] [PubMed] · DOI
  63. Nakasone, A.; Prendinger, H.; Ishizuka, M. Emotion recognition from electromyography and skin conductance. In Proceedings of the 5th International Workshop on Biosignal Interpretation, Tokyo, Japan, 6-8 September 2005; pp. 219-222.
  64. Khair, N.M.; Yaacob, S.; Hariharan, M.; Basah, S.N. A study of human emotional: Review. In Proceedings of the 2012 International Conference on Biomedical Engineering (ICoBE), Penang, Malaysia, 27-28 February 2012; pp. 393-399. [CrossRef] · DOI
  65. Boone, R.T.; Cunningham, J.G. Children's expression of emotional meaning in music through expressive body movement. J. Nonverbal Behav. 2001, 25, 21-41. [CrossRef] · DOI
  66. Zhou, F.; Qu, X.; Jiao, J.R.; Helander, M.G. Emotion prediction from physiological signals: A comparison study between visual and auditory elicitors. Interact. Comput. 2014, 26, 285-302. [CrossRef] · DOI
  67. Mehta, D.; Siddiqui, M.; Javaid, A. Facial emotion recognition: A survey and real-world user experiences in mixed reality. Sensors 2018, 18, 416. [CrossRef] · DOI
  68. Soleymani, M.; Lichtenauer, J.; Pun, T.; Pantic, M. A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 2012, 3, 42-55. [CrossRef] · DOI · Paper page
  69. Abadi, M.K.; Subramanian, R.; Kia, S.M.; Avesani, P.; Patras, I.; Sebe, N. DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses. IEEE Trans. Affect. Comput. 2015, 6, 209-222. [CrossRef] · DOI
  70. Koelstra, S.; Muhl, C.; Soleymani, M.; Lee, J.S.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. Deap: A database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 2012, 3, 18-31. [CrossRef] · DOI · Paper page
  71. Zheng, W.L.; Lu, B.L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 2015, 7, 162-175. [CrossRef] · DOI
  72. Perrin, A.F.N.M.; Xu, H.; Kroupi, E.; Řeřábek, M.; Ebrahimi, T. Multimodal dataset for assessment of quality of experience in immersive multimedia. In Proceedings of the 23rd ACM international conference on Multimedia, Brisbane, Australia, 26-30 October 2015; pp. 1007-1010.
  73. Perrin, A.F.; Řeřábek, M.; Ebrahimi, T. Towards prediction of Sense of Presence in immersive audiovisual communications. Electron. Imaging 2016, 2016, 1-8. [CrossRef] · DOI
  74. Ruohonen, J.; Hyrynsalmi, S. Evaluating the use of internet search volumes for time series modeling of sales in the video game industry. Electron. Mark. 2017, 27, 351-370. [CrossRef] · DOI
  75. Bernhaupt, R.; Mueller, F.F. Game User Experience Evaluation. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 7-12 May 2016; pp. 940-943.
  76. BERNHAUPT, R. Game User Experience Evaluation; Springer: Berlin/Heidelberg, Germany, 2016.
  77. Kivikangas, J.M.; Chanel, G.; Cowley, B.; Ekman, I.; Salminen, M.; Järvelä, S.; Ravaja, N. A review of the use of psychophysiological methods in game research. J. Gaming Virtual Worlds 2011, 3, 181-199. [CrossRef] · DOI
  78. Ijsselsteijn, W.; van den Hoogen, W.; Klimmt, C.; de Kort, Y.; Lindley, C.; Mathiak, K.; Poels, K.; Ravaja, N.; Turpeinen, M.; Vorderer, P. Measuring the experience of digital game enjoyment. In Proceedings of the 6th International Conference on Methods and Techniques in Behavioral Research, Maastricht, The Netherlands, 26-29 August 2008; pp. 88-89.
  79. Drachen, A.; Nacke, L.E.; Yannakakis, G.; Pedersen, A.L. Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, Los Angeles, CA, USA, 28-29 July 2010; pp. 49-54.
  80. Ballard, M.E.; Wiest, J.R. Mortal Kombat (tm): The Effects of Violent Videogame Play on Males' Hostility and Cardiovascular Responding. J. Appl. Soc. Psychol. 1996, 26, 717-730. [CrossRef] · DOI
  81. Drachen, A.; Nacke, L.E.; Yannakakis, G.N.; Pedersen, A.L. Psychophysiological Correlations with Gameplay Experience Dimensions. arXiv 2010, arXiv:1004.0243. · arXiv
  82. Yannakakis, G.N.; Hallam, J.; Lund, H.H. Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapt. Interact. 2008, 18, 207-243. [CrossRef] · DOI
  83. Pedersen, C.; Togelius, J.; Yannakakis, G.N. Modeling player experience in Super Mario Bros. In Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Games, Milano, Italy, 7-10 September 2009; pp. 132-139.
  84. McMahan, T.; Parberry, I.; Parsons, T.D. Modality specific assessment of video game player's experience using the Emotiv. Entertain. Comput. 2015, 7, 1-6. [CrossRef] · DOI
  85. Nacke, L.E.; Grimshaw, M.N.; Lindley, C.A. More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game. Interact. Comput. 2010, 22, 336-343. [CrossRef] 100. Cowley, B.; Kosunen, I.; Lankoski, P.; Kivikangas, J.M.; Järvelä, S.; Ekman, I.; Kemppainen, J.; Ravaja, N. Experience assessment and design in the analysis of gameplay. Simul. Gaming 2014, 45, 41-69. [CrossRef] 101. Stein, A.; Yotam, Y.; Puzis, R.; Shani, G.; Taieb-Maimon, M. EEG-triggered dynamic difficulty adjustment for multiplayer games. Entertain. Comput. 2018, 25, 14-25. [CrossRef] 102. Drachen, A.; Canossa, A.; Yannakakis, G.N. Player modeling using self-organization in Tomb Raider: Underworld. In Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Games, Milano, Italy, 7-10 September 2009; pp. 1-8. 103. · DOI
  86. Togelius, J.; Nardi, R.D.; Lucas, S. Making Racing Fun through Player Modeling and Track Evolution. In Proceedings of the SAB'06 Workshop on Adaptive Approaches for Optimizing Player Satisfaction in Computer and Physical Games, Rome, Italy, 1 October 2006; pp. 61-70. 104. Missura, O.; Gärtner, T. Player Modeling for Intelligent Difficulty Adjustment. In Proceedings of the 12th International Conference on Discovery Science (DS '09), Porto, Portugal, 3-5 October 2009; pp. 197-211. 105. Čertický, M.; Čertický, M.; Sinčák, P.; Magyar, G.; Vaščák, J.; Cavallo, F. Psychophysiological Indicators for Modeling User Experience in Interactive Digital Entertainment. Sensors 2019, 19, 989. [CrossRef] 106. · DOI
  87. Lobel, A.; Gotsis, M.; Reynolds, E.; Annetta, M.; Engels, R.C.; Granic, I. Designing and Utilizing Biofeedback Games for Emotion Regulation: The Case of Nevermind. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 7-12 May 2016; pp. 1945-1951. 107. Bethel, C.L.; Burke, J.L.; Murphy, R.R.; Salomon, K. Psychophysiological experimental design for use in human-robot interaction studies. In Proceedings of the 2007 International Symposium on Collaborative Technologies and Systems, Orlando, FL, USA, 25-25 May 2007; pp. 99-105. 108. Itoh, K.; Miwa, H.; Nukariya, Y.; Zecca, M.; Takanobu, H.; Roccella, S.; Carrozza, M.C.; Dario, P.; Takanishi, A. Development of a bioinstrumentation system in the interaction between a human and a robot. In Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China, 9-15 October 2006; pp. 2620-2625. 109. Bekele, E.; Sarkar, N. Psychophysiological feedback for adaptive human-robot interaction (HRI). In Advances in Physiological Computing; Springer: Berlin/Heidelberg, Germany, 2014; pp. 141-167. 110.
  88. Rani, P.; Sarkar, N. Making robots emotion-sensitive-preliminary experiments and results. In Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication (ROMAN 2005), Nashville, TN, USA, 13-15 August 2005; pp. 1-6. 111. Croft, D. Estimating intent for human-robot interaction. In Proceedings of the IEEE International Conference on Advanced Robotics, Coimbra, Portugal, 30 June-3 July 2003; pp. 810-815. 112. Dehais, F.; Sisbot, E.A.; Alami, R.; Causse, M. Physiological and subjective evaluation of a human-robot object hand-over task. Appl. Ergon. 2011, 42, 785-791. [CrossRef] [PubMed] 113. Paletta, L.; Dini, A.; Murko, C.; Yahyanejad, S.; Schwarz, M.; Lodron, G.; Ladstätter, S.; Paar, G.; Velik, R. Towards real-time probabilistic evaluation of situation awareness from human gaze in human-robot interaction. In Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria, 6-9 March 2017; pp. 247-248. 114. Podevijn, G.; O'grady, R.; Mathews, N.; Gilles, A.; Fantini-Hauwel, C.; Dorigo, M. Investigating the effect of increasing robot group sizes on the human psychophysiological state in the context of human-swarm interaction. Swarm Intell. 2016, 10, 193-210. [CrossRef] 115. Swangnetr, M.; Kaber, D.B. Emotional state classification in patient-robot interaction using wavelet analysis and statistics-based feature selection. IEEE Trans. Hum.-Mach. Syst. 2013, 43, 63-75. [CrossRef] 116. Kulic, D.; Croft, E. Anxiety detection during human-robot interaction. In Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, AB, Canada, 2-6 August 2005; pp. 616-621. 117. Kulić, D.; Croft, E. Physiological and subjective responses to articulated robot motion. Robotica 2007, 25, 13-27. [CrossRef] 118. Cortellessa, G.; Fracasso, F.; Sorrentino, A.; Orlandini, A.; Bernardi, G.; Coraci, L.; De Benedictis, R.; Cesta, A. ROBIN, a telepresence robot to support older users monitoring and social inclusion: Development and evaluation. Telemed. e-Health 2018, 24, 145-154. [CrossRef] 119. Tiberio, L.; Cesta, A.; Cortellessa, G.; Padua, L.; Pellegrino, A.R. Assessing affective response of older users to a telepresence robot using a combination of psychophysiological measures. In Proceedings of the 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, Paris, France, 9-13 September 2012; pp. 833-838. 120. Goljar, N.; Javh, M.; Poje, J.; Ocepek, J.; Novak, D.; Ziherl, J.; Olenšek, A.; Mihelj, M.; Munih, M. Psychophysiological responses to robot training in different recovery phases after stroke. In Proceedings of the 2011 IEEE International Conference on Rehabilitation Robotic, Zurich, Switzerland, 29 June-1 July 2011; pp. 1-6. 121. · DOI
  89. Ting, C.H.; Mahfouf, M.; Nassef, A.; Linkens, D.A.; Panoutsos, G.; Nickel, P.; Roberts, A.C.; Hockey, G.R.J. Real-time adaptive automation system based on identification of operator functional state in simulated process control operations. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 2010, 40, 251-262. [CrossRef] 122. Munih, M.; Mihelj, M. Psychophysiological responses during robot-assisted rehabilitation. In Rehabilitation Robotics; Elsevier: Amsterdam, The Netherlands, 2018; pp. 305-317. 123. Zhang, L.; Wade, J.; Bian, D.; Fan, J.; Swanson, A.; Weitlauf, A.; Warren, Z.; Sarkar, N., Multimodal Fusion for Cognitive Load Measurement in an Adaptive Virtual Reality Driving Task for Autism Intervention. In Universal Access in Human-Computer Interaction. Access to Learning, Health and Well-Being: 9th International Conference, UAHCI 2015, Held as Part of HCI International 2015, Los Angeles, CA, USA, August 2-7, 2015, Proceedings, Part III; · DOI
  90. Baig, M.Z.; Kavakli, M. Analyzing Novice and Expert User's Cognitive Load in using a Multi-Modal Interface System. In Proceedings of the 2018 26th International Conference on Systems Engineering (ICSEng), Sydney, Australia, 18-20 December 2018; pp. 1-7. [CrossRef] 127. · DOI
  91. Puma, S.; Matton, N.; Paubel, P.V.; Raufaste, É.; El-Yagoubi, R. Using theta and alpha band power to assess cognitive workload in multitasking environments. Int. J. Psychophysiol. 2018, 123, 111-120. [CrossRef] 128. Wilson, G.M. Psychophysiological indicators of the impact of media quality on users. In Proceedings of the CHI'01 extended abstracts on Human Factors in Computing Systems, Seattle, WA, USA, 31 March-5 April 2001; pp. 95-96. 129. · DOI
  92. Ward, R.D.; Marsden, P.H. Physiological responses to different WEB page designs. Int. J. Hum.-Comput. Stud. 2003, 59, 199-212. [CrossRef] 130. Bruun, A. It's not Complicated: A Study of Non-Specialists Analyzing GSR Sensor Data to Detect UX Related Events. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction, Oslo, Norway, 29 September-3 October 2018; pp. 170-183. 131. Lin, T.; Omata, M.; Hu, W.; Imamiya, A. Do physiological data relate to traditional usability indexes? In Proceedings of the 17th Australia conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future, Canberra, Australia, 21-25 November 2005; pp. 1-10. 132. · DOI
  93. Meehan, M.; Insko, B.; Whitton, M.; Brooks, F.P., Jr. Physiological measures of presence in stressful virtual environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, San Antonio, TX, USA, 23-26 July 2002; Volume 21, pp. 645-652. 133. Gerě, I.; Jaušcvec, N. Multimedia: Differences in cognitive processes observed with EEG. Educ. Technol. Res. Dev. 1999, 47, 5-14. [CrossRef] 134. Al Madi, N.S.; Khan, J.I. Measuring learning performance and cognitive activity during multimodal comprehension. In Proceedings of the 2016 7th International Conference on Information and Communication Systems (ICICS), Irbid, Jordan, 5-7 April 2016; pp. 50-55. 135. · DOI
  94. Novak, D.; Mihelj, M.; Munih, M. Dual-task performance in multimodal human-computer interaction: A psychophysiological perspective. Multimed. Tools Appl. 2012, 56, 553-567. [CrossRef] 136. Grassmann, M.; Vlemincx, E.; von Leupoldt, A.; Mittelstädt, J.M.; Van den Bergh, O. Respiratory changes in response to cognitive load: A systematic review. Neural Plast. 2016, 2016, 8146809. [CrossRef] [PubMed] 137. Lee, S.; Matteson, A.; Hooshyar, D.; Kim, S.; Jung, J.; Nam, G.; Lim, H. Comparing Programming Language Comprehension between Novice and Expert Programmers Using EEG Analysis. In Proceedings of the 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE), Taichung, Taiwan, 31 October-2 November 2016; pp. 350-355. 138. Ferreira, E.; Ferreira, D.; Kim, S.; Siirtola, P.; Röning, J.; Forlizzi, J.F.; Dey, A.K. Assessing real-time cognitive load based on psycho-physiological measures for younger and older adults. In Proceedings of the 2014 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), Orlando, FL, USA, 9-12 December 2014; pp. 39-48. 139. Liu, Y.; Ritchie, J.M.; Lim, T.; Kosmadoudi, Z.; Sivanathan, A.; Sung, R.C. A fuzzy psycho-physiological approach to enable the understanding of an engineer's affect status during CAD activities. Comput.-Aided Des. 2014, 54, 19-38. [CrossRef] 140. Nguyen, T.A.; Zeng, Y. A physiological study of relationship between designer's mental effort and mental stress during conceptual design. Comput.-Aided Desi. 2014, 54, 3-18. [CrossRef] 141. Nguyen, T.A.; Zeng, Y. Effects of stress and effort on self-rated reports in experimental study of design activities. J. Intell. Manuf. 2017, 28, 1609-1622. [CrossRef] 142. · DOI
  95. Baig, M.Z.; Kavakli, M. Connectivity Analysis Using Functional Brain Networks to Evaluate Cognitive Activity during 3D Modelling. Brain Sci. 2019, 9. [CrossRef] 143. · DOI
  96. Baig, M.Z.; Kavakli, M. EEG Signal Analysis in 3D Modelling to Identify Correlations Between Task Completion in Design User's Cognitive Activities. In Neural Information Processing;
  97. Cheng, L., Leung, A.C.S., Ozawa, S., Eds.; Springer International Publishing: Cham, Switzerland, 2018; pp. 340-352. 144.
  98. Fairclough, S.H. Fundamentals of physiological computing. Interact. Comput. 2009, 21, 133-145. [CrossRef] · DOI · Paper page

Cited by

Sorted by citations

  • The Peripersonal Space in a social world

    Bogdanova, O. V., Bogdanov, V. B., Dureux, A., Farnè, A., & Hadj-Bouziane, F. (2021) · Cortex · doi:10.1016/j.cortex.2021.05.005 · Cited by 86 · see citing works

    “As a direct measure of sympathetic or parasympathetic nervous system activation following psychological or physiological arousal (Baig & Kavakli, 2019; Taylor & Epstein, 1967), they could inform us about the inner state of the subjects facing diverse contexts.”
    Background

Authors
Baig, M. Z., & Kavakli, M.
Affiliations
Macquarie University (AU)
Corresponding author
Baig, Muhammad Zeeshan
ORCID
Venue
Multimodal Technologies and Interaction
Publisher
MDPI AG
Published
2019-05-28
Volume
3
Issue
2
Pages
37
ISSN
2414-4088 (ISSN-L)
DOI
10.3390/mti3020037
OpenAlex
W2947155785
Cited by
77
References
6
Citations by year
2026: 8 · 2025: 16 · 2024: 7 · 2023: 11 · 2022: 14 · 2021: 11 · 2020: 10
FWCI
10.30
Type
journal article
Language
en
Open access
Licence
Locations
  • Multimodal Technologies and Interaction (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Macquarie University
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
Keywords
multimodal systems, affective computing, emotion recognition, cognitive state estimation, skin conductance, EEG, electrocardiogram, EEG-based emotion recognition, human-computer interaction, game studies, human-robot interaction, mobile interaction, self-reported data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review