The Psychophysiology Primer: A Guide to Methods and a Broad Review with a Focus on Human–Computer Interaction
Open accessThe DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Abstract
Digital monitoring of physiological signals can allow computer systems to adapt unobtrusively to users, so as to enhance personalised ‘smart’ interactions. In recent years, physiological computing has grown as a research field, and it is increasingly considered in diverse applications, ranging from specialised work contexts to consumer electronics. Working in this emerging field requires comprehension of several physiological signals, psychophysiological states or ‘indices’, and analysis techniques. The resulting literature encompasses a complex array of knowledge and techniques, presenting a clear challenge to the practitioner. We provide a foundational review of the field of psychophysiology to serve as a primer for the novice, enabling rapid familiarisation with the core concepts, or as a quick-reference resource for advanced readers. We place special emphasis on everyday human–computer interface applications, drawing a distinction from clinical or sports applications, which are more commonplace. The review provides a framework of commonly understood terms associated with experiential constructs and physiological signals. Then, 12 short and precisely focused review chapters describe 10 individual signals or signal sources and present two technical discussions of online data fusion and processing. A systematic review of multimodal studies is provided in the form of a reference table. We conclude with a general discussion of the application of psychophysiology to human–computer interaction, including guidelines and challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Delorme, A., & Makeig, S. (2004) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2003.10.009 · Cited by 24,371
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Event-related EEG/MEG synchronization and desynchronization: basic principles
Pfurtscheller, G., & Lopes da Silva, F. H. (1999) · Clinical Neurophysiology · doi:10.1016/s1388-2457(99)00141-8 · Cited by 7,328
Kirschbaum, C., Pirke, K.-M., & Hellhammer, D. H. (2008) · Neuropsychobiology · doi:10.1159/000119004 · Cited by 6,093
The brainweb: phase synchronization and large-scale integration
Varela, F., Lachaux, J.-P., Rodriguez, E., & Martinerie, J. (2001) · Nature Reviews Neuroscience · doi:10.1038/35067550 · Cited by 4,761
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
Heart rate variability: origins, methods, and interpretive caveats
BERNTSON, G. G., THOMAS BIGGER, J., ECKBERG, D. L., GROSSMAN, P., KAUFMANN, P. G., MALIK, M., NAGARAJA, H. N., PORGES, S. W., SAUL, J. P., STONE, P. H., et al. (1997) · Psychophysiology · doi:10.1111/j.1469-8986.1997.tb02140.x · Cited by 3,962
Looking at pictures: Affective, facial, visceral, and behavioral reactions
LANG, P. J., GREENWALD, M. K., BRADLEY, M. M., & HAMM, A. O. (1993) · Psychophysiology · doi:10.1111/j.1469-8986.1993.tb03352.x · Cited by 3,190
Autonomic nervous system activity in emotion: a review
Kreibig, S. D. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.03.010 · Cited by 3,188
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Rayner, K. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210902816461 · Cited by 2,682
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 1×
“These include openSMILE – the Munich Versatile and Fast Open-Source Audio Feature Extractor (Eyben et al., 2010) – and AMMON (Affective and Mental Health Monitor) (Chang et al.”
MethodPOSNER, J., RUSSELL, J. A., & PETERSON, B. S. (2005) · Development and Psychopathology · doi:10.1017/s0954579405050340 · Cited by 2,602 · cited 1×
“The competing theories have resulted in creation of several, very different models, such as the PAD circumplex of emotion (Posner et al., 2005) (see Figure 2.”
BackgroundToward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334
A Survey on Wearable Sensor-Based Systems for Health Monitoring and Prognosis
Pantelopoulos, A., & Bourbakis, N. G. (2010) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2009.2032660 · Cited by 2,296 · cited 1×
“, 2014), while in health care wearable sensors enable continuous physiological monitoring at a relatively low cost (Pantelopoulos and Bourbakis, 2010).”
BackgroundDetecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207
The functional role of cross-frequency coupling
Canolty, R. T., & Knight, R. T. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.09.001 · Cited by 2,101
Vocal communication of emotion: A review of research paradigms
Scherer, K. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00084-5 · Cited by 2,044
Task-evoked pupillary responses, processing load, and the structure of processing resources.
Beatty, J. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.91.2.276 · Cited by 1,959 · cited 1×
“5 to 5 seconds after initiation, with the time depending on the nature of the task (Beatty, 1982).”
BackgroundAffect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1×
“Usually these are high-level reviews considering how fusing and combining modalities is possible in detection of emotions, and they seldom go into the details of each signal or metric (Calvo and D’Mello, 2010).”
BackgroundBioelectromagnetismPrinciples and Applications of Bioelectric and Biomagnetic Fields
Malmivuo, J., & Plonsey, R. (1995) · doi:10.1093/acprof:oso/9780195058239.001.0001 · Cited by 1,834
On the utility of P3 amplitude as a measure of processing capacity
Kok, A. (2001) · Psychophysiology · doi:10.1017/s0048577201990559 · Cited by 1,766
Storage of 7 +/- 2 short-term memories in oscillatory subcycles
Lisman, J. E., & Idiart, M. A. P. (1995) · Science · doi:10.1126/science.7878473 · Cited by 1,471
The LF/HF ratio does not accurately measure cardiac sympatho-vagal balance
Billman, G. E. (2013) · Frontiers in Physiology · doi:10.3389/fphys.2013.00026 · Cited by 1,401
Committee report: Guidelines for human startle eyeblink electromyographic studies
Blumenthal, T. D., Cuthbert, B. N., Filion, D. L., Hackley, S., Lipp, O. V., & Van Boxtel, A. (2005) · Psychophysiology · doi:10.1111/j.1469-8986.2005.00271.x · Cited by 1,169
Pupil Size in Relation to Mental Activity during Simple Problem-Solving
Hess, E. H., & Polt, J. M. (1964) · Science · doi:10.1126/science.143.3611.1190 · Cited by 1,169
Emotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944
Automatic recognition and analysis of human faces and facial expressions: a survey
Samal, A., & Iyengar, P. A. (1992) · Pattern Recognition · doi:10.1016/0031-3203(92)90007-6 · Cited by 910
Yasuma, F., & Hayano, J.-I. (2004) · Chest · doi:10.1378/chest.125.2.683 · Cited by 907
EEG and MEG: Relevance to Neuroscience
Lopes da Silva, F. (2013) · Neuron · doi:10.1016/j.neuron.2013.10.017 · Cited by 900
Parati, G., Saul, J. P., Di Rienzo, M., & Mancia, G. (1995) · Hypertension · doi:10.1161/01.hyp.25.6.1276 · Cited by 849
Pupil size variation as an indication of affective processing
Partala, T., & Surakka, V. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00017-x · Cited by 836
Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., Bertrand, O., & Lécuyer, A. (2010) · Presence: Teleoperators and Virtual Environments · doi:10.1162/pres.19.1.35 · Cited by 830
The functional significance of delta oscillations in cognitive processing
Harmony, T. (2013) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2013.00083 · Cited by 788
EEG delta oscillations as a correlate of basic homeostatic and motivational processes
Knyazev, G. G. (2012) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2011.10.002 · Cited by 776
A supramodal accumulation-to-bound signal that determines perceptual decisions in humans
O'Connell, R. G., Dockree, P. M., & Kelly, S. P. (2012) · Nature Neuroscience · doi:10.1038/nn.3248 · Cited by 755
Discriminating Stress From Cognitive Load Using a Wearable EDA Device
Setz, C., Arnrich, B., Schumm, J., La Marca, R., Troster, G., & Ehlert, U. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036164 · Cited by 742
Fundamentals of physiological computing
Fairclough, S. H. (2009) · Interacting with Computers · doi:10.1016/j.intcom.2008.10.011 · Cited by 608
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
Mechanisms and functions of theta rhythms
Colgin, L. L. (2013) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062012-170330 · Cited by 559 · cited 1×
“• θ: Hippocampal communications and the functional inhibition of executive functions (Colgin, 2013)”
BackgroundCardiovascular, electrodermal, and respiratory response patterns to fear‐ and sadness‐inducing films
Kreibig, S. D., Wilhelm, F. H., Roth, W. T., & Gross, J. J. (2007) · Psychophysiology · doi:10.1111/j.1469-8986.2007.00550.x · Cited by 495
An Analysis of Mental Workload in Pilots During Flight Using Multiple Psychophysiological Measures
Wilson, G. F. (2002) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1201_2 · Cited by 488
Gil, E., Orini, M., Bailón, R., Vergara, J. M., Mainardi, L., & Laguna, P. (2010) · Physiological Measurement · doi:10.1088/0967-3334/31/9/015 · Cited by 461
Physiological parameter monitoring from optical recordings with a mobile phone
Scully, C. G., Lee, J., Meyer, J., Gorbach, A. M., Granquist-Fraser, D., Mendelson, Y., & Chon, K. H. (2012) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2011.2163157 · Cited by 452 · cited 1×
“It has been demonstrated that the optical sensor of a mobile phone can detect, for example, the following elements from touch: breathing rate, heart rate, blood oxygen saturation, and even atrial fibrillation or blood loss (Scully et al., 2012).”
BackgroundPhysiological workload reactions to increasing levels of task difficulty
VELTMAN, J. A., & GAILLARD, A. W. K. (1998) · Ergonomics · doi:10.1080/001401398186829 · Cited by 433
BCILAB: a platform for brain–computer interface development
Kothe, C. A., & Makeig, S. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/5/056014 · Cited by 412
Nonlinear feature based classification of speech under stress.
Zhou, G., Hansen, J. H. L., & Kaiser, J. F. (2001) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.905995 · Cited by 409 · cited 1×
“Instead of direct comparisons involving individual features or combinations of them, modern approaches tend to use machine learning methods to improve detection rates (Zhou et al., 2001).”
BackgroundCognitive control of saccadic eye movements.
Hutton, S. B. (2008) · Brain and Cognition · doi:10.1016/j.bandc.2008.08.021 · Cited by 409
Estimating workload using EEG spectral power and ERPs in the n-back task
Brouwer, A.-M., Hogervorst, M. A., van Erp, J. B. F., Heffelaar, T., Zimmerman, P. H., & Oostenveld, R. (2012) · Journal of Neural Engineering · doi:10.1088/1741-2560/9/4/045008 · Cited by 407
Dugué, L., Marque, P., & VanRullen, R. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.1161-11.2011 · Cited by 364 · cited 1×
“Further studies support this claim, implying that even though this robust brain wave is linked to inhibitory processes, it does not represent an ‘idling’ rhythm of the brain but, rather, expresses ‘windows of excitability’ (Dugué et al., 2011).”
ResultGunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360 · cited 1×
“Although some approaches pointing in this direction have been put forward (Conati and Maclaren, 2009; Peter and Herbon, 2006; Gunes and Schuller, 2013; Gunes and Pantic, 2010), consensus is still lacking, as the nature of emotions is not fully understood.”
BackgroundEmpirically Building and Evaluating a Probabilistic Model of User Affect
Conati, C., & Maclaren, H. (2009) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-009-9062-8 · Cited by 328 · cited 1×
“Although some approaches pointing in this direction have been put forward (Conati and Maclaren, 2009; Peter and Herbon, 2006; Gunes and Schuller, 2013; Gunes and Pantic, 2010), consensus is still lacking, as the nature of emotions is not fully understood.”
BackgroundAn empirical study of machine learning techniques for affect recognition in human–robot interaction
Rani, P., Liu, C., Sarkar, N., & Vanman, E. (2006) · Pattern Analysis and Applications · doi:10.1007/s10044-006-0025-y · Cited by 313
Local and long-range functional connectivity is reduced in concert in autism spectrum disorders
Khan, S., Gramfort, A., Shetty, N. R., Kitzbichler, M. G., Ganesan, S., Moran, J. M., Lee, S. M., Gabrieli, J. D. E., Tager-Flusberg, H. B., Joseph, R. M., et al. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1214533110 · Cited by 303
Hogervorst, M. A., Brouwer, A.-M., & van Erp, J. B. F. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00322 · Cited by 292
Monitoring of mental workload levels during an everyday life office-work scenario
Cinaz, B., Arnrich, B., La Marca, R., & Tröster, G. (2013) · Personal and Ubiquitous Computing · doi:10.1007/s00779-011-0466-1 · Cited by 235
Baldi, E., & Bucherelli, C. (2005) · Nonlinearity in Biology, Toxicology, Medicine · doi:10.2201/nonlin.003.01.002 · Cited by 220
The effects of emotional behaviour on components of the respiratory cycle.
Boiten, F. A. (1998) · Biological Psychology · doi:10.1016/s0301-0511(98)00025-8 · Cited by 205
Interacting with human physiology
Pavlidis, I., Dowdall, J., Sun, N., Puri, C., Fei, J., & Garbey, M. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.11.018 · Cited by 175
Nickel, P., & Nachreiner, F. (2003) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.45.4.575.27094 · Cited by 167
Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data
Wagner, J., Andre, E., Lingenfelser, F., & Kim, J. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.12 · Cited by 156
Self-Alert Training: Volitional modulation of autonomic arousal improves sustained attention
O’Connell, R. G., Bellgrove, M. A., Dockree, P. M., Lau, A., Fitzgerald, M., & Robertson, I. H. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.12.018 · Cited by 124
Social copresence in anonymous social interactions using a mobile video telephone
Kang, S.-H., Watt, J. H., & Ala, S. K. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357295 · Cited by 37 · cited 1×
“Once a face is detected, it may reveal various attributes of the user, such as emotional engagement as assessed from facial expressions (Kang et al., 2008).”
BackgroundPhysical task stress and speaker variability in voice quality.
Godin, K. W., & Hansen, J. H. L. (2015) · EURASIP Journal on Audio, Speech, and Music Processing · doi:10.1186/s13636-015-0072-7 · Cited by 32
The effect of respiratory frequency on heart rate variability
Nesterov, S. V., Nesterov, V. P., & Burdygin, A. I. (2005) · Doklady Biological Sciences · doi:10.1007/s10630-005-0033-x · Cited by 8 · cited 1×
“Although respiration (respiration rate) is an important topic in health science, in physiology studies in a games or learning context it is less well researched than are many other psychophysiological signals, suffering from neglect even in studies of the cardiovascular system, with which it interacts intimately (Nest…”
Background
Cited by
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400
Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze
Krejtz, K., Duchowski, A. T., Niedzielska, A., Biele, C., & Krejtz, I. (2018) · PLOS ONE · doi:10.1371/journal.pone.0203629 · Cited by 349
Duchowski, A. T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M., & Giannopoulos, I. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173856 · Cited by 228
The Low/High Index of Pupillary Activity
Duchowski, A. T., Krejtz, K., Gehrer, N. A., Bafna, T., & Bækgaard, P. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376394 · Cited by 94
Gashi, S., Di Lascio, E., & Santini, S. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3314400 · Cited by 74
Metadata
- Authors
- Cowley, B., Filetti, M., Lukander, K., Torniainen, J., Henelius, A., Ahonen, L., Barral, O., Kosunen, I., Valtonen, T., Huotilainen, M., Ravaja, N., & Jacucci, G.
- Venue
- Foundations and Trends® in Human–Computer Interaction
- Publisher
- Emerald
- Published
- 2016-11-03
- Volume
- 9
- Issue
- 3-4
- Pages
- 151-308
- ISSN
- 1551-3955 (ISSN-L)
- 1551-3963
- OpenAlex
- W2547549442
- Cited by
- 110
- References
- 65
- FWCI
- 21.05
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.