A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subject’s psychological state.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
Heart rate variability: Standards of measurement, physiological interpretation, and clinical use
Malik, M., Bigger, J. T., Camm, A. J., Kleiger, R. E., Malliani, A., Moss, A. J., & Schwartz, P. J. (1996) · European Heart Journal · doi:10.1093/oxfordjournals.eurheartj.a014868 · Cited by 15,289
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · cited 1×
“Two examples of commonly used and well-validated self-report methods in psychophysiology are the Self-Assessment Manikin (Bradley and Lang, 1994) and the NASA-TLX (Hart and Staveland, 1988).”
MethodPicard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
Autonomic nervous system activity in emotion: a review
Kreibig, S. D. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.03.010 · Cited by 3,188 · cited 4×
“, 2001; Leon et al., 2004) or a small number of subjects. They are especially preferable when focusing on a specific, small group of the population with a limited amount of available subjects (e.g. six autistic children in Liu et al. (2008), seven air traffic controllers in Wilson and Russell (2003a)).”
BackgroundMethodInfluentialA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
An empirical comparison of supervised learning algorithms
Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143865 · Cited by 2,780
A review of classification algorithms for EEG-based brain–computer interfaces
Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., & Arnaldi, B. (2007) · Journal of Neural Engineering · doi:10.1088/1741-2560/4/2/r01 · Cited by 2,665
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · cited 10×
“This approach (called the ‘baseline matrix’) was used by, for example, Picard et al. (2001) and van den Broek et al. (2010).”
BackgroundMethodResultInfluentialDetecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 3×
“Both psychophysiology and affective computing have explored many avenues of research, including speech, facial expressions, gestures, central nervous system responses and autonomic nervous system responses (Zeng et al., 2009; Calvo and D’Mello, 2010).”
BackgroundMethodMehrabian, A. (1996) · Current Psychology · doi:10.1007/bf02686918 · Cited by 1,493
Emotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944 · cited 4×
“…performing a task or exposed to affective stimuli) are then normalized by either subtracting the baseline value (e.g. Christie and Friedman, 2004; Kim et al., 2004; Stephens et al., 2010), dividing by the baseline value (e.g. Zhai and Barreto, 2006; Arroyo-Palacios and Romano, 2010), subtracting…”
BackgroundMethodInfluentialDiscriminating Stress From Cognitive Load Using a Wearable EDA Device
Setz, C., Arnrich, B., Schumm, J., La Marca, R., Troster, G., & Ehlert, U. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036164 · Cited by 742 · cited 16×
“…subject-independent studies include 20 or more subjects (e.g. 20 in Setz et al. (2009), 20 in Chanel et al. (2011), 24 in Kapoor et al. (2007), 33 in Setz et al. (2010), 34 in Nasoz et al. (2010), 41 in Bailenson et al. (2008), 59 in Arroyo-Palacios and Romano (2010), 75 in Tognetti et al. (2010)).”
BackgroundMethodResultInfluentialFundamentals of physiological computing
Fairclough, S. H. (2009) · Interacting with Computers · doi:10.1016/j.intcom.2008.10.011 · Cited by 608
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
Cardiovascular, electrodermal, and respiratory response patterns to fear‐ and sadness‐inducing films
Kreibig, S. D., Wilhelm, F. H., Roth, W. T., & Gross, J. J. (2007) · Psychophysiology · doi:10.1111/j.1469-8986.2007.00550.x · Cited by 495 · cited 12×
“78.4 (3-class), 61.8 (4-class) None Katsis et al. (2006) Basic emotions 5 4 D 86 EMG Katsis et al. (2008) Basic emotions 4 10 ?”
BackgroundInfluentialEmotion Recognition Using Bio-sensors: First Steps towards an Automatic System
Haag, A., Goronzy, S., Schaich, P., & Williams, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24842-2_4 · Cited by 451
Affect-aware tutors: recognising and responding to student affect
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009) · International Journal of Learning Technology · doi:10.1504/ijlt.2009.028804 · Cited by 371
Psychophysiology and adaptive automation
Byrne, E. A., & Parasuraman, R. (1996) · Biological Psychology · doi:10.1016/0301-0511(95)05161-9 · Cited by 360 · cited 1×
“Such adaptation is not a new idea by any means; for instance, a review by Byrne and Parasuraman (1996) discusses several early attempts to use ANS responses to control the level of automation in a task.”
BackgroundStatLog: Comparison of classification algorithms on large real-world problems
KING, R. D., FENG, C., & SUTHERLAND, A. (1995) · Applied Artificial Intelligence · doi:10.1080/08839519508945477 · Cited by 320
An empirical study of machine learning techniques for affect recognition in human–robot interaction
Rani, P., Liu, C., Sarkar, N., & Vanman, E. (2006) · Pattern Analysis and Applications · doi:10.1007/s10044-006-0025-y · Cited by 313 · cited 14×
“78.4 (3-class), 61.8 (4-class) None Katsis et al. (2006) Basic emotions 5 4 D 86 EMG Katsis et al. (2008) Basic emotions 4 10 ?”
BackgroundInfluentialBen‐Shakhar, G. (1985) · Psychophysiology · doi:10.1111/j.1469-8986.1985.tb01603.x · Cited by 249 · cited 1×
“Though improved correlations with self-report questionnaires and better discrimination between emotions as a result of normalization were established decades ago (e.g. Lykken et al., 1966; Ben-Shakhar, 1985), the exact effects of normalization are difficult to gauge.”
BackgroundAutonomic specificity of basic emotions: Evidence from pattern classification and cluster analysis
Stephens, C. L., Christie, I. C., & Friedman, B. H. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.03.014 · Cited by 209 · cited 2×
“…or exposed to affective stimuli) are then normalized by either subtracting the baseline value (e.g. Christie and Friedman, 2004; Kim et al., 2004; Stephens et al., 2010), dividing by the baseline value (e.g. Zhai and Barreto, 2006; Arroyo-Palacios and Romano, 2010), subtracting the baseline…”
Background
Cited by
Katsigiannis, S., & Ramzan, N. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2688239 · Cited by 1,135
EEG-Based BCI Emotion Recognition: A Survey
Torres, E. P., Torres, E. A., Hernández-Álvarez, M., & Yoo, S. G. (2020) · Sensors · doi:10.3390/s20185083 · Cited by 291
A survey of sensor fusion methods in wearable robotics
Novak, D., & Riener, R. (2015) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2014.08.012 · Cited by 252
Self-citationShoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Muñoz, J. E., Pope, A. T., & Velez, L. E. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-27950-9_12 · Cited by 24
Metadata
- Authors
- Novak, D., Mihelj, M., & Munih, M.
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-05
- Volume
- 24
- Issue
- 3
- Pages
- 154-172
- ISSN
- 0953-5438 (ISSN-L)
- OpenAlex
- W2023912726
- Cited by
- 163
- References
- 28
- FWCI
- 17.37
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.