Connecting Brains and Bodies: Applying Physiological Computing to Support Social Interaction
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Physiological and affective computing propose methods to improve human–machine interactions by adapting machines to the users’ states. Recently, social signal processing (SSP) has proposed to apply similar methods to human–human interactions with the hope of better understanding and modeling social interactions. Most of the social signals employed are facial expressions, body movements and speech, but studies using physiological signals remain scarce. In this paper, we motivate the use of physiological signals in the context of social interactions. Specifically, we review studies which have investigated the relationship between various physiological indices and social interactions. We then propose two main directions to apply physiological SSP: using physiological signals of individual users as new social cues displayed in the group and using inter-user physiology to measure properties of the interactions such as conflict and social presence. We conclude that physiological measures have the potential to enhance social interactions and to connect people.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
Moral Emotions and Moral Behavior
Tangney, J. P., Stuewig, J., & Mashek, D. J. (2007) · Annual Review of Psychology · doi:10.1146/annurev.psych.56.091103.070145 · Cited by 3,222 · cited 1×
“This research was supported by the National Center of Competence in Research (NCCR) Affective sciences financed by the Swiss National Science Foundation (n° 51NF40-104897) and hosted by the University of Geneva.”
MethodLooking at pictures: Affective, facial, visceral, and behavioral reactions
LANG, P. J., GREENWALD, M. K., BRADLEY, M. M., & HAMM, A. O. (1993) · Psychophysiology · doi:10.1111/j.1469-8986.1993.tb03352.x · Cited by 3,190
Autonomic nervous system activity in emotion: a review
Kreibig, S. D. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.03.010 · Cited by 3,188 · cited 2×
“Different signals assessing the peripheral and central nervous system activity have been associated with emotional responses (Kreibig, 2010) and found relevant for the detection of emotions (Christian Mühl, Allison, et al., 2014; Novak, Mihelj, & Munih, 2012).”
BackgroundA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Autonomic nervous system activity distinguishes among emotions
Ekman, P., Levenson, R. W., & Friesen, W. V. (1983) · Science · doi:10.1126/science.6612338 · Cited by 2,692
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 2×
“The video capture of facial expressions together with voice capture are certainly the most studied modalities for the purpose of emotion assessment (Calvo & D’Mello, 2010; Cowie et al., 2001; Gunes et al., 2008; Pantic & Rothkrantz, 2003; Zeng et al., 2009).”
BackgroundMethodA Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
Cognitive neuroscience of human social behaviour
Adolphs, R. (2003) · Nature Reviews Neuroscience · doi:10.1038/nrn1056 · Cited by 1,657 · cited 1×
“Research in social psychology and neuroscience, has shown that physiological activity is related to several social processes such as empathy (Levenson & Ruef, 1992) and many other social behaviors (Adolphs, 2003; Levenson & Gottman, 1983).”
BackgroundToward a second-person neuroscience
Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., & Vogeley, K. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000660 · Cited by 1,510
How Emotions Regulate Social Life: The Emotions as Social Information (EASI) Model
Van Kleef, G. A. (2009) · Current Directions in Psychological Science · doi:10.1111/j.1467-8721.2009.01633.x · Cited by 1,457
Frontal EEG asymmetry as a moderator and mediator of emotion
Coan, J. A., & Allen, J. J. B. (2004) · Biological Psychology · doi:10.1016/j.biopsycho.2004.03.002 · Cited by 1,426 · cited 2×
“For example, the so-called frontal alpha asymmetry (Coan & Allen, 2004; Schmidt & Trainor, 2001), measuring a lateralization of the brain activity towards either the left or the right frontal cortices, has been found indicative of the valence dimension, pointing to a positive versus negative…”
BackgroundMethodBarrett, L. F., Mesquita, B., Ochsner, K. N., & Gross, J. J. (2007) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085709 · Cited by 1,425
Frontal theta activity in humans increases with memory load in a working memory task
Jensen, O., & Tesche, C. D. (2002) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.2002.01975.x · Cited by 1,379
Pfurtscheller, G., Stancák, A., & Neuper, C. (1996) · International Journal of Psychophysiology · doi:10.1016/s0167-8760(96)00066-9 · Cited by 1,205
Brain-to-brain coupling: a mechanism for creating and sharing a social world
Hasson, U., Ghazanfar, A. A., Galantucci, B., Garrod, S., & Keysers, C. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.12.007 · Cited by 1,140
Inter-Brain Synchronization during Social Interaction
Dumas, G., Nadel, J., Soussignan, R., Martinerie, J., & Garnero, L. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012166 · Cited by 966 · cited 2×
“…a shared action representation relative to the goal of the action (Ménoret et al., 2014), some studies have found asymmetric components that were related to the role assignment, leader versus follower, during the interaction (Dumas et al., 2010; Konvalinka et al., 2014; Ménoret et al., 2014).”
BackgroundMemory processes, brain oscillations and EEG synchronization
Klimesch, W. (1996) · International Journal of Psychophysiology · doi:10.1016/s0167-8760(96)00057-8 · Cited by 923 · cited 2×
“…distinguished via neurophysiological signals: prefrontal activity in the form of theta oscillations and alpha oscillations (Jensen & Tesche, 2002; Klimesch, 1996; Laursen, Jensen, Garde, & Jørgensen, 2002; Mühl, Jeunet, & Lotte, 2014), as well as pre-frontal blood flow (Herff et al., 2013;…”
BackgroundMarital interaction: Physiological linkage and affective exchange.
Levenson, R. W., & Gottman, J. M. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.3.587 · Cited by 855 · cited 2×
“(Levenson & Gottman, 1983) used such a method to measure the physiological coupling of spouses during marital conflicting situations.”
BackgroundMethodSocial signal processing: Survey of an emerging domain
Vinciarelli, A., Pantic, M., & Bourlard, H. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.11.007 · Cited by 842
Boucsein, W. (1992) · doi:10.1007/978-1-4757-5093-5 · Cited by 818
Cui, X., Bryant, D. M., & Reiss, A. L. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.09.003 · Cited by 807
Toward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771
Empathy: A physiological substrate.
Levenson, R. W., & Ruef, A. M. (1992) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.63.2.234 · Cited by 709
The self and social cognition: the role of cortical midline structures and mirror neurons
Uddin, L. Q., Iacoboni, M., Lange, C., & Keenan, J. P. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.01.001 · Cited by 704 · cited 1×
“1 / 1 Connecting brains and bodies: applying physiological computing to support social interaction”
BackgroundNeuroimaging Support for Discrete Neural Correlates of Basic Emotions: A Voxel-based Meta-analysis
Vytal, K., & Hamann, S. (2010) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21366 · Cited by 680 · cited 2×
“Despite the long-standing debate about their specific involvement (with constructivists (Lindquist & Barrett, 2012) versus localist (Vytal & Hamann, 2010) positions as extreme poles), there is a long line of research showing the accessibility of affective states from neurophysiological measurements.”
BackgroundMethodFundamentals of physiological computing
Fairclough, S. H. (2009) · Interacting with Computers · doi:10.1016/j.intcom.2008.10.011 · Cited by 608
Social neuroscience and hyperscanning techniques: past, present and future.
Babiloni, F., & Astolfi, L. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2012.07.006 · Cited by 573 · cited 1×
“Hyperscanning techniques aim at the creation of “spatio-temporal maps of cerebral regions involved in the generation of the social task investigated” in a study (see Babiloni & Astolfi, 2012).”
MethodNeural Synchronization during Face-to-Face Communication
Jiang, J., Dai, B., Peng, D., Zhu, C., Liu, L., & Lu, C. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.2926-12.2012 · Cited by 489 · cited 2×
“, 2012; Lindenberger, Li, Gruber, & Muller, 2009; Müller, Sänger, & Lindenberger, 2013), as well as for more complex interactive scenarios, for example face-to-face communication (Jiang et al., 2012) or aircraft take-off and landing (L Astolfi et al.”
BackgroundInterpersonal synchrony: a survey of evaluation methods across disciplines.
Delaherche, E., Chetouani, M., Mahdhaoui, A., Saint-Georges, C., Viaux, S., & Cohen, D. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.12 · Cited by 474
Physiological parameter monitoring from optical recordings with a mobile phone
Scully, C. G., Lee, J., Meyer, J., Gorbach, A. M., Granquist-Fraser, D., Mendelson, Y., & Chon, K. H. (2012) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2011.2163157 · Cited by 452 · cited 1×
“Physiological indexes can also be measured by means of cameras either by recording faces (Poh, McDuff, & Picard, 2010) or by placing your finger on a cell-phone camera (Scully et al., 2012).”
MethodBrains swinging in concert: cortical phase synchronization while playing guitar.
Lindenberger, U., Li, S.-C., Gruber, W., & Müller, V. (2009) · BMC Neuroscience · doi:10.1186/1471-2202-10-22 · Cited by 444 · cited 1×
“Two research directions are proposed to use physiological signals for social signal processing: o the display of new physiological and social cues; o the use of multi-person physiological activity as a measure of social interactions.”
BackgroundEmotional Contagion and Empathy
Hatfield, E., Rapson, R. L., & Le, Y.-C. L. (2009) · The Social Neuroscience of Empathy · doi:10.7551/mitpress/9780262012973.003.0003 · Cited by 419
Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing
Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D'Errico, F., & Schroeder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.27 · Cited by 381
Konvalinka, I., Bauer, M., Stahlhut, C., Hansen, L. K., Roepstorff, A., & Frith, C. D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.03.003 · Cited by 228 · cited 2×
“…a shared action representation relative to the goal of the action (Ménoret et al., 2014), some studies have found asymmetric components that were related to the role assignment, leader versus follower, during the interaction (Dumas et al., 2010; Konvalinka et al., 2014; Ménoret et al., 2014).”
BackgroundBrains “in concert”: frontal oscillatory alpha rhythms and empathy in professional musicians
Babiloni, C., Buffo, P., Vecchio, F., Marzano, N., Del Percio, C., Spada, D., Rossi, S., Bruni, I., Rossini, P. M., & Perani, D. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.12.008 · Cited by 136
Physiological compliance for social gaming analysis: Cooperative versus competitive play
Chanel, G., Kivikangas, J. M., & Ravaja, N. (2012) · Interacting with Computers · doi:10.1016/j.intcom.2012.04.012 · Cited by 110
Self-citationImaging the Social Brain by Simultaneous Hyperscanning during Subject Interaction
Astolfi, L., Toppi, J., De Vico Fallani, F., Vecchiato, G., Cincotti, F., Wilke, C., Yuan, H., Mattia, D., Salinari, S., He, B., et al. (2011) · IEEE Intelligent Systems · doi:10.1109/mis.2011.61 · Cited by 110
Social psychophysiological compliance in a four-person research team
Henning, R. A., Armstead, A. G., & Ferris, J. K. (2009) · Applied Ergonomics · doi:10.1016/j.apergo.2009.04.009 · Cited by 69
Cited by
What multimodal data can tell us about the students’ regulation of their learning process?
Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2021) · Learning and Instruction · doi:10.1016/j.learninstruc.2019.04.004 · Cited by 163
Multimodal data to design visual learning analytics for understanding regulation of learning
Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.019 · Cited by 140
“Whilst physiological measures such as EDA can be used to trace skin reactivity changes of learners in challenging moments (i.e., emotional arousal; Chanel & Mühl, 2015; Pecchinenda & Smith, 1996), video data can be used to reveal sequential and temporal dynamics of learners’ regulated learning…”
BackgroundMethodInfluentialWhat does physiological synchrony reveal about metacognitive experiences and group performance?
Dindar, M., Järvelä, S., & Haataja, E. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.12981 · Cited by 70
Metadata
- Authors
- Chanel, G., & Mühl, C.
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-04-26
- Volume
- 27
- Issue
- 5
- Pages
- 534-550
- ISSN
- 0953-5438 (ISSN-L)
- 1873-7951
- OpenAlex
- W2114909357
- Cited by
- 60
- References
- 42
- FWCI
- 9.90
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.