Combining body pose, gaze, and gesture to determine intention to interact in vision-based interfaces,
Abstract
Vision-based interfaces, such as those made popular by the Microsoft Kinect, suffer from the Midas Touch problem: every user motion can be interpreted as an interaction. In response, we developed an algorithm that combines facial features, body pose and motion to approximate a user's intention to interact with the system. We show how this can be used to determine when to pay attention to a user's actions and when to ignore them. To demonstrate the value of our approach, we present results from a 30-person lab study conducted to compare four engagement algorithms in single and multi-user scenarios. We found that combining intention to interact with a 'raise an open hand in front of you' gesture yielded the best results. The latter approach offers a 12% improvement in accuracy and a 20% reduction in time to engage over a baseline 'wave to engage' gesture currently used on the Xbox 360.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The user's body pose can be used to estimate their intent to engage with an embodied agent.
“The user's body pose can be used to estimate their intent to engage with an embodied agent (Bohus and Horvitz, 2009; Sanghvi et al., 2011; Schwarz et al., 2014).”
From Modeling Feedback in Interaction With Conversational Agents—A Review (2022) · cites this paper as Schwarz et al., 2014
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466 · see citing works
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
Affective Body Expression Perception and Recognition: A Survey
Kleinsmith, A., & Bianchi-Berthouze, N. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.16 · Cited by 646 · see citing works
A survey of design issues in spatial input
Hinckley, K., Pausch, R. F., Goble, J. C., & Kassell, N. F. (1994) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/192426.192501 · Cited by 366 · see citing works
Determining driver visual attention with one camera
Smith, P., Shah, M. A., & Lobo, N. D. V. (2003) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2003.821342 · Cited by 317 · see citing works
Automated Posture Analysis for Detecting Learner's Interest Level
Mota, S., & Picard, R. W. (2003) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2003.10047 · Cited by 315 · see citing works
Recognizing engagement in human-robot interaction
Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2010.5453163 · Cited by 266 · see citing works
Automatic analysis of affective postures and body motion to detect engagement with a game companion,
Sanghvi, J., Castellano, G., Leite, I., Pereira, A., McOwan, P. W., & Paiva, A. (2011) · Proceedings of the 6th international conference on Human-robot interaction · doi:10.1145/1957656.1957781 · Cited by 250 · see citing works
Understanding the Role of Body Movement in Player Engagement
Bianchi‐Berthouze, N. (2012) · Human-Computer Interaction · doi:10.1080/07370024.2012.688468 · Cited by 226 · see citing works
A spatial model of engagement for a social robot
Michalowski, M. P., Šabanović, S., & Simmons, R. G. (2006) · IEEE International Workshop on Advanced Motion Control (AMC) · doi:10.1109/amc.2006.1631755 · Cited by 201 · see citing works
Walter, R. F., Bailly, G., & Müller, J. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470774 · Cited by 171 · see citing works
Estimating user's engagement from eye-gaze behaviors in human-agent conversations
Nakano, Y., & Ishii, R. (2010) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1719970.1719990 · Cited by 159 · see citing works
Probabilistic combination of multiple modalities to detect interest
Kapoor, A., Picard, R. W., & Ivanov, Y. A. (2004) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2004.1334690 · Cited by 100 · see citing works
Multiple mice for retention tasks in disadvantaged schools
Pawar, U. S., Pal, J., Gupta, R., & Toyama, K. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240864 · Cited by 95 · see citing works
Bohus, D., & Horvitz, E. (2009) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1647314.1647323 · Cited by 93 · see citing works
Learning to predict engagement with a spoken dialog system in open-world settings
Bohus, D., & Horvitz, E. (2009) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.3115/1708376.1708411 · Cited by 83 · see citing works
Hartmann, B., Morris, M. R., Benko, H., & Wilson, A. D. (2010) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1718918.1718989 · Cited by 72 · see citing works
A framework for robust and flexible handling of inputs with uncertainty
Schwarz, J., Hudson, S. E., Mankoff, J. C., & Wilson, A. D. (2010) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1866029.1866039 · Cited by 62 · see citing works
Self-citationDesign issues for vision-based computer interaction systems
Kjeldsen, R., & Hartman, J. (2001) · doi:10.1145/971478.971511 · Cited by 47 · see citing works
Recognizing engagement in human-robot interaction
Rich, C., Ponsleur, B., Holroyd, A., & Sidner, C. L. (2010) · doi:10.5555/1734454.1734580 · Cited by 31 · see citing works
Gesture-Based, Touch-Free Multi-User Gaming on Wall-Sized, High-Resolution Tiled Displays
Stødle, D., Hagen, T.-M. S., Bjørndalen, J. M., & Anshus, O. J. (2024) · Journal of Virtual Reality and Broadcasting · doi:10.20385/1860-2037/5.2008.10 · Cited by 28 · see citing works
Feature Extraction and Selection for Inferring User Engagement in an HCI Environment
Asteriadis, S., Karpouzis, K. C., & Kollias, S. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02574-7_3 · Cited by 26 · see citing works
What Can Body Movement Tell Us About Players' Engagement?
Bianchi‐Berthouze, N. (2012) · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Shlizerman, E., Dery, L., Schoen, H., & Kemelmacher-Shlizerman, I. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00790 · Cited by 154 · see citing works
Carter, M., Velloso, E., Downs, J., Sellen, A. J., O'Hara, K. P., & Vetere, F. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858284 · Cited by 53 · see citing works
On Measuring Engagement Level During Child-Robot Interaction in Education
Lytridis, C., Bazinas, C., Papakostas, G. A., & Kaburlasos, V. G. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-26945-6_1 · Cited by 50 · see citing works
Freeman, E., Brewster, S. A., & Lantz, V. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858308 · Cited by 47 · see citing works
Modeling Feedback in Interaction With Conversational Agents—A Review
Axelsson, A., Buschmeier, H., & Skantze, G. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.744574 · Cited by 33 · see citing works
“The user’s body pose can be used to estimate their intent to engage with an embodied agent (Bohus and Horvitz, 2009; Sanghvi et al., 2011; Schwarz et al., 2014).”
BackgroundFaceEngage: Robust Estimation of Gameplay Engagement from User-Contributed (YouTube) Videos
Chen, X., Niu, L., Veeraraghavan, A., & Sabharwal, A. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2945014 · Cited by 31 · see citing works
Beyan, C., Katsageorgiou, V., & Murino, V. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123404 · Cited by 30 · see citing works
Sun, M., Zhao, Z., & Ma, X. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025469 · Cited by 29 · see citing works
MUMBAI: multi-person, multimodal board game affect and interaction analysis dataset
Doyran, M., Schimmel, A., Baki, P., Ergin, K., Türkmen, B., Salah, A. A. A., Bakkes, S. C. J., Kaya, H., Poppe, R. W., & Salah, A. A. (2021) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00364-0 · Cited by 30 · see citing works
Self-supervised prediction of the intention to interact with a service robot
Abbate, G., Giusti, A., Schmuck, V., Çeliktutan, O., & Paolillo, A. (2023) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2023.104568 · Cited by 24 · see citing works
GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents
Zeng, X., Wang, X., Zhang, T., Yu, C., Zhao, S., & Chen, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3698145 · Cited by 37 · see citing works
Tran, N., DeVries, P. S., Seita, M., Kushalnagar, R., Glasser, A., & Vogler, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642094 · Cited by 20 · see citing works
Feature Fusion of Face and Body for Engagement Intensity Detection
Li, Y., & Hung, Y. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803488 · Cited by 19 · see citing works
Uncertainty-Aware Gaze Tracking for Assisted Living Environments
Her, P., Manderle, L., Dias, P. A., Medeiros, H. P., & Odone, F. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3253253 · Cited by 19 · see citing works
Recognizing Unintentional Touch on Interactive Tabletop
Xu, X., Yu, C., Wang, Y., & Shi, Y. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3381011 · Cited by 17 · see citing works
Engagement Detection in MeetingsPreprint
Frank, M., Tofighi, G., Gu, H., & Fruchter, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.08711 · Cited by 16 · see citing works
Your Mouse Reveals Your Next Activity: Towards Predicting User Intention from Mouse Interaction
Fu, E. Y., Kwok, T. C. K., Wu, E. Y., Leong, H. V., Ngai, G., & Chan, S. C. F. (2017) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2017.270 · Cited by 14 · see citing works
Lytridis, C., Kaburlasos, V. G., Bazinas, C., Papakostas, G. A., Sidiropoulos, G. K., Nikopoulou, V. A., Holeva, V., Papadopoulou, M. T., & Evangeliou, A. (2022) · Sensors · doi:10.3390/s22020621 · Cited by 13 · see citing works
Investigating Clutching Interactions for Touchless Medical Imaging Systems
Cronin, S., Freeman, E., & Doherty, G. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517512 · Cited by 12 · see citing works
Vision-based engagement detection in Virtual Reality
Tofighi, G., Gu, H., & Raahemifar, K. (2016) · Digital Media Industry & Academic Forum (DMIAF) · doi:10.1109/dmiaf.2016.7574933 · Cited by 11 · see citing works
Investigating mid-air gestures and handhelds in motion tracked environments
Mäkelä, V., Korhonen, H., Ojala, J., Järvi, A., Väänänen, K., Raisamo, R., & Turunen, M. (2016) · ACM International Symposium on Pervasive Displays (PerDis) · doi:10.1145/2914920.2915015 · Cited by 9 · see citing works
Every Little Movement Has a Meaning of Its Own
Kwok, T. C. K., Fu, E. Y., Wu, E. Y., Huang, M. X., Ngai, G., & Leong, H. V. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3173002 · Cited by 9 · see citing works
Speakers' head and gaze dynamics weakly correlate in group conversation
Vrzáková, H., Bednarik, R., Nakano, Y., & Nihei, F. (2016) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/2857491.2857522 · Cited by 9 · see citing works
Posture Based Recognition of the Visual Focus of Attention for Adaptive Mobile Information Systems
Westhoven, M., Plegge, C., Henrich, T., & Alexander, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-39955-3_39 · Cited by 5 · see citing works
Embodied Interaction in Play: Body-Based and Natural Interaction in Games
Maurer, B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46152-6_15 · Cited by 5 · see citing works
Using Eye Gaze to Train an Adaptive Myoelectric InterfacePreprint
Chou, A. H. Y., Madduri, M. M., Li, S. J., Isa, J., Christensen, A., Hutchison, F. (., Burden, S. A., & Orsborn, A. L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.08.588608 · Cited by 5 · see citing works
DeVries, P. S., Tran, N., Delk, K., Miga, M., Taulbee, R. C., Pidathala, P., Glasser, A., Kushalnagar, R., & Vogler, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651075 · Cited by 4 · see citing works
Han, V. Y., Gonzalez, J. T., Yang, C., Wang, Z., Hudson, S. E., & Ion, A. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747726 · Cited by 4 · see citing works
Self-citationPactolo Bar: An Approach to Mitigate the Midas Touch Problem in Non-Conventional Interaction
de Freitas, A. A., Santos, D., Lima, R. S. D. A. D., dos Santos, C. G. R., & Meiguins, B. S. (2023) · Sensors · doi:10.3390/s23042110 · Cited by 4 · see citing works
Human-robot interaction assessment using dynamic engagement profiles
Poltorak, N., & Drimus, A. M. (2017) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2017.8246941 · Cited by 3 · see citing works
XR Input Error Mediation for Hand-Based Input: Task and Context Influences a User’s Preference
Lin, T., Lafreniere, B., Xu, Y., Grossman, T., Wigdor, D., & Glueck, M. (2023) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar59233.2023.00117 · Cited by 3 · see citing works
Toward implicit gaze-based map interaction: predicting selection intentions from eye movements
Lu, B., Zhang, C., Zhou, Y., & Liao, H. (2026) · Geo-spatial Information Science · doi:10.1080/10095020.2026.2633646 · Cited by 2 · see citing works
Touchless Tactile Interaction with Unconventional Permeable Displays
Sand, A., Rakkolainen, I., Surakka, V., Raisamo, R., & Brewster, S. A. (2022) · Human-computer interaction series · doi:10.1007/978-3-031-04043-6_8 · Cited by 2 · see citing works
Piya, C., Vinayak, V., & Karthik Ramani, A. (2016) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4034266 · Cited by 2 · see citing works
A Dataset of Naturally Occurring, Whole-Body Background Activity to Reduce Gesture ConflictsPreprint
Freeman, D., Jota, R., Vogel, D., Wigdor, D., & Balakrishnan, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06109 · Cited by 2 · see citing works
Dancing with the sound in edge computing environments
Hao, W., Han, M., Li, S., & Li, F. (2021) · Wireless Networks · doi:10.1007/s11276-021-02810-z · Cited by 1 · see citing works
A Software Toolbox for Behavioral Analysis in Robot-Assisted Special Education
Lytridis, C., Kaburlasos, V. G., Bazinas, C., Papakostas, G. A., Papadopoulou, C. I., & Nikopoulou, V. A. (2021) · International Conference on Software, Telecommunications and Computer Networks (SoftCOM) · doi:10.23919/softcom52868.2021.9559093 · Cited by 1 · see citing works
Towards a Cognitive Architecture for Socially Adaptive Human-Robot Interaction
Tanevska, A. (2020) · CINECA IRIS Institutial Research Information System (University of Genoa) · doi:10.15167/tanevska-ana_phd2020-03-03 · Cited by 1 · see citing works
ProtoRes: Proto-Residual Architecture for Deep Modeling of Human Pose.Preprint
Oreshkin, B. N., Bocquelet, F., Harvey, F. G., Raitt, B., & Laflamme, D. (2021) · arXiv (Cornell University)
Tanevska, A., Rea, F., Sandini, G., & Sciutti, A. (2017)
Kilger, M. T., Philipp, E., & Maierhöfer, V. (2025) · Mensch und Computer (MuC) · doi:10.1145/3743049.3743055
Dunphy, N., Gezae, G. W., DeVries, P. S., Li, M., Pidathala, P., Glasser, A., Kushalnagar, R., & Vogler, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31285-3_4
- Authors
- Schwarz, J., Marais, C. C., Leyvand, T., Hudson, S. E., & Mankoff, J.
- Affiliations
- Carnegie Mellon University (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2014-04-26
- Pages
- 3443-3452
- OpenAlex
- W2138207021
- Cited by
- 45
- References
- 23
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4 · 2023: 4 · 2022: 4 · 2021: 6 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 5 · 2016: 9 · 2015: 2
- FWCI
- 4.11
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National Science Foundation
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- vision-based interfaces, body pose, gaze, gesture recognition, Midas touch problem, user engagement, multi-user interaction, hand gestures, Kinect sensor, facial features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
