Robot deictics: how gesture and context shape referential communication
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
As robots collaborate with humans in increasingly diverse environments, they will need to effectively refer to objects of joint interest and adapt their references to various physical, environmental, and task conditions. Humans use a broad range of deictic gestures-gestures that direct attention to collocated objects, persons, or spaces-that include pointing, touching, and exhibiting to help their listeners understand their references. These gestures offer varying levels of support under different conditions, making some gestures more or less suitable for different settings. While these gestures offer a rich space for designing communicative behaviors for robots, a better understanding of how different deictic gestures affect communication under different conditions is critical for achieving effective human-robot interaction. In this paper, we seek to build such an understanding by implementing six deictic gestures on a humanlike robot and evaluating their communicative effectiveness in six diverse settings that represent physical, environmental, and task conditions under which robots are expected to employ deictic communication. Our results show that gestures which come into physical contact with the object offer the highest overall communicative accuracy and that specific settings benefit from the use of particular types of gestures. Our results highlight the rich design space for deictic gestures and inform how robots might adapt their gestures to the specific physical, environmental, and task conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“All of these systems are based on rules and/or procedural generation, which limits the degree of naturalness, fluidity, and variability that can be achieved in the resulting animations, and makes it difficult to match the gesture quality produced by contemporary generative models for co-speech gestures as described above.”
From Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents (2024) · cites this paper as [25]
Citations
Cites
1 retracted
RETRACTED: Communicative gestures facilitate problem solving for both communicators and recipients
Lozano, S. C., & Tversky, B. (2006) · Journal of Memory and Language · doi:10.1016/j.jml.2005.09.002 · Cited by 44
RetractedHand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
Gesture Visible Action as Utterance
Kendon, A. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511807572 · Cited by 3,891
Pointing: Where Language, Culture, and Cognition Meet
(2003) · Psychology Press eBooks · doi:10.4324/9781410607744 · Cited by 695
The role of gesture in communication and thinking
Goldin-Meadow, S. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01397-2 · Cited by 692
Vogel, D., & Balakrishnan, R. (2004) · Proceedings of the 17th annual ACM symposium on User interface software and technology · doi:10.1145/1029632.1029656 · Cited by 660
Towards a Descriptive Framework for Spatial Deixis
Fillmore, C. J. (1982) · Medical Entomology and Zoology · Cited by 655
Effects of nonverbal communication on efficiency and robustness in human-robot teamwork
Breazeal, C., Kidd, C. D., Thomaz, A. L., Hoffman, G., & Berlin, M. (2005) · 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2005.1545011 · Cited by 557
A comparative analysis of animals' understanding of the human pointing gesture
Miklósi, Á., & Soproni, K. (2006) · Animal Cognition · doi:10.1007/s10071-005-0008-1 · Cited by 421
Enabling Effective Human–Robot Interaction Using Perspective-Taking in Robots
Trafton, J. G., Cassimatis, N. L., Bugajska, M. D., Brock, D. P., Mintz, F. E., & Schultz, A. C. (2005) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/tsmca.2005.850592 · Cited by 309
Research works printed but not linked: 4
- 10.1037/0022-0663.89.1.183 · doi:10.1037/0022-0663.89.1.183 · Paper page
- S. Harrison . The creation and implementation of a gesture code for factory communication . In Proc. GESPIN 2011 , 2011 . S. Harrison. The creation and implementation of a gesture code for factory communication. In Proc. GESPIN 2011, 2011.
- 10.5555/2447556.2447665 · doi:10.5555/2447556.2447665
- 10.2307/1128700 · doi:10.2307/1128700 · Paper page
Cited by
Items 101–130 of 130
Human Multi-Robot Interaction based on Gesture Recognition
Ángulo, C. (2015)
Towards learning through robotic interaction alone: the joint guided search taskPreprint
DePalma, N., & Breazeal, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02632
Bidirectional gaze guiding and indexing in human-robot interaction through a situated architecture
DePalma, N. B. (2017) · DSpace@MIT (Massachusetts Institute of Technology)
Using Automatic Speech Recognition for Manipulation And Control of Robotic Platforms
al., M. P. P. E. A. M. P. P. E., & TJPRC (2020) · International journal of mechanical and production engineering research and development · doi:10.24247/ijmperdoct20203
DePalma, N., & Hodgins, J. K. (2021) · IEEE RO-MAN · doi:10.1109/ro-man50785.2021.9515416
Yamada, R., Akizuki, S., & Hashimoto, M. (2024) · International Workshop on Advanced Imaging Technology (IWAIT) · doi:10.1117/12.3018713
Examination of Spatial Deictic Markers in KingoniExamination of Spatial Deictic Markers in Kingoni
Haule, E. J. (2024) · MUST Journal of Research and Development. · doi:10.62277/mjrd2024v5i20044
Borgognon, N. S., Moccozet, L., & Molinari, G. (2025) · Conference on l'Interaction Homme-Machine (IHM) · doi:10.1145/3765712.3765730
Zhu, Y., Torres, A., Aloia, Z., & Williams, T. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246567
The social role of AI advisers
Longin, L. (2023) · doi:10.5282/edoc.32546
Technologies and paradigms for natural human-robot collaboration
Shukla, D. (2018) · Digital Library of the University of Innsbruck (University of Innsbruck)
Designing Non-Verbal Expressions for Appearance-Constrained Robots
SONG, S., ソウ, シ., & SONG, S. (2019) · Institutional Repositories DataBase (IRDB)
Word Embeddings para la anotación automática de gestos aplicada a Nao
Almagro Cádiz, M. (2017) · Espacio Tiempo y Forma Serie I Prehistoria y Arqueología
Evaluating Dynamic Surface Compensation for Robots with Projected AR
Wang, H., Dinh, N. B., & Zhao, H. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3776734.3794434
CLIO: Coordinate User Visual Attention with Audio Narratives for Enhanced Engagement
Chen, Y., Lo, I. L. T., Guo, B., Kawmali, N., Chan, C. K., Wang, R., Pan, J., & Yang, L. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3776734.3794565
Teaching AI through Object Demonstrations and Language Instructions
周中一 (2024) · Institutional Repositories DataBase (IRDB) · doi:10.15083/0002012805
Toward adaptation in human-robot collaboration
Vianello, L. (2022) · theses.fr (ABES) · doi:10.70675/76c35fa3z6a62z4fb4zaefazb17b495ea98a
Exploration of Pointing Forms in Customer–Shopkeeper Interactions
Jiang, Y., Doering, M., & Kanda, T. (2026) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3811026
Abstract Movement Design for Non-Humanoid Robots: Opening, Closing, and Turn-Taking Gestures
Lee, H. C., & Jung, E.-C. (2026) · Archives of Design Research · doi:10.15187/adr.2026.05.39.2.145
Goodge, T. A., Pollick, F. E., & Brewster, S. A. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3820654
Perceived Social Intelligence and Human Compliance in Incidental Human–Robot Encounters
Chan, Y., Hauser, E., Modak, S., Biswas, J., & Hart, J. (2026) · Expert Systems · doi:10.1111/exsy.70332
Lewis, C., Collins, S., Hanson, A., Smith, J., & Williams, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.03289
Merwin, E. R., Hammack, J. S., Riera‐Gomez, L., Keebler, J. R., Engeberg, E. D., & Wilcox, T. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100356
Συνεργασία Ανθρώπου-Ρομπότ: Ενίσχυση της προβλεπτικής συμπεριφοράς του ανθρώπου
Psarakis, L. (2026) · DSpace - NTUA (National Technical University of Athens) · doi:10.26240/heal.ntua.33233
Hein, I., Ullrich, D., Brunner, C., & Diefenbach, S. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1869586
Kannan, N., Kidd, J., Konrad, S., Grant, J. B., Kunold, L., & Roesler, E. (2026) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813261484485
MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation PlanningPreprint
Lu, M., Xiao, A., & Hsu, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.24995
Vidyadharan, A., Tennison, J., Goewert, S., Shimizu, M., & Gorlewicz, J. L. (2026) · International Journal of Social Robotics · doi:10.1007/s12369-026-01451-6
Networked Humanoid Intelligence: Communication, Coordination, Collaboration, and Social InteractionPreprint
Nguyen, L. X., Le, H. Q., Yoon, J. S., Tran, N. H. K., Eui-Nam, H., & Hong, C. S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7669301
Metadata
- Authors
- Sauppé, A., & Mutlu, B.
- Affiliations
- University of Wisconsin–Madison (US)
- Venue
- Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction
- Publisher
- ACM
- Published
- 2014-03-03
- Pages
- 342-349
- OpenAlex
- W2036765125
- Cited by
- 93
- References
- 30
- Citations by year
- 2026: 11 · 2025: 2 · 2024: 6 · 2023: 6 · 2022: 7 · 2021: 14 · 2020: 9 · 2019: 6 · 2018: 7 · 2017: 4 · 2016: 7 · 2015: 10 · 2014: 4
- FWCI
- 16.36
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Human-Robot Interaction (HRI) (publishedVersion)
- Funders
- National Science Foundation
- University of Wisconsin-Madison
- National Institutes of Health
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- pointing gestures, human-robot interaction, robot communication, pointing, humanoid robots, gesture design, environmental factors, task conditions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.