The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 7 of the 23 sources its claims cite were checked for support.
Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents
Open accessAbstract
This paper focuses on enhancing human-agent communication by integrating spatial context into virtual agents’ non-verbal behaviors, specifically gestures. Recent advances in co-speech gesture generation have primarily utilized data-driven methods, which create natural motion but limit the scope of gestures to those performed in a void. Our work aims to extend these methods by enabling generative models to incorporate scene information into speech-driven gesture synthesis. We introduce a novel synthetic gesture dataset tailored for this purpose. This development represents a critical step toward creating embodied conversational agents that interact more naturally with their environment and users.
Overview
CoChat Summary
TL;DR
This paper introduces a synthetic gesture dataset that augments an existing co-speech gesture dataset with pointing gestures and synthesized demonstrative speech, aiming to enable spatial-context-aware gesture generation for virtual agents. It reports dataset construction details but no model training or evaluation results.
Key findings
- Combined a ~6-hour single-speaker co-speech motion/speech dataset with a pointing dataset of ~140 isolated single-target pointing gestures (0.5–9 s), then mirrored and time-stretched clips in [0.7, 1.6] three times to produce 1,160 pointing clips; mean pointing gesture duration was 4.85 s.
- Generated 1,400 demonstrative sentences with GPT-4, synthesized speech with a commercial TTS engine, and matched speech to pointing gestures using the Hungarian algorithm with cost based on length difference and demonstrative alignment.
- The resulting synthetic dataset is proposed as a benchmark for joint co-speech and referential gesture generation, but the paper reports no trained model or evaluation outcomes.
Methods
- Dataset construction: combined existing co-speech gesture dataset [11] (~6 h synchronized motion capture and speech, single English speaker) with pointing gesture dataset [7] (~140 isolated single-target pointing gestures, 0.5–9 s, with mocap, 3D target locations, and synchronized speech).
- Augmentation and synthesis: mirrored and time-stretched pointing clips three times in [0.7, 1.6] to 1,160 clips; used GPT-4 with beat-dataset transcripts as context to generate exophoric demonstrative references; synthesized 1,400 sentences with TTS and SSML emphasis; matched speech to motion via Hungarian algorithm and padded speech with silence to animation length.
- Analysis: descriptive comparison of length distributions and demonstrative alignment (Figures 1–3); no generative model training or quantitative evaluation reported.
Limitations
- Stated: the original pointing dataset was small; there is a lack of datasets with relevant environment/spatial context; future datasets need scene annotations (objects, spatial configurations) and sophisticated data handling for generalization; evaluation methods for situated gestures are not yet established.
- Stated: the work is a first step and the synthetic dataset is simple; more datasets are needed for naturalistic situated gesture.
- Evident from text: the paper does not train or evaluate a spatial-context-conditioned gesture generation model, and the presented dataset construction does not itself include scene/object annotations for the combined data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
5 Valid · 3 Not valid · 4 Not checked · 9 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As can be seen from Figures 1 and 3, there is a discrepancy between animation and audio lengths. This is intentional since we wanted the speech segments to be fully contained within the animation clips after the demonstratives are aligned in the two streams.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We introduce a novel synthetic gesture dataset tailored for this purpose.”
- Also stated in: § CCS CONCEPTS · plain
“In this paper, we propose a novel synthetic gesture dataset that augments an existing co-speech dataset with multimodal referring expressions, as a first step towards extending speech-driven gesture synthesis with scene information conditioning.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § CCS CONCEPTS · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To account for the small size of this dataset, all motion files were mirrored and each of the individual pointing gesture clips was stretched in time in the range of [0.7, 1.6] three times. This resulted in a pointing gesture dataset of 1160 clips.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The original pointing dataset contained approximately 140 isolated single target pointing gestures, ranging from 0.5 to 9 seconds in duration.”
What the sources say
- Not matchedcites [7] [7] Anna Deichler, Siyang Wang, Simon Alexanderson, and Jonas Beskow. 2022. Towards context-aware human-like pointing gestures with rl motion imitation. In Context-Awareness in Human-Robot Interaction: Approaches and Challenges, workshop at 2022 ACM/IEEE International Conference on Human-Robot Interaction . -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“We utilized the solely available dataset that includes both motion capture data of pointing gestures and the corresponding 3D target locations [7].”
What the sources say
- Not matchedcites [7] [7] Anna Deichler, Siyang Wang, Simon Alexanderson, and Jonas Beskow. 2022. Towards context-aware human-like pointing gestures with rl motion imitation. In Context-Awareness in Human-Robot Interaction: Approaches and Challenges, workshop at 2022 ACM/IEEE International Conference on Human-Robot Interaction . -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
1 more claim
Create a free account to see all 21 claims and every source verdict.
- It may state two separate claims as one. 6
- Probably not a substantial claim. 10
- The quote only partly states this claim. 1
Citations
Cites
BEAT: the Behavior Expression Animation Toolkit
Cassell, J., Vilhjálmsson, H. H., & Bickmore, T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383315 · Cited by 739
Animated agents for procedural training in virtual reality: Perception, cognition, and motor control
Rickel, J., & Johnson, W. L. (1999) · Applied Artificial Intelligence · doi:10.1080/088395199117315 · Cited by 480
Synthesizing multimodal utterances for conversational agents
Kopp, S., & Wachsmuth, I. (2004) · Computer Animation and Virtual Worlds · doi:10.1002/cav.6 · Cited by 243 · cited 1×
“Early work in gesture synthesis for virtual agents was rule-based, based on inverse kinematics and procedural animation to generate co-speech gestures [5, 15, 20].”
HUMANISE: Language-Conditioned Human Motion Generation in 3D Scenes
Wang, Z., Chen, Y., Liu, T., Zhu, Y., Liang, W., & Huang, S. (2022) · neural information processing systems · doi:10.52202/068431-1088 · Cited by 208
Diffusion-based Generation, Optimization, and Planning in 3D Scenes
Huang, S., Wang, Z., Li, P., Jia, B., Liu, T., Zhu, Y., Liang, W., & Zhu, S.-C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01607 · Cited by 192 · cited 2×
“In recent years there has been more focus on scene-conditioned human motion generation [13, 26, 27].”
Generating 3D People in Scenes Without People
Zhang, Y., Hassan, M. S., Neumann, H., Black, M. J., & Tang, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00623 · Cited by 183 · cited 1×
“These works in human-scene interaction focus mainly on non-conversational human locomotion generation, ranging from generating static human poses in 3D environments [29] to temporally extended human motion synthesis, conditioning on the 3D layout of the environment and human-object interaction [28].”
(2020) · doi:10.1111/cgf.13946 · Cited by 176
Self-citationVirtual character performance from speech
Marsella, S., Xu, Y., Lhommet, M., Feng, A., Scherer, S., & Shapiro, A. (2013) · Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/2485895.2485900 · Cited by 168 · cited 1×
“Early work in gesture synthesis for virtual agents was rule-based, based on inverse kinematics and procedural animation to generate co-speech gestures [5, 15, 20].”
Gesticulator: A framework for semantically-aware speech-driven gesture generation
Kucherenko, T., Jonell, P., van Waveren, S., Henter, G. E., Alexandersson, S., Leite, I., & Kjellström, H. (2020) · Proceedings of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3382507.3418815 · Cited by 166
Deictic believability: Coordinated gesture, locomotion, and speech in lifelike pedagogical agents
Lester, J. C., Voerman, J. L., Towns, S. G., & Callaway, C. B. (1999) · Applied Artificial Intelligence · doi:10.1080/088395199117324 · Cited by 161
Research works printed but not linked: 1
- Anna Deichler, Siyang Wang, Simon Alexanderson, and Jonas Beskow. 2022. Towards context-aware human-like pointing gestures with rl motion imitation. In Context-Awareness in Human-Robot Interaction: Approaches and Challenges, workshop at 2022 ACM/IEEE International Conference on Human-Robot Interaction.
Cited by
Grounded Gesture Generation: Language, Motion, and SpacePreprint
Deichler, A., O’Regan, J., Guichoux, T., Johansson, D., & Beskow, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.04522 · Cited by 1
“The paper [5] proposes integrating spatial context into virtual agents’ gesture generation by combining co-speech and pointing gestures.”
Self-citationHielscher, T., Scaparro, F., & Arras, K. O. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3757279.3785635 · Cited by 3
“These can incorporate multi-modal temporal [5], semantic [17, 42], or spatial grounding [8].”
InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control
Pinyoanuntapong, E., Deogade, A., Streli, P., et al. (2026) · doi:10.48550/arxiv.2608.25734
Metadata
- Authors
- Deichler, A., Alexanderson, S., & Beskow, J.
- Affiliations
- KTH Royal Institute of Technology (SE)
- Venue
- Proceedings of the ACM International Conference on Intelligent Virtual Agents
- Publisher
- ACM
- Published
- 2024-09-16
- Pages
- 1-4
- OpenAlex
- W4403443375
- Cited by
- 0
- References
- 20
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Intelligent Virtual Agents (IVA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- gesture generation, virtual agents, co-speech gestures, spatial awareness, scene context, nonverbal communication, embodied conversational agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents. https://app.cochat.ai/papers/10.1145/3652988.3673936
- The paper
- Deichler, A., Alexanderson, S., & Beskow, J. (2024). Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents. Proceedings of the ACM International Conference on Intelligent Virtual Agents. https://doi.org/10.1145/3652988.3673936
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.