Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 7 of the 23 sources its claims cite were checked for support.

Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents

Open access
Deichler, Anna, Alexanderson, Simon, Beskow, Jonas · 2024 · Proceedings of the ACM International Conference on Intelligent Virtual Agents

Abstract

This paper focuses on enhancing human-agent communication by integrating spatial context into virtual agents’ non-verbal behaviors, specifically gestures. Recent advances in co-speech gesture generation have primarily utilized data-driven methods, which create natural motion but limit the scope of gestures to those performed in a void. Our work aims to extend these methods by enabling generative models to incorporate scene information into speech-driven gesture synthesis. We introduce a novel synthetic gesture dataset tailored for this purpose. This development represents a critical step toward creating embodied conversational agents that interact more naturally with their environment and users.

Overview

CoChat Summary

TL;DR

This paper introduces a synthetic gesture dataset that augments an existing co-speech gesture dataset with pointing gestures and synthesized demonstrative speech, aiming to enable spatial-context-aware gesture generation for virtual agents. It reports dataset construction details but no model training or evaluation results.

Key findings

  • Combined a ~6-hour single-speaker co-speech motion/speech dataset with a pointing dataset of ~140 isolated single-target pointing gestures (0.5–9 s), then mirrored and time-stretched clips in [0.7, 1.6] three times to produce 1,160 pointing clips; mean pointing gesture duration was 4.85 s.
  • Generated 1,400 demonstrative sentences with GPT-4, synthesized speech with a commercial TTS engine, and matched speech to pointing gestures using the Hungarian algorithm with cost based on length difference and demonstrative alignment.
  • The resulting synthetic dataset is proposed as a benchmark for joint co-speech and referential gesture generation, but the paper reports no trained model or evaluation outcomes.

Methods

  • Dataset construction: combined existing co-speech gesture dataset [11] (~6 h synchronized motion capture and speech, single English speaker) with pointing gesture dataset [7] (~140 isolated single-target pointing gestures, 0.5–9 s, with mocap, 3D target locations, and synchronized speech).
  • Augmentation and synthesis: mirrored and time-stretched pointing clips three times in [0.7, 1.6] to 1,160 clips; used GPT-4 with beat-dataset transcripts as context to generate exophoric demonstrative references; synthesized 1,400 sentences with TTS and SSML emphasis; matched speech to motion via Hungarian algorithm and padded speech with silence to animation length.
  • Analysis: descriptive comparison of length distributions and demonstrative alignment (Figures 1–3); no generative model training or quantitative evaluation reported.

Limitations

  • Stated: the original pointing dataset was small; there is a lack of datasets with relevant environment/spatial context; future datasets need scene annotations (objects, spatial configurations) and sophisticated data handling for generalization; evaluation methods for situated gestures are not yet established.
  • Stated: the work is a first step and the synthetic dataset is simple; more datasets are needed for naturalistic situated gesture.
  • Evident from text: the paper does not train or evaluate a spatial-context-conditioned gesture generation model, and the presented dataset construction does not itself include scene/object annotations for the combined data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3652988.3673936

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Distribution of length of pointing motion clips and synthesized audio clips.
Figure 1 Distribution of length of pointing motion clips and synthesized audio clips.From the PDF
Figure 2 Alignment of demonstratives in motion and speech, with pre-and post-padding of audio clips.
Figure 2 Alignment of demonstratives in motion and speech, with pre-and post-padding of audio clips.From the PDF
Figure 3 Demonstrative location against total length (in seconds) for motion (BVH) and audio files.
Figure 3 Demonstrative location against total length (in seconds) for motion (BVH) and audio files.From the PDF
Figure 4 Visualization of dataset examples containing synchronized clips of audio and gesture for (a) pointing and (b) beat gestures.
Figure 4 Visualization of dataset examples containing synchronized clips of audio and gesture for (a) pointing and (b) beat gestures.From the PDF
Image 5
From the PDF

Verification

Claims the paper makes

5 Valid · 3 Not valid · 4 Not checked · 9 Own results and interpretations

1 more claim

Create a free account to see all 21 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 6
  • Probably not a substantial claim. 10
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 20 references

Research works printed but not linked: 1

  1. Anna Deichler, Siyang Wang, Simon Alexanderson, and Jonas Beskow. 2022. Towards context-aware human-like pointing gestures with rl motion imitation. In Context-Awareness in Human-Robot Interaction: Approaches and Challenges, workshop at 2022 ACM/IEEE International Conference on Human-Robot Interaction.

Cited by

See all 3

Metadata

Authors
Deichler, A., Alexanderson, S., & Beskow, J.
Affiliations
KTH Royal Institute of Technology (SE)
Venue
Proceedings of the ACM International Conference on Intelligent Virtual Agents
Publisher
ACM
Published
2024-09-16
Pages
1-4
OpenAlex
W4403443375
Cited by
0
References
20
FWCI
0.00
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Intelligent Virtual Agents (IVA) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
gesture generation, virtual agents, co-speech gestures, spatial awareness, scene context, nonverbal communication, embodied conversational agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents. https://app.cochat.ai/papers/10.1145/3652988.3673936
The paper
Deichler, A., Alexanderson, S., & Beskow, J. (2024). Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents. Proceedings of the ACM International Conference on Intelligent Virtual Agents. https://doi.org/10.1145/3652988.3673936

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.