The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network
Abstract
We present a novel framework to automatically generate natural gesture motions accompanying speech from audio utterances. Based on a Bi-Directional LSTM Network, our deep network learns speech-gesture relationships with both backward and forward consistencies over a long period of time. Our network regresses a full 3D skeletal pose of a human from perceptual features extracted from the input audio in each time step. Then, we apply combined temporal filters to smooth out the generated pose sequences. We utilize a speech-gesture dataset recorded with a headset and marker-based motion capture to train our network. We validated our approach with a subjective evaluation and compared it against "original" human gestures and "mismatched" human gestures taken from a different utterance. The evaluation result shows that our generated gestures are significantly better than the "mismatched" gestures with respect to time consistency. The generated gesture also shows marginally significant improvement in terms of semantic consistency when compared to "mismatched" gestures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recent advances in co-speech gesture generation have primarily utilized data-driven methods, which create natural motion but limit the scope of gestures to those performed in a void.”
From Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents (2024) · cites this paper as [12]
In the citing paper
“Most of these systems take audio as input [3, 10, 12], and are limited to generating beat gestures aligned with speech.”
From Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents (2024) · cites this paper as [12]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
(2010) · International Conference on Machine Learning · Cited by 13,995
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,839
Davis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465 · cited 3×
“As shown, first,aspeechaudioofonesentence(.wav)isconvertedtoMFCC[9] featurevectorsforeachtimewindow.”
MethodInfluentialHand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
Survey on speech emotion recognition: Features, classification schemes, and databases
El Ayadi, M., Kamel, M. S., & Karray, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,073
Embodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Research works printed but not linked: 7
- 10.5555/2041666.2041683 · doi:10.5555/2041666.2041683
- Han Cheolho, 2017, Proceedings of the Linguistic And Cognitive Approaches To Dialog Agents (LaCATODA), Workshop of the 26th International Joint Conference on Artificial Intelligence (IJCAI
- Awni Hannun Carl Case Jared Casper Bryan Catanzaro Greg Diamos Erich Elsen Ryan Prenger Sanjeev Satheesh Shubho Sengupta Adam Coates etal 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567. Awni Hannun Carl Case Jared Casper Bryan Catanzaro Greg Diamos Erich Elsen Ryan Prenger Sanjeev Satheesh Shubho Sengupta Adam Coates et al. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567. · arXiv
- Ioffe Sergey, 2015, Proceedings of the International Conference on Machine Learning(ICML). 448--456
- Kingma Diederik, 2015, Proceedings of the International Conference on Learning Representations (ICLR).
- Nair Vinod, 2010, Proceedings of the International Conference on Machine Learning(ICML). 807--814
- 10.5555/2627435.2670313 · doi:10.5555/2627435.2670313
Cited by
Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
Alexanderson, S., Nagy, R., Beskow, J., & Henter, G. E. (2023) · ACM Transactions on Graphics · doi:10.1145/3592458 · Cited by 194
Gesticulator: A framework for semantically-aware speech-driven gesture generation
Kucherenko, T., Jonell, P., van Waveren, S., Henter, G. E., Alexandersson, S., Leite, I., & Kjellström, H. (2020) · Proceedings of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3382507.3418815 · Cited by 166
MoGlow: probabilistic and controllable motion synthesis using normalising flows
Henter, G. E., Alexanderson, S., & Beskow, J. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417836 · Cited by 176
Analyzing Input and Output Representations for Speech-Driven Gesture Generation
Kucherenko, T., Hasegawa, D., Henter, G. E., Kaneko, N., & Kjellström, H. (2019) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3308532.3329472 · Cited by 171
Self-citation(2020) · doi:10.1111/cgf.13946 · Cited by 176
Metadata
- Authors
- Hasegawa, D., Kaneko, N., Shirakawa, S., Sakuta, H., & Sumi, K.
- Affiliations
- Hokkai Gakuen University (JP)
- Aoyama Gakuin University (JP)
- Yokohama National University (JP)
- Venue
- Proceedings of the 18th International Conference on Intelligent Virtual Agents
- Publisher
- ACM
- Published
- 2018-11-05
- Pages
- 79-86
- OpenAlex
- W2901872500
- Cited by
- 131
- References
- 26
- Citations by year
- 2026: 5 · 2025: 17 · 2024: 21 · 2023: 20 · 2022: 23 · 2021: 20 · 2020: 13 · 2019: 12
- FWCI
- 16.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent Virtual Agents (IVA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- gesture synthesis, bidirectional LSTM, temporal smoothing, marker-based motion capture, time consistency, semantic consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network. https://app.cochat.ai/papers/10.1145/3267851.3267878
- The paper
- Hasegawa, D., Kaneko, N., Shirakawa, S., Sakuta, H., & Sumi, K. (2018). Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network. Proceedings of the 18th International Conference on Intelligent Virtual Agents. https://doi.org/10.1145/3267851.3267878
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.