The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Speech gesture generation from the trimodal context of text, audio, and speaker identity
Open accessAbstract
For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human-agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures. By incorporating a multimodal context and an adversarial training scheme, the proposed model outputs gestures that are human-like and that match with speech content and rhythm. We also introduce a new quantitative evaluation metric for gesture generation models. Experiments with the introduced metric and subjective human evaluation showed that the proposed gesture generation model is better than existing end-to-end generation models. We further confirm that our model is able to work with synthesized audio in a scenario where contexts are constrained, and show that different gesture styles can be generated for the same speech by specifying different speaker identities in the style embedding space that is learned from videos of various speakers. All the code and data is available at https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015
UMAP: Uniform Manifold Approximation and Projection
McInnes, L., Healy, J., Saul, N., & Großberger, L. (2018) · Journal of Open Source Software · doi:10.21105/joss.00861 · Cited by 9,826
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 7×
“FastText (Bo-janowski et al. 2017) was favored over GloVe (Pennington et al. 2014) since FastText is using subword information so that it gives accurate representation for unseen words.”
BackgroundMethodInfluentialGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Robust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
Hand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1803.01271 · Cited by 4,655
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 1×
“6M dataset (Ionescu et al. 2013) containing motion capture data of 7 dierent actors and 17 dierent scenarios including discussion and making purchases showing co-speech gestures.”
Background3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training
Pavllo, D., Feichtenhofer, C., Grangier, D., & Auli, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00794 · Cited by 1,297
Pros and cons of GAN evaluation measures
Borji, A. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.10.009 · Cited by 963
McNeill, D. (2011) · Advances in Visual Semiotics · doi:10.1515/9783110874259.351 · Cited by 927
(2018) · Human Communication Research · doi:10.1111/(issn)1468-2958 · Cited by 668
How representational gestures help speaking
Kita, S. (2000) · Language and Gesture · doi:10.1017/cbo9780511620850.011 · Cited by 637
FVD: A new Metric for Video Generation
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., & Gelly, S. (2019) · International Conference on Learning Representations · Cited by 627
Gesture and speech in interaction: An overview
Wagner, P., Malisz, Z., & Kopp, S. (2014) · Speech Communication · doi:10.1016/j.specom.2013.09.008 · Cited by 456
Towards a common framework for multimodal generation: the behavior markup language,
Kopp, S., Krenn, B., Marsella, S., Marshall, A. N., Pelachaud, C., Pirker, H., Thórisson, K. R., & Vilhjálmsson, H. (2006) · Lecture Notes in Computer Science · doi:10.1007/11821830_17 · Cited by 417
Nonverbal Behaviors, Persuasion, and Credibility
BURGOON, J. K., BIRK, T., & PFAU, M. (1990) · Human Communication Research · doi:10.1111/j.1468-2958.1990.tb00229.x · Cited by 395
Learning Individual Styles of Conversational Gesture
Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., & Malik, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00361 · Cited by 335
Language2Pose: Natural Language Grounded Pose Forecasting
Ahuja, C., & Morency, L.-P. (2019) · 2019 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2019.00084 · Cited by 271
Fréchet Audio Distance: A Metric for Evaluating Music Enhancement AlgorithmsPreprint
Kilgour, K., Zuluaga, M., Roblek, D., & Sharifi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08466 · Cited by 260
Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots
Yoon, Y., Ko, W.-R., Jang, M., Lee, J., Kim, J., & Lee, G. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793720 · Cited by 247
Self-citationDiversity-Sensitive Conditional Generative Adversarial NetworksPreprint
Yang, D., Hong, S., Jang, Y., Zhao, T., & Lee, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09024 · Cited by 233
Gesture modeling and animation based on a probabilistic re-creation of speaker style
Neff, M., Kipp, M., Albrecht, I., & Seidel, H.-P. (2008) · ACM Transactions on Graphics · doi:10.1145/1330511.1330516 · Cited by 232
Gesture generation by imitation : from human behavior to computer character animation
Kipp, M. (2003) · Publications of the UdS (Saarland University) · doi:10.22028/d291-25852 · Cited by 210
Analyzing Input and Output Representations for Speech-Driven Gesture Generation
Kucherenko, T., Hasegawa, D., Henter, G. E., Kaneko, N., & Kjellström, H. (2019) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3308532.3329472 · Cited by 199
(2020) · doi:10.1111/cgf.13946 · Cited by 176
Virtual character performance from speech
Marsella, S., Xu, Y., Lhommet, M., Feng, A., Scherer, S., & Shapiro, A. (2013) · Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/2485895.2485900 · Cited by 168
BEAT: the Behavior Expression Animation Toolkit
Cassell, J., Vilhjálmsson, H. H., & Bickmore, T. (2004) · Cognitive Technologies · doi:10.1007/978-3-662-08373-4_8 · Cited by 159
Levine, S., Krähenbühl, P., Thrun, S., & Koltun, V. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778861 · Cited by 139
Folk Dance Evaluation Using Laban Movement Analysis
Aristidou, A., Stavrakis, E., Charalambous, P., Chrysanthou, Y., & Himona, S. L. (2015) · Journal on Computing and Cultural Heritage · doi:10.1145/2755566 · Cited by 120
Hand and Mind: What Gestures Reveal About Thought.
Studdert‐Kennedy, M. (1994) · Language and Speech · doi:10.1177/002383099403700208 · Cited by 116
Learning-based modeling of multimodal behaviors for humanlike robots
Huang, C.-M., & Mutlu, B. (2014) · Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction · doi:10.1145/2559636.2559668 · Cited by 99
Multi-objective adversarial gesture generation
Ferstl, Y., Neff, M., & McDonnell, R. (2019) · Motion in Games · doi:10.1145/3359566.3360053 · Cited by 94
Predicting Co-verbal Gestures: A Deep and Temporal Modeling Approach
Chiu, C., Morency, L., & Marsella, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21996-7_17 · Cited by 91
Clark, A. P., Howard, K. L., Woods, A. T., Penton‐Voak, I. S., & Neumann, C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0190393 · Cited by 75
Effects of personality and social situation on representational gesture production
Hostetter, A. B., & Potthoff, A. L. (2012) · Gesture · doi:10.1075/gest.12.1.04hos · Cited by 74
Synchronization of speech and gesture: Evidence for interaction in action.
Chu, M., & Peter Hagoort, A. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0036281 · Cited by 69
Speech-driven animation with meaningful behaviors
Sadoughi, N., & Busso, C. (2019) · Speech Communication · doi:10.1016/j.specom.2019.04.005 · Cited by 67
The Relation of Speech and Gestures: Temporal Synchrony Follows Semantic Synchrony
von Bergmann, K., Aksu, V., & Kopp, S. (2011) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 47
Extensions of Gaussian processes for ranking: semi-supervised and active learning
Chu, W., & Ghahramani, Z. (2005) · Cited by 46
Hand Gestures and Verbal Acknowledgments Improve Human-Robot Rapport
Wilson, J. R., Lee, N. Y., Saechao, A., Hershenson, S., Scheutz, M. J., & Tickle‐Degnen, L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70022-9_33 · Cited by 35
Kim, T., & Lee, J.-H. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196948 · Cited by 19
Self-citation
Cited by
Bengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2024) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 362
Speech2Video: Cross-Modal Distillation for Speech to Video Generation
Si, S., Wang, J., Qu, X., Cheng, N., Wei, W., Zhu, X., & Xiao, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1996 · Cited by 12
Mehta, S., Deichler, A., O’Regan, J., Moëll, B., Beskow, J., Henter, G. E., & Alexanderson, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00201 · Cited by 7
Metadata
- Authors
- Yoon, Y., Cha, B., Lee, J.-H., Jang, M., Lee, J., Kim, J., & Lee, G.
- Venue
- ACM Transactions on Graphics
- Published
- 2020-11-27
- Volume
- 39
- Issue
- 6
- Pages
- 1-16
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W3083173864
- Cited by
- 328
- References
- 51
- FWCI
- 23.93
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.