The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Synthesis of Compositional Animations from Textual Descriptions
Abstract
"How can we animate 3D-characters from a movie script or move robots by simply telling them what we would like them to do?" "How unstructured and complex can we make a sentence and still generate plausible movements from it?" These are questions that need to be answered in the long-run, as the field is still in its infancy. Inspired by these problems, we present a new technique for generating compositional actions, which handles complex input sentences. Our output is a 3D pose sequence depicting the actions in the input sentence. We propose a hierarchical two-stream sequential model to explore a finer joint-level mapping between natural language sentences and 3D pose sequences corresponding to the given motion. We learn two manifold representations of the motion, one each for the upper body and the lower body movements. Our model can generate plausible pose sequences for short sentences describing single actions as well as long complex sentences describing multiple sequential and compositional actions. We evaluate our proposed model on the publicly available KIT Motion-Language Dataset containing 3D pose data with human-annotated sentences. Experimental results show that our model advances the state-of-the-art on text-based motion synthesis in objective evaluations by a margin of 50%. Qualitative evaluations based on a user study indicate that our synthesized motions are perceived to be the closest to the ground-truth motion captures for both short and compositional sentences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · cited 1×
“Different strategies have been implemented over the years using temporal convolutional neural networks [14, 41, 10], graph convolutional networks [5, 50] and recurrent neural networks [47, 26, 68, 38].”
MethodA Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600
Convolutional Sequence to Sequence Model for Human Dynamics
Li, C., Zhang, Z., Lee, W. S., & Lee, G. H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00548 · Cited by 364
Research works printed but not linked: 63
- 10.3389/fnbot.2016.00005 · DOI
- 10.1109/LRA.2018.2852838 · DOI
- xu, 2020, Hierarchical stylebased networks for motion synthesis
- 10.1609/aaai.v33i01.33018553 · DOI
- 10.1109/WACV45572.2020.9093627 · DOI
- 10.1145/3072959.3073663 · DOI · Paper page
- hermann, 2017, Grounded language learning in a simulated 3d world
- henter, 2020, Moglow: Probabilistic and controllable motion synthesis using normalising flows, ACM Transactions on Graphics (TOG), 39, 1, 10.1145/3414685.3417836 · DOI · Paper page
- 10.1109/ICRA.2018.8460699 · DOI · Paper page
- 10.1111/cgf.14028 · DOI
- 10.1145/2820903.2820918 · DOI · Paper page
- 10.1145/2897824.2925975 · DOI · Paper page
- 10.1016/j.robot.2014.12.008 · doi:10.1016/j.robot.2014.12.008
- tang, 2018, Long-term human motion prediction by modeling motion context and enhancing motion dynamic
- 10.1145/3125739.3132594 · DOI
- tenney, 2019, What do you learn from context? probing for sentence structure in contextualized word representations
- hasan, 2019, Forecasting people trajectories and head poses by jointly reasoning on tracklets and vislets, IEEE Transactions on Pattern Analysis and Machine Intelligence
- 10.1109/HUMANOIDS.2014.7041470 · doi:10.1109/humanoids.2014.7041470
- hanser, 2009, Scenemaker: Intelligent multimodal visualisation of natural language scripts, Irish Conference on Artificial Intelligence and Cognitive Science, 144
- villegas, 2017, Learning to generate longterm future via hierarchical prediction, International Conference on Machine Learning, 3560
- 10.1109/WACV.2018.00134 · DOI
- 10.1109/ICCV.2017.361 · DOI · Paper page
- 10.1109/TVCG.2019.2938520 · DOI
- 10.1109/VR.2019.8798178 · DOI
- 10.1145/3340555.3356090 · doi:10.1145/3340555.3356090
- 10.1109/ICRA.2018.8460608 · DOI
- 10.1109/ICCV.2015.494 · DOI · Paper page
- 10.1109/CVPR.2016.265 · DOI · Paper page
- gatys, 2015, Texture Synthesis Using Convolutional Neural Networks
- 10.1109/3DV.2017.00059 · DOI · Paper page
- habibie, 2017, A recurrent variational autoencoder for human motion synthesis, 28th British Machine Vision Conference
- 10.1109/CVPR42600.2020.01443 · DOI · Paper page
- pavllo, 2018, Quaternet A quaternion-based recurrent model for human motion
- 10.1177/0278364915587923 · DOI
- 10.1109/ICRA.2012.6225331 · DOI
- 10.1109/IROS.2007.4399344 · doi:10.1109/iros.2007.4399344
- socher, 2013, Zeroshot learning through cross-modal transfer
- shimada, 2020, Physcap: Physically plausible monocular 3d motion capture in real time, ACM Transactions on Graphics (TOG), 39, 1, 10.1145/3414685.3417877 · DOI · Paper page
- 10.1016/j.robot.2018.07.006 · DOI
- 10.1089/big.2016.0028 · DOI · Paper page
- cheema, 2018, Dilated temporal fully-convolutional network for semantic segmentation of motion capture data
- 10.1109/WACV.2019.00156 · DOI
- 10.1037/xap0000189 · DOI
- cho, 2014, On the properties of neural machine translation Encoder-decoder approaches
- 10.18653/v1/W19-4828 · DOI · Paper page
- 10.1016/j.ins.2020.08.123 · DOI
- 10.1109/CVPR42600.2020.00655 · DOI · Paper page
- du, 2019, Stylistic locomotion modeling with conditional variational autoencoder, In Eurographics (Short papers), 9
- du, 2015, Hierarchical recurrent neural network for skeleton based action recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1110
- 10.1016/j.cag.2020.09.009 · DOI
- 10.1109/3DV.2019.00084 · DOI · Paper page
- 10.1007/978-3-030-58523-5_15 · DOI
- 10.1145/3272127.3275038 · DOI
- 10.1109/ICCV.2019.00724 · DOI · Paper page
- 10.1109/CVPRW.2018.00191 · DOI · Paper page
- mikolov, 2013, Distributed Representations of Words and Phrases and Their Compositionality
- 10.1109/CVPR.2017.388 · DOI
- 10.1109/WACV45572.2020.9093350 · DOI
- 10.1109/TRO.2016.2572685 · DOI
- mei, 2016, Listen, attend, and walk: Neural mapping of navigational instructions to action sequences, Proceedings of the AAAI Conference on Artificial Intelligence, 30, 10.1609/aaai.v30i1.10364 · DOI · Paper page
- 10.1109/CVPR.2017.497 · DOI · Paper page
- lin, 2018, Generating animated videos of human activities from natural language descriptions, Learning, 2018, 2
- li, 2017, Auto-conditioned recurrent networks for extended complex human motion synthesis
Cited by
Generating Diverse and Natural 3D Human Motions from Text
Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., & Cheng, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00509 · Cited by 575
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355414 · Cited by 424
Executing your Commands via Motion Diffusion in Latent Space
Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., & Yu, G. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01726 · Cited by 356
TEMOS: Generating Diverse Human Motions from Textual Descriptions
Petrovich, M., Black, M. J., & Varol, G. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20047-2_28 · Cited by 353
Metadata
- Authors
- Ghosh, A., Cheema, N., Oguz, C., Theobalt, C., & Slusallek, P.
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 1376-1386
- OpenAlex
- W3144253442
- Cited by
- 169
- References
- 11
- FWCI
- 40.71
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Synthesis of Compositional Animations from Textual Descriptions. https://app.cochat.ai/papers/10.1109/iccv48922.2021.00143
- The paper
- Ghosh, A., Cheema, N., Oguz, C., Theobalt, C., & Slusallek, P. (2021). Synthesis of Compositional Animations from Textual Descriptions. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv48922.2021.00143
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.