The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Language2Pose: Natural Language Grounded Pose Forecasting
Open accessAbstract
Generating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion planning. These sentences can describe different kinds of actions, speeds and direction of these actions, and possibly a target destination. The core modeling challenge in this language-to-pose application is how to map linguistic concepts to motion animations. In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language-to-Pose (or JL2P), which learns a joint embedding of language and pose. This joint embedding space is learned end-to-end using a curriculum learning approach which emphasizes shorter and easier sequences first before moving to longer and harder ones. We evaluate our proposed model on a publicly available corpus of 3D pose data and human-annotated sentences. Both objective metrics and human judgment evaluation confirm that our proposed approach is able to generate more accurate animations and are deemed visually more representative by humans than other data driven approaches.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Each token of the sentence is converted into a distributional embedding using a pre-trained Word2Vec model [22].”
MethodRobust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“Starting with simpler examples before moving on to tougher ones can be beneficial to the training process [4, 37, 36].”
BackgroundNote on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.471 · Cited by 2,688
Robust Estimation of a Location Parameter
Huber, P. J. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_35 · Cited by 1,455
Wright, S. J. (2015) · Mathematical Programming · doi:10.1007/s10107-015-0892-3 · Cited by 1,438
Hand Keypoint Detection in Single Images Using Multiview Bootstrapping
Simon, T., Joo, H., Matthews, I., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.494 · Cited by 1,119
A deep learning framework for character motion synthesis and editing
Holden, D., Saito, J., & Komura, T. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925975 · Cited by 655
Pose Guided Person Image Generation
Ma, L., Jia, X., Sun, Q., Schiele, B., Tuytelaars, T., & Gool, L. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09368 · Cited by 590
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.497 · Cited by 563
Phase-functioned neural networks for character control
Holden, D., Komura, T., & Saito, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073663 · Cited by 546
A deep learning approach for generalized speech animation
Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J., & Matthews, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303
Zaremba, W., & Sutskever, I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.4615 · Cited by 283
The KIT Motion-Language Dataset
Plappert, M., Mandery, C., & Asfour, T. (2016) · Big Data · doi:10.1089/big.2016.0028 · Cited by 272
Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
Yang, J., Reed, S., Yang, M., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.00706 · Cited by 246
Action-Agnostic Human Pose Forecasting
Chiu, H.-K., Adeli, E., Wang, B., Huang, D.-A., & Niebles, J. C. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00156 · Cited by 212
Learning Human Motion Models for Long-Term Predictions
Ghosh, P., Song, J., Aksan, E., & Hilliges, O. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00059 · Cited by 207
Text2Action: Generative Adversarial Synthesis from Language to Action
Ahn, H., Ha, T., Choi, Y., Yoo, H., & Oh, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460608 · Cited by 197
QuaterNet: A Quaternion-based Recurrent Model for Human Motion
Pavllo, D., Grangier, D., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06485 · Cited by 174
Plappert, M., Mandery, C., & Asfour, T. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.07.006 · Cited by 168
Pose Guided Human Video Generation
Yang, C., Wang, Z., Zhu, X., Huang, C., Shi, J., & Lin, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_13 · Cited by 157
Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamics
Tang, Y., Ma, L., Liu, W., & Zheng, W. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/130 · Cited by 139
Forecasting Human Dynamics from Static Images
Chao, Y.-W., Yang, J., Price, B. L., Cohen, S. D., & Deng, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.388 · Cited by 133
Generating Animated Videos of Human Activities from Natural Language Descriptions
Lin, A. S., Wu, L., Corona, R., Tai, K., Huang, Q., & Mooney, R. J. (2018) · Cited by 105
Human Motion Modeling using DVGANsPreprint
Lin, X., & Amer, M. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.10652 · Cited by 101
Predicting Co-verbal Gestures: A Deep and Temporal Modeling Approach
Chiu, C., Morency, L., & Marsella, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21996-7_17 · Cited by 91
Self-citationHow to Train Your Avatar: A Data Driven Approach to Gesture Generation
Chiu, C., & Marsella, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23974-8_14 · Cited by 87
Yamada, T., Matsunaga, H., & Ogata, T. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2852838 · Cited by 59
Takano, W., & Nakamura, Y. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364915587923 · Cited by 46
Takeuchi, K., Hasegawa, D., Shirakawa, S., Kaneko, N., Sakuta, H., & Sumi, K. (2017) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3125739.3132594 · Cited by 38
Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion DynamicPreprint
Tang, Y., Ma, L., Liu, W., & Zheng, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.02513 · Cited by 33
Generating Animations from Screenplays
Zhang, Y., Tsipidi, E., Schriber, S., Kapadia, M., Groß, M. H., & Modi, A. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1032 · Cited by 28
SceneMaker: Intelligent Multimodal Visualisation of Natural Language Scripts
Hanser, E., Kevitt, P. M., Lunney, T., & Condell, J. V. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-17080-5_17 · Cited by 24
Takano, W., & Nakamura, Y. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225331 · Cited by 16
Cited by
Generating Diverse and Natural 3D Human Motions from Text
Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., & Cheng, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00509 · Cited by 575
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
Petrovich, M., Black, M. J., & Varol, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01080 · Cited by 493
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355414 · Cited by 424
Action2Motion: Conditioned Generation of 3D Human Motions
Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., & Cheng, L. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413635 · Cited by 398
Metadata
- Authors
- Ahuja, C., & Morency, L.-P.
- Venue
- 2019 International Conference on 3D Vision (3DV)
- Publisher
- IEEE
- Published
- 2019-09
- Pages
- 719-728
- OpenAlex
- W2954878489
- Cited by
- 271
- References
- 37
- FWCI
- 15.97
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.