A deep learning framework for character motion synthesis and editing
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
We present a framework to synthesize character movements based on high level parameters, such that the produced movements respect the manifold of human motion, trained on a large motion capture dataset. The learned motion manifold, which is represented by the hidden units of a convolutional autoencoder, represents motion data in sparse components which can be combined to produce a wide range of complex movements. To map from high level parameters to the motion manifold, we stack a deep feedforward neural network on top of the trained autoencoder. This network is trained to produce realistic motion sequences from parameters such as a curve over the terrain that the character should follow, or a target location for punching and kicking. The feedforward control network and the motion manifold are trained independently, allowing the user to easily switch between feedforward networks according to the desired interface, without re-training the motion manifold. Once motion is generated it can be edited by performing optimization in the space of the motion manifold. This allows for imposing kinematic constraints, or transforming the style of the motion, while ensuring the edited motion remains natural. As a result, the system can produce smooth, high quality motion sequences without any manual pre-processing of the training data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
(2014) · Cited by 33,419
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
(2010) · International Conference on Machine Learning · Cited by 13,995
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
A Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852
A Neural Algorithm of Artistic Style
Gatys, L., Ecker, A., & Bethge, M. (2016) · Journal of Vision · doi:10.1167/16.12.326 · Cited by 2,612
Hierarchical recurrent neural network for skeleton based action recognition
Du, Y., Wang, W., & Wang, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298714 · Cited by 1,895
Hypercolumns for object segmentation and fine-grained localization
Hariharan, B., Arbelaez, P., Girshick, R., & Malik, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298642 · Cited by 1,511
Interactive control of avatars animated with human motion data
Lee, J., Chai, J., Reitsma, P. S. A., Hodgins, J. K., & Pollard, N. S. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566607 · Cited by 864
Recurrent Network Models for Human Dynamics
Fragkiadaki, K., Levine, S., Felsen, P., & Malik, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.494 · Cited by 860
Verbs and adverbs: multidimensional motion interpolation
Rose, C., Cohen, M. F., & Bodenheimer, B. (1998) · IEEE Computer Graphics and Applications · doi:10.1109/38.708559 · Cited by 730
Style-based inverse kinematics
Grochow, K., Martin, S. L., Hertzmann, A., & Popović, Z. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015755 · Cited by 642
Kovar, L., Gleicher, M., & Pighin, F. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566605 · Cited by 597
A hierarchical approach to interactive motion editing for human-like figures
Lee, J., & Shin, S. Y. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311539 · Cited by 566
Interactive motion generation from examples
Arikan, O., & Forsyth, D. A. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566606 · Cited by 539
Berkeley MHAD: A comprehensive Multimodal Human Action Database
Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., & Bajcsy, R. (2013) · 2013 IEEE Workshop on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2013.6474999 · Cited by 484
Automated extraction and parameterization of motions in large data sets
Kovar, L., & Gleicher, M. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015760 · Cited by 483
Interactive control of avatars animated with human motion data
Lee, J., Chai, J., Reitsma, P. S. A., Hodgins, J. K., & Pollard, N. S. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566607 · Cited by 467
Gaussian Process Dynamical Models
Wang, J., Hertzmann, A., & Blei, D. M. (2005) · Cited by 438
Interactive motion generation from examples
Arikan, O., & Forsyth, D. A. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566606 · Cited by 391
Documentation Mocap Database HDM05
Müller, M., Röder, T., Clausen, M. V., Eberhardt, B., Krüger, B., & Weber⋆, A. (2007) · Cited by 378
Factored conditional restricted Boltzmann Machines for modeling motion style
Taylor, G. W., & Hinton, G. E. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553505 · Cited by 358
Learning motion manifolds with convolutional autoencoders
Holden, D., Saito, J., Komura, T., & Joyce, T. (2015) · SIGGRAPH Asia 2015 Technical Briefs · doi:10.1145/2820903.2820918 · Cited by 263
Self-citationHeck, R., & Gleicher, M. (2007) · Proceedings of the 2007 symposium on Interactive 3D graphics and games · doi:10.1145/1230100.1230123 · Cited by 209
Natural motion animation through constraining and deconstraining at will
Yamane, K., & Nakamura, Y. (2003) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2003.1207443 · Cited by 203
Realtime style transfer for unlabeled heterogeneous human motion
Xia, S., Wang, C., Chai, J., & Hodgins, J. (2015) · ACM Transactions on Graphics · doi:10.1145/2766999 · Cited by 201
Geostatistical motion interpolation
Mukai, T., & Kuriyama, S. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073313 · Cited by 199
Motion graphs++: a compact generative model for semantic motion analysis and synthesis
Min, J., & Chai, J. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366172 · Cited by 197
Continuous character control with low-dimensional embeddings
Levine, S., Wang, J. M., Haraux, A., Popović, Z., & Koltun, V. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185524 · Cited by 170
Precomputing avatar behavior from human motion data
Lee, J., & Lee, K. H. (2004) · Computer animation and simulation · doi:10.1145/1028523.1028535 · Cited by 168
Artist-Directed Inverse-Kinematics Using Radial Basis Function Interpolation
Rose, C. F., Sloan, P. J., & Cohen, M. F. (2001) · Computer Graphics Forum · doi:10.1111/1467-8659.00516 · Cited by 154
Two Distributed-State Models For Generating High-Dimensional Time Series
Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2021035 · Cited by 151
Learning Complex Neural Network Policies with Trajectory Optimization
Levine, S., & Koltun, V. (2014) · International Conference on Machine Learning · Cited by 150
Synchronized multi-character motion editing
Kim, M., Hyun, K., Kim, J., & Lee, J. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531385 · Cited by 114
Tan, J., Gu, Y., Liu, C. K., & Turk, G. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601121 · Cited by 95
Motion fields for interactive character locomotion
Lee, Y., Wampler, K., Bernstein, G., Popović, J., & Popović, Z. (2010) · ACM Transactions on Graphics · doi:10.1145/1882261.1866160 · Cited by 91
Interactive control of diverse complex characters with neural networks
Mordatch, I., Lowrey, K., Andrew, G., Popović, Z., & Todorov, E. V. (2015) · neural information processing systems · Cited by 74
Structured Recurrent Temporal Restricted Boltzmann Machines
Mittelman, R., Kuipers, B. J., Savarese, S., & Lee, H. (2014) · Cited by 61
Cited by
AMASS: Archive of Motion Capture As Surface Shapes
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00554 · Cited by 1,323
DeepMimic: example-guided deep reinforcement learning of physics-based character skills
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 880
Phase-functioned neural networks for character control
Holden, D., Komura, T., & Saito, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073663 · Cited by 546
Self-citationAI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Li, R., Yang, S., Ross, D. A., & Kanazawa, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01315 · Cited by 553
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
Petrovich, M., Black, M. J., & Varol, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01080 · Cited by 493
Metadata
- Authors
- Holden, D., Saito, J., & Komura, T.
- Venue
- ACM Transactions on Graphics
- Published
- 2016-07-11
- Volume
- 35
- Issue
- 4
- Pages
- 1-11
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2469134594
- Cited by
- 655
- References
- 43
- FWCI
- 40.96
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.