The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
DeepMimic: example-guided deep reinforcement learning of physics-based character skills
Open accessAbstract
A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We show that well-known reinforcement learning (RL) methods can be adapted to learn robust control policies capable of imitating a broad range of example motion clips, while also learning complex recoveries, adapting to changes in morphology, and accomplishing user-specified goals. Our method handles keyframed motions, highly-dynamic actions such as motion-captured flips and spins, and retargeted motions. By combining a motion-imitation objective with a task objective, we can train characters that react intelligently in interactive settings, e.g., by walking in a desired direction or throwing a ball at a user-specified target. This approach thus combines the convenience and motion quality of using motion clips to define the desired style and appearance, with the flexibility and generality afforded by RL methods and physics-based animation. We further explore a number of methods for integrating multiple clips into the learning process to develop multi-skilled agents capable of performing a rich repertoire of diverse skills. We demonstrate results using multiple characters (human, Atlas robot, bipedal dinosaur, dragon) and a large variety of skills, including locomotion, acrobatics, and martial arts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 4×
“Policy gradient methods have emerged as the algorithms of choice for many continuous control problems [Schulman et al. 2015a, 2017; Sutton and Barto 1998].”
MethodInfluentialIntroduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
(1999) · Cited by 5,147
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Self-citationSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
High-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103
Self-citationGenerative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
Benchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114
Self-citationLearning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Self-citationSynthesis and stabilization of complex behaviors through online trajectory optimization
Tassa, Y., Erez, T., & Todorov, E. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386025 · Cited by 744
Overcoming Exploration in Reinforcement Learning with Demonstrations
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463162 · Cited by 712
Self-citationEmergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657
A deep learning framework for character motion synthesis and editing
Holden, D., Saito, J., & Komura, T. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925975 · Cited by 655
Phase-functioned neural networks for character control
Holden, D., Komura, T., & Saito, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073663 · Cited by 546
Iterative Training of Dynamic Skills Inspired by Human Coaching Techniques
Ha, S., & Liu, C. K. (2014) · ACM Transactions on Graphics · doi:10.1145/2682626 · Cited by 535
peng (2017) · doi:10.1145/3072959.3073602 · Cited by 502
VIME: Variational Information Maximizing Exploration
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09674 · Cited by 472
Self-citationDiscovery of complex behaviors through contact-invariant optimization
Mordatch, I., Todorov, E., & Popović, Z. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185539 · Cited by 451
EPOpt: Learning Robust Neural Network Policies Using Model EnsemblesPreprint
Rajeswaran, A., Ghotra, S., Ravindran, B., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01283 · Cited by 373
Self-citationSIMBICON: simple biped locomotion control
Yin, K., Loken, K., & van de Panne, M. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276509 · Cited by 337
Self-citationSIMBICON: simple biped locomotion control
Yin, K., Loken, K., & van de Panne, M. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276509 · Cited by 328
Self-citationOptimizing locomotion controllers using biologically-based actuators and objectives
Wang, J. M., Hamner, S. R., Delp, S. L., & Koltun, V. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185521 · Cited by 282
Unifying Count-Based Exploration and Intrinsic Motivation
Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016) · arXiv · doi:10.48550/arxiv.1606.01868 · Cited by 264
Learning human behaviors from motion capture by adversarial imitationPreprint
Merel, J., Tassa, Y., Dhruva, T., Srinivasan, S., Lemmon, J., Wang, Z.-Y., Wayne, G., & Heess, N. (2017) · arXiv.org · doi:10.48550/arxiv.1707.02201 · Cited by 224
Generalized biped walking control
Coros, S., Beaudoin, P., & van de Panne, M. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1781156 · Cited by 218
Self-citationTerrain-adaptive locomotion skills using deep reinforcement learning
Peng, X. B., Berseth, G., & van de Panne, M. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925881 · Cited by 200
Self-citationLearning and Transfer of Modulated Locomotor ControllersPreprint
Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., & Silver, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05182 · Cited by 198
Learning locomotion skills using DeepRL: does the choice of action space matter?
Peng, X. B., & van de Panne, M. (2017) · ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) · doi:10.1145/3099564.3099567 · Cited by 174
Self-citationDistral: Robust Multitask Reinforcement Learning
Teh, Y. W., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R. T., Heess, N., & Pascanu, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04175 · Cited by 172
Continuous character control with low-dimensional embeddings
Levine, S., Wang, J. M., Haraux, A., Popović, Z., & Koltun, V. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185524 · Cited by 170
Self-citationSimulating biped behaviors from human motion data
Sok, K. W., Kim, M., & Lee, J. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276511 · Cited by 170
Contact-aware nonlinear control of dynamic characters
Muico, U., Lee, Y., Popović, J., & Popović, Z. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531387 · Cited by 170
Lee, Y., Kim, S., & Lee, J. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1781155 · Cited by 169
Sampling-based contact-rich motion control
Liu, L., Yin, K., van de Panne, M., Shao, T., & Xu, W. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778865 · Cited by 161
Guided Learning of Control Graphs for Physics-Based Characters
Liu, L., van de Panne, M., & Yin, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2893476 · Cited by 146
Simulating biped behaviors from human motion data
Sok, K. W., Kim, M., & Lee, J. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276511 · Cited by 143
Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning
Liu, L., & Hodgins, J. K. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3083723 · Cited by 120
Stable Proportional-Derivative Controllers
Tan, J., Liu, C. K., & Turk, G. (2011) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2011.30 · Cited by 105
Online control of simulated humanoids using particle belief propagation
Hämäläinen, P., Rajamäki, J., & Liu, C. K. (2015) · ACM Transactions on Graphics · doi:10.1145/2767002 · Cited by 103
Locomotion control for many-muscle humanoids
Lee, Y., Park, M. S., Kwon, T., & Lee, J. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661233 · Cited by 99
Optimal feedback control for character animation using an abstract model
Ye, Y., & Liu, C. K. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778811 · Cited by 97
Robust task-based control policies for physics-based characters
Coros, S., Beaudoin, P., & van de Panne, M. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618516 · Cited by 96
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
Fu, J., Co-Reyes, J. D., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01260 · Cited by 95
Self-citationMotion fields for interactive character locomotion
Lee, Y.-J., Wampler, K., Bernstein, G., Popović, J., & Popović, Z. (2010) · ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia) · doi:10.1145/1866158.1866160 · Cited by 85
Dynamic terrain traversal skills using reinforcement learning
Peng, X. B., Berseth, G., & van de Panne, M. (2015) · ACM Transactions on Graphics · doi:10.1145/2766910 · Cited by 80
Self-citationSynthesis of Responsive Motion Using a Dynamic Model
Ye, Y., & Liu, C. K. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01625.x · Cited by 67
Learning locomotion skills using DeepRL: does the choice of action space matter?Preprint
Peng, X. B., & van de Panne, M. (2017) · arXiv (Cornell University) · Cited by 62
Self-citationLee, Y., Kim, S., & Lee, J. (2010) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1833349.1781155 · Cited by 56
How to train your dragon: example-guided control of flapping flight
Won, J., Park, J.-H., Kim, K., & Lee, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130833 · Cited by 53
Synthesis of Controllers for Stylized Planar Bipedal Walking
Sharon, D., & van de Panne, M. (2006) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2005.1570470 · Cited by 52
Learning to schedule control fragments for physics-based characters using deep q-learning
Liu, L., & Hodgins, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3083723 · Cited by 49
Generalizing locomotion style to new animals with inverse optimal regression
Wampler, K., Popović, Z., & Popović, J. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601192 · Cited by 49
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks.Preprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · Cited by 44
Self-citationDiverse motion variations for physics-based character animation
Agrawal, S., Shen, S., & van de Panne, M. (2013) · ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) · doi:10.1145/2485895.2485907 · Cited by 19
Cited by
Continuous control for robot based on deep reinforcement learning
Zhang, S. (2020) · doi:10.32657/10356/90191 · Cited by 951
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Self-citationLearning robust perceptive locomotion for quadrupedal robots in the wild
Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2022) · Science Robotics · doi:10.1126/scirobotics.abk2822 · Cited by 813
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Self-citationAMP: adversarial motion priors for stylized physics-based character control
Peng, X. B., Ma, Z., Abbeel, P., Levine, S., & Kanazawa, A. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459670 · Cited by 396
Self-citation
Metadata
- Authors
- Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M.
- Venue
- ACM Transactions on Graphics
- Published
- 2018-07-30
- Volume
- 37
- Issue
- 4
- Pages
- 1-14
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2796290181
- Cited by
- 880
- References
- 55
- FWCI
- 49.33
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.