Learning to Walk Via Deep Reinforcement Learning
Open accessAbstract
Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions.In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics.Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters.While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning.In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies.We apply this method to learning walking gaits on a real-world Minitaur robot.Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment.We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters.Videos of training and the learned policy can be found on the project website 3 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Application of reinforcement learning to legged locomotion has largely been confined to laboratory environments and conditions.
“However, application of RL to legged locomotion has largely been confined to laboratory environments and conditions.”
From Learning quadrupedal locomotion over challenging terrain (2020) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · see citing works
Self-citationTrust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
Self-citationAddressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · see citing works
Raibert, M. H., & Tello, E. R. (1986) · IEEE Expert · doi:10.1109/mex.1986.4307016 · Cited by 2,752 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Self-citationSoft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · see citing works
Self-citationThe CMA Evolution Strategy: A Comparing Review
Hansen, N. (2007) · Studies in fuzziness and soft computing · doi:10.1007/3-540-32494-1_4 · Cited by 1,907 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Learning agile and dynamic motor skills for legged robots
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., & Hutter, M. (2019) · Science Robotics · doi:10.1126/scirobotics.aau5872 · Cited by 1,482 · see citing works
ANYmal - a highly mobile and dynamic quadrupedal robot
Hutter, M., Gehring, C., Jud, D., Lauber, A., Bellicoso, C. D., Tsounis, V., Hwangbo, J., Bodie, K., Fankhauser, P., Bloesch, M., et al. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7758092 · Cited by 969 · see citing works
Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control
Carlo, J. D., Wensing, P. M., Katz, B., Bledt, G., & Kim, S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594448 · Cited by 760 · see citing works
MIT Cheetah 3: Design and Control of a Robust, Dynamic Quadruped Robot
Bledt, G., Powell, M. J., Katz, B., Carlo, J. D., Wensing, P. M., & Kim, S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593885 · Cited by 739 · see citing works
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735 · see citing works
Self-citationSim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · arXiv · doi:10.48550/arxiv.1804.10332 · Cited by 734 · see citing works
Self-citationPolicy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 676 · see citing works
Emergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657 · see citing works
Reinforcement Learning with Deep Energy-Based PoliciesPreprint
Haarnoja, T., Tang, H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08165 · Cited by 638 · see citing works
Self-citationDesign of HyQ – a hydraulically and electrically actuated quadruped robot
Semini, C., Tsagarakis, N. G., Guglielmino, E., Focchi, M., Cannella, F., & Caldwell, D. G. (2011) · Proceedings of the Institution of Mechanical Engineers Part I Journal of Systems and Control Engineering · doi:10.1177/0959651811402275 · Cited by 630 · see citing works
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · see citing works
Policy Gradient Methods for Robotics
Peters, J., & Stefan Schaal, A. (2006) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2006.282564 · Cited by 511 · see citing works
Levine, S., & Koltun, V. (2013) · Cited by 481 · see citing works
Self-citationReward Constrained Policy OptimizationPreprint
Tessler, C., Mankowitz, D. J., & Mannor, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11074 · Cited by 399 · see citing works
Deep Reinforcement Learning that MattersPreprint
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06560 · Cited by 340 · see citing works
Robot trajectory optimization using approximate inference
Toussaint, M. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553508 · Cited by 325 · see citing works
The CMA Evolution Strategy: A Comparing Review
Hansen, N. (2006) · Springer eBooks · doi:10.1007/11007937_4 · Cited by 323 · see citing works
Design Principles for a Family of Direct-Drive Legged Robots
Kenneally, G., De, A., & Koditschek, D. E. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2528294 · Cited by 316 · see citing works
A Deep Hierarchical Approach to Lifelong Learning in Minecraft
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D. J., & Mannor, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10744 · Cited by 300 · see citing works
Bridging the Gap Between Value and Policy Based Reinforcement LearningPreprint
Nachum, O., Norouzi, M., Xu, K., & Schuurmans, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08892 · Cited by 297 · see citing works
Bayesian optimization for learning gaits under uncertainty
Calandra, R., Andre Seyfarth, A., Peters, J., & Deisenroth, M. P. (2015) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-015-9463-9 · Cited by 287 · see citing works
ANYmal - A Highly Mobile and Dynamic Quadrupedal Robot
Hutter, M., Gehring, C., Jud, D., et al. (2016) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-010686165 · Cited by 264 · see citing works
Maximum a Posteriori Policy Optimisation
Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., & Riedmiller, M. (2018) · arXiv · doi:10.48550/arxiv.1806.06920 · Cited by 259 · see citing works
General duality between optimal control and estimation
Todorov, E. (2008) · 2008 47th IEEE Conference on Decision and Control · doi:10.1109/cdc.2008.4739438 · Cited by 248 · see citing works
Terrain-adaptive locomotion skills using deep reinforcement learning
Peng, X. B., Berseth, G., & van de Panne, M. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925881 · Cited by 208 · see citing works
Feedback Control For Cassie With Deep Reinforcement Learning
Xie, Z., Berseth, G., Clary, P., HURST, J. W., & van de Panne, M. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593722 · Cited by 196 · see citing works
Composable Deep Reinforcement Learning for Robotic Manipulation
Haarnoja, T., Pong, V. H., Zhou, A., Dalal, M., Abbeel, P., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460756 · Cited by 193 · see citing works
Self-citationEquivalence Between Policy Gradients and Soft Q-LearningPreprint
Schulman, J., Chen, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06440 · Cited by 180 · see citing works
Quadrupedal locomotion using hierarchical operational space control
Hutter, M., Sommer, H., Gehring, C., Hoepflinger, M. A., Bloesch, M., & Siegwart, R. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913519834 · Cited by 149 · see citing works
A Deep Hierarchical Approach to Lifelong Learning in MinecraftPreprint
Tessler, C., Givony, S., Zahavy, T., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.07255 · Cited by 137 · see citing works
Fast Online Trajectory Optimization for the Bipedal Robot Cassie
Apgar, T., Clary, P., Green, K., Fern, A., & HURST, J. W. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.054 · Cited by 132 · see citing works
Benchmarking Reinforcement Learning Algorithms on Real-World RobotsPreprint
Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., & Bergstra, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07731 · Cited by 129 · see citing works
Latent Space Policies for Hierarchical Reinforcement LearningPreprint
Haarnoja, T., Hartikainen, K., Abbeel, P., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02808 · Cited by 128 · see citing works
Self-citationGehring, C., Hoepflinger, M. A., Siegwart, R., Coros, S., Hutter, M., Bellicoso, C. D., Heijnen, H., Diethelm, R., Bloesch, M., Fankhauser, P., et al. (2016) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2015.2505910 · Cited by 120 · see citing works
Combining policy gradient and Q-learning
O'Donoghue, B., Munos, R., Kavukcuoglu, K., & Mnih, V. (2016) · arXiv · doi:10.48550/arxiv.1611.01626 · Cited by 90 · see citing works
On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference
Rawlik, K., Toussaint, M., & Vijayakumar, S. (2012) · Robotics: Science and Systems · doi:10.15607/rss.2012.viii.045 · Cited by 90 · see citing works
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement LearningPreprint
Eysenbach, B., Gu, S., Ibarz, J., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06782 · Cited by 76 · see citing works
Self-citationBayesian Optimization Using Domain Knowledge on the ATRIAS Biped
Rai, A., Antonova, R., Song, S., et al. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461237 · Cited by 73 · see citing works
Automated Deep Reinforcement Learning Environment for Hardware of a Modular Legged Robot
Ha, S., Kim, J., & Yamane, K. (2018) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/urai.2018.8442201 · Cited by 42 · see citing works
Self-citationOn Stochastic Optimal Control and Reinforcement Learning by Approximate Inference
Rawlik, K., Toussaint, M., & Vijayakumar, S. (2013) · The MIT Press eBooks · doi:10.7551/mitpress/9816.003.0050 · Cited by 39 · see citing works
MIT Cheetah 3: Design and Control of a Robust, Dynamic Quadruped Robot
Bledt, G., Powell, M. J., Katz, B., et al. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 33 · see citing works
Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion ControlPreprint
Berseth, G., Xie, C., Cernek, P. T., & van de Panne, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04765 · Cited by 32 · see citing works
Value constrained model-free continuous controlPreprint
Bohez, S., Abdolmaleki, A., Neunert, M., Buchli, J., Heess, N., & Hadsell, R. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04623 · Cited by 29 · see citing works
Towards automatic discovery of agile gaits for quadrupedal robots
Gehring, C., Coros, S., Hutter, M., Bloesch, M., Fankhauser, P., Hoepflinger, M. A., & Siegwart, R. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907476 · Cited by 26 · see citing works
Towards Automatic Discovery of Agile Gaits for Quadrupedal Robots
Gehring, C., Coros, S., Hutter, M., et al. (2014) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.3929/ethz-a-010183016 · Cited by 21 · see citing works
Feedback Control For Cassie With Deep Reinforcement LearningPreprint
Xie, Z., Berseth, G., Clary, P., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.05580 · Cited by 21 · see citing works
Smoothed Action Value Functions for Learning Gaussian PoliciesPreprint
Nachum, O., Norouzi, M., Tucker, G., & Schuurmans, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02348 · Cited by 20 · see citing works
Self-citationMarsland, S. (2014) · Machine Learning · doi:10.1201/b17476-11 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), 2017.
- A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, V. Sindhwani, and V. Vanhoucke. Policies modulating trajectory generators. In Conference on Robot Learning (CoRL). PMLR.
- R. Tedrake, T. W. Zhang, and H. S. Seung. Learning to walk in 20 minutes. In Proceedings of the Fourteenth Yale Workshop on Adaptive and Learning Systems, volume 95585, pages 1939-1412. Yale University New Haven (CT), 2005.
Cited by
Sorted by citations
Learning quadrupedal locomotion over challenging terrain
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020) · Science Robotics · doi:10.1126/scirobotics.abc5986 · Cited by 1,156 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Self-citationLearning Agile Robotic Locomotion Skills by Imitating Animals
Bin Peng, X., Coumans, E., Zhang, T., Lee, T.-W., Tan, J., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.064 · Cited by 357 · see citing works
Self-citationNeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila
Wang-Chen, S., Stimpfling, V. A., Lam, T. K. C., Özdil, P. G., Genoud, L., Hurtak, F., & Ramdya, P. (2024) · Nature Methods · doi:10.1038/s41592-024-02497-y · Cited by 43 · see citing works
Human Motion Control of Quadrupedal Robots using Deep Reinforcement Learning
Kim, S., Sorokin, M., Lee, J., & Ha, S. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.021 · Cited by 9 · see citing works
Self-citation
- Authors
- Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S.
- Affiliations
- Google (United States) (US)
- University of California, Berkeley (US)
- Venue
- Robotics: Science and Systems XV
- Published
- 2019-06-22
- OpenAlex
- W2907537824
- Cited by
- 430
- References
- 62
- Citations by year
- 2026: 20 · 2025: 47 · 2024: 57 · 2023: 83 · 2022: 85 · 2021: 77 · 2020: 47 · 2019: 14
- FWCI
- 21.33
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Robotics: Science and Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
- Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
- Keywords
- deep reinforcement learning, maximum entropy reinforcement learning, sample efficiency, legged robots, hyperparameter robustness, model-free control, policy robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
