CoChat
Sign Up
PreprintCited by 53Reinforcement Learning in RoboticsRobotic Locomotion and ControlProsthetics and Rehabilitation Robotics

Learning to Walk via Deep Reinforcement Learning

Open access
Haarnoja, Tuomas, Ha, Sehoon, Zhou, Aurick, Tan, Jie, Tucker, George, Levine, Sergey·2018·arXiv
DOI 10.48550/arxiv.1812.11103arXiv 1812.11103
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1812.11103

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The modularity of the framework makes it possible for users to implement additional signal processing by downstream olfactory centers (e.g., lateral horn or mushroom body).

    “The modularity of our framework makes it possible for users to implement additional signal processing by downstream olfactory centers (e.g., lateral horn or mushroom body 71,72 ).”

    From NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila (2024) · cites this paper as [71]

    The Asynchronous Learning System section (pp. 2–3) describes separate subsystems for robotic reinforcement learning, not a framework for modeling olfactory centers. Searches for lateral horn, mushroom body, and downstream olfactory signal processing found no passage in our copy addressing the attributed claim. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. [2][2] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920 , 2018. - · arXiv
  2. [3][3] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML) , 2017. -
  3. [4][4] G. Berseth, C. Xie, P. Cernek, and M. Van de Panne. Progressive reinforcement learning with distillation for multi-skilled motion control. International Conference on Learning Representations (ICLR) , 2018. -
  4. [5][5] G. Bledt, M. J. Powell, B. Katz, J. Di Carlo, P. M. Wensing, and S. Kim. MIT cheetah 3: Design and control of a robust, dynamic quadruped robot. In International Conference on Intelligent Robots and Systems (IROS) , 2018. -
  5. [6][6] S. Bohez, A. Abdolmaleki, M. Neunert, J. Buchli, N. Heess, and R. Hadsell. Value constrained modelfree continuous control. arXiv preprint arXiv:1902.04623 , 2019. - · arXiv
  6. [7][7] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540 , 2016. - · arXiv
  7. [8][8] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence , 76(1-2): 5-23, 2016. -
  8. [9][9] J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim. Dynamic locomotion in the mit cheetah 3 through convex model-predictive control. In International Conference on Intelligent Robots and Systems (IROS) , 2018. -
  9. [10][10] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations (ICLR) , 2018. -
  10. [11][11] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML) , 2018. -
  11. [12][12] C. Gehring, S. Coros, M. Hutter, M. Bloesch, P. Fankhauser, M. A. Hoepflinger, and R. Siegwart. Towards automatic discovery of agile gaits for quadrupedal robots. In International Conference on Robotics and Automation (ICRA) , 2014. -
  12. [13][13] C. Gehring, M. Coros, S. Hutler, C. D. Bellicoso, H. Heijnen, R. Diethelm, M. Bloesch, P. Fankhauser, J. Hwangbo, M. Hoepflinger, et al. Practice makes perfect: An optimization-based approach to controlling agile motions for a quadruped robot. IEEE Robotics & Automation Magazine , 23(1):34-43, 2016. -
  13. [14][14] S. Ha, J. Kim, and K. Yamane. Automated deep reinforcement learning environment for hardware of a modular legged robot. In International Conference on Ubiquitous Robots (UR) . IEEE, 2018. -
  14. [15][15] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (ICML) , 2017. -
  15. [16][16] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine. Composable deep reinforcement learning for robotic manipulation. In International Conference on Robotics and Automation (ICRA) . IEEE, 2018. -
  16. [17][17] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML) , 2018. -
  17. [18][18] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018. - · arXiv
  18. [19][19] N. Hansen. The CMA evolution strategy: a comparing review. In Towards a new evolutionary computation , pages 75-102. Springer, 2006. -
  19. [20][20] N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, A. Eslami, M. Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 , 2017. - · arXiv
  20. [21][21] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In Conference on Artificial Intelligence (AAAI) . AAAI, 2018. -
  21. [22][22] M. Hutter, H. Sommer, C. Gehring, M. Hoepflinger, M. Bloesch, and R. Siegwart. Quadrupedal locomotion using hierarchical operational space control. The International Journal of Robotics Research , 33(8):1047-1062, 2014. -
  22. [23][23] M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch, et al. Anymal-a highly mobile and dynamic quadrupedal robot. In International Conference on Intelligent Robots and Systems (IROS) , pages 38-44. IEEE, 2016. -
  23. [24][24] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics , 4(26), 2019. doi: 10.1126/scirobotics.aau5872. - · DOI · Paper page
  24. [25][25] A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, V. Sindhwani, and V. Vanhoucke. Policies modulating trajectory generators. In Conference on Robot Learning (CoRL) . PMLR. -
  25. [26][26] Gavin D Kenneally, Avik De, and Daniel E Koditschek. Design principles for a family of direct-drive legged robots. IEEE Robotics and Automation Letters , 1(2): 900-907, 2016. -
  26. [27][27] D. Kingma and J. Ba. Adam: A method for stochastic 28. optimization. In International Conference for Learning Presentations (ICLR) , 2015. -
  27. [28][28] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. - · arXiv
  28. [29][29] N. Kohl and P. Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In International Conference on Robotics and Automation (ICRA) . IEEE, 2004. -
  29. [30][30] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015. - · arXiv
  30. [31][31] A. R. Mahmood, D. Korenkevych, G. Vasan, W. Ma, and J. Bergstra. Benchmarking reinforcement learning algorithms on real-world robots. In Conference on Robot Learning (CoRL) , 2018. -
  31. [32][32] O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2017. -
  32. [33][33] Brendan O'Donoghue, Rémi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Pgq: Combining policy gradient and q. arXiv preprint arXiv:1611.01626 , 2016. - · arXiv
  33. [34][34] X. B. Peng, G. Berseth, and M. Van de Panne. Terrainadaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (TOG) , 35 (4):81, 2016. -
  34. [35][35] J. Peters and S. Schaal. Policy gradient methods for robotics. In International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2006. -
  35. [36][36] A. Rai, R. Antonova, S. Song, W. Martin, H. Geyer, and C. G. Atkeson. Bayesian optimization using domain knowledge on the ATRIAS biped. arXiv preprint arXiv:1709.06047 , 2017. - · arXiv
  36. [37][37] M. H. Raibert. Legged robots that balance . MIT press, 1986. -
  37. [39][39] J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML) , pages 1889-1897, 2015. -
  38. [40][40] J. Schulman, P. Abbeel, and X. Chen. Equivalence between policy gradients and soft Q-learning. arXiv preprint arXiv:1704.06440 , 2017. - · arXiv
  39. [41][41] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. - · arXiv
  40. [42][42] C. Semini, N. G Tsagarakis, E. Guglielmino, M. Focchi, F. Cannella, and D. G. Caldwell. Design of hyq-a hydraulically and electrically actuated quadruped robot. Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering , 225(6):831-849, 2011. -
  41. [43][43] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998. -
  42. [45][45] R. Tedrake, T. W. Zhang, and H. S. Seung. Learning to walk in 20 minutes. In Proceedings of the Fourteenth Yale Workshop on Adaptive and Learning Systems , volume 95585, pages 1939-1412. Yale University New Haven (CT), 2005. -
  43. [46][46] C. Tessler, S. Givony, T. Zahavy, D. J. Mankowitz, and S. Mannor. A deep hierarchical approach to lifelong learning in minecraft. In AAAI Conference on Artificial Intelligence (AAAI) , volume 3, page 6, 2017. -
  44. [47][47] C. Tessler, D. J. Mankowitz, and S. Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 , 2018. - · arXiv
  45. [48][48] E. Todorov. General duality between optimal control and estimation. In Conference on Decision and Control (CDC) , pages 4286-4292. IEEE, 2008. -
  46. [49][49] M. Toussaint. Robot trajectory optimization using approximate inference. In International Conference on Machine Learning (ICML) , pages 1049-1056. ACM, 2009. -
  47. [50][50] Z. Xie, G. Berseth, P. Clary, J. Hurst, and M. van de Panne. Feedback control for cassie with deep reinforcement learning. arXiv preprint arXiv:1803.05580 , 2018. - · arXiv
  48. [51][51] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI) , pages 1433-1438, 2008.

Cited by

Sorted by citations

Authors
Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S.
Affiliations
  • Google (United States) (US)
  • University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1812.11103
arXiv
1812.11103
OpenAlex
W2963339188
Cited by
53
References
0
Citations by year
2025: 2 · 2024: 3 · 2023: 8 · 2022: 4 · 2021: 14 · 2020: 15 · 2019: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
  • Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
Keywords
deep reinforcement learning, maximum entropy reinforcement learning, sample efficiency, hyperparameter sensitivity, legged robots, model-free control, robustness to environmental variation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review