CoChat
Sign Up
PreprintCited by 296Reinforcement Learning in RoboticsRobotic Locomotion and ControlProsthetics and Rehabilitation Robotics

The DOI matches its DataCite record.

Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning

Open access
Rudin, Nikita, Hoeller, David, Reist, Philipp, Hutter, Marco·2021·arXiv
DOI 10.48550/arxiv.2109.11978arXiv 2109.11978
Open this paper in CoChat — free account

Abstract

In this work, we present and study a training set-up that achieves fast policy generation for real-world robotic tasks by using massive parallelism on a single workstation GPU. We analyze and discuss the impact of different training algorithm components in the massively parallel regime on the final policy performance and training times. In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel. We evaluate the approach by training the quadrupedal robot ANYmal to walk on challenging terrain. The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain. This represents a speedup of multiple orders of magnitude compared to previous work. Finally, we transfer the policies to the real robot to validate the approach. We open-source our training code to help accelerate further research in the field of learned legged locomotion.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2109.11978

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The chosen hyperparameters are based on common practices in legged locomotion research and preliminary tuning on a small subset of embodiments.

    “The chosen hyperparameters are based on common practices in legged locomotion research [41, 74] and preliminary tuning on a small subset of embodiments.”

    From Towards Embodiment Scaling Laws in Robot Locomotion (2025) · cites this paper as [74]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 22

  1. [1][1] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
  2. [2][2] S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manip- ulation with asynchronous off-policy updates. In IEEE International Conference on Robotics and Automation (ICRA), May 2017.
  3. [3][3] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
  4. [5][5] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
  5. [7][7] J. Hwangbo, J. Lee, and M. Hutter. Per-contact iteration method for solving contact dynamics. IEEE Robotics and Automation Letters, 3(2), 2018. URL www.raisim.com.
  6. [8][8] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance GPU based physics simu- lation for robot learning. In Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
  7. [9][9] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. A. Riedmiller, and D. Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.02286, 2017. · arXiv
  8. [10][10] A. Stooke and P. Abbeel. Accelerated methods for deep reinforcement learning. CoRR, abs/1803.02811, 2018. · arXiv
  9. [11][11] B. Shacklett, E. Wijmans, A. Petrenko, M. Savva, D. Batra, V. Koltun, and K. Fatahalian. Large batch simulation for deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
  10. [12][12] J. Liang, V. Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox. Gpu-accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning (CoRL), 2018.
  11. [13][13] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax - a differ- entiable physics engine for large scale rigid body simulation. In 35th Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
  12. [14][14] A. Bouman, M. F. Ginting, N. Alatur, M. Palieri, D. D. Fan, T. Touma, T. Pailevanian, S.- K. Kim, K. Otsu, J. Burdick, and A.-a. Agha-Mohammadi. Autonomous spot: Long-range autonomous exploration of extreme environments with legged locomotion. In IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), 2020.
  13. [16][16] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science Robotics, 5(47), 2020.
  14. [17][17] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters, PP, 03 2020.
  15. [18][18] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning. In IEEE Interna- tional Conference on Robotics and Automation (ICRA), 2021.
  16. [20][20] J. Siekmann, K. Green, J. Warila, A. Fern, and J. W. Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning. CoRR, abs/2105.08328, 2021. · arXiv
  17. [21][21] C. Gregg and K. Hazelwood. Where is the data? why you cannot debate cpu vs. gpu per- formance without the answer. In IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2011.
  18. [22][22] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. · arXiv
  19. [23][23] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Confer- ence on Learning Representations (ICLR), 2016.
  20. [24][24] F. Pardo, A. Tavakoli, V. Levdik, and P. Kormushev. Time limits in reinforcement learning. CoRR, abs/1712.00378, 2017. · arXiv
  21. [28][28] R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (POET): end- lessly generating increasingly complex and diverse learning environments and their solutions. CoRR, abs/1901.01753, 2019. · arXiv
  22. [29][29] Z. Xie, H. Y. Ling, N. H. Kim, and M. van de Panne. Allsteps: Curriculum-driven learn- ing of stepping stone skills. Proceedings of ACM SIGGRAPH / Eurographics Symposium on Computer Animation, 2020.

Cited by

Sorted by citations

Metadata

Authors
Rudin, N., Hoeller, D., Reist, P., & Hutter, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
DOI
10.48550/arxiv.2109.11978
arXiv
2109.11978
OpenAlex
W3200962355
Cited by
296
References
0
Citations by year
2026: 1 · 2025: 43 · 2024: 139 · 2023: 77 · 2022: 36
FWCI
29.11
Type
conference paper
Language
en
Open access
Licence
Locations
  • Repository for Publications and Research Data (ETH Zurich) (acceptedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • ETH Zürich Research Collection
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robotic Locomotion and Control
  • Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
Keywords
quadrupedal walking, ANYmal, deep reinforcement learning, GPU parallelization, curriculum learning, simulated robots, rough terrain locomotion, training acceleration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review