CoChat
Sign Up
PreprintCited by 4Robotic Locomotion and ControlReinforcement Learning in RoboticsProsthetics and Rehabilitation Robotics

One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion

Open access
Bohlinger, Nico, Czechmanowski, Grzegorz, Krupka, Maciej, Kicki, Piotr, Walas, Krzysztof, Peters, Jan, Tateo, Davide·2024·arXiv
DOI 10.48550/arxiv.2409.06366arXiv 2409.06366
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion. While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments. We introduce URMA, the Unified Robot Morphology Architecture, to close this gap. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology. The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a potential first step in building a foundation model for legged robot locomotion. Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.06366

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The chosen hyperparameters are based on common practices in legged locomotion research and preliminary tuning on a small subset of embodiments.

    “The chosen hyperparameters are based on common practices in legged locomotion research [41, 74] and preliminary tuning on a small subset of embodiments.”

    From Towards Embodiment Scaling Laws in Robot Locomotion (2025) · cites this paper as [41]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. [2][2] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning , pages 22-31. PMLR, 2023. -
  2. [4][4] K. Caluwaerts, A. Iscen, J. C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Saliceti, V. Zhuang, et al. Barkour: Benchmarking animal-level agility with quadruped robots. arXiv preprint arXiv:2305.14654 , 2023. - · arXiv
  3. [5][5] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference on Robot Learning (CoRL) , 2023. -
  4. [6][6] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. In RoboLetics: Workshop on Robot Learning in Athletics@ CoRL 2023 , 2023. -
  5. [7][7] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning , pages 91-100. PMLR, 2022. -
  6. [9][9] L. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860 , 2022. - · arXiv
  7. [10][10] L. Smith, Y. Cao, and S. Levine. Grow your limits: Continuous improvement with real-world rl for robotic locomotion. arXiv preprint arXiv:2310.17634 , 2023. - · arXiv
  8. [12][12] A. Kumar, Z. Li, J. Zeng, D. Pathak, K. Sreenath, and J. Malik. Adapting rapid motor adaptation for bipedal robots. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1161-1168. IEEE, 2022. -
  9. [13][13] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. arXiv:2303.03381 , 2023. - · arXiv
  10. [14][14] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3803-3810. IEEE, 2018. -
  11. [15][15] L. Campanaro, S. Gangapurwala, W. Merkt, and I. Havoutis. Learning and deploying robust locomotion policies with minimal dynamics randomization. arXiv preprint arXiv:2209.12878 , 2022. - · arXiv
  12. [17][17] F. Jenelten, J. He, F. Farshidian, and M. Hutter. Dtc: Deep tracking control-a unifying approach to model-based planning and reinforcement-learning for versatile and robust locomotion. arXiv preprint arXiv:2309.15462 , 2023. - · arXiv
  13. [18][18] M. Kasaei, M. Abreu, N. Lau, A. Pereira, and L. P. Reis. A cpg-based agile and versatile locomotion framework using proximal symmetry loss. arXiv preprint arXiv:2103.00928 , 2021. - · arXiv
  14. [19][19] J. Lee, L. Schroth, V. Klemm, M. Bjelonic, A. Reske, and M. Hutter. Evaluation of constrained reinforcement learning algorithms for legged locomotion. arXiv preprint arXiv:2309.15430 , 2023. - · arXiv
  15. [20][20] Y. Kim, H. Oh, J. Lee, J. Choi, G. Ji, M. Jung, D. Youm, and J. Hwangbo. Not only rewards but also constraints: Applications on legged robot locomotion. IEEE Transactions on Robotics , 2024. -
  16. [21][21] T. He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi. Agile but safe: Learning collision-free high-speed legged locomotion. arXiv preprint arXiv:2401.17583 , 2024. - · arXiv
  17. [22][22] Y. Ouyang, J. Li, Y. Li, Z. Li, C. Yu, K. Sreenath, and Y. Wu. Long-horizon locomotion and manipulation on a quadrupedal robot with large language models. arXiv preprint arXiv:2404.05291 , 2024. - · arXiv
  18. [23][23] W. Song, H. Zhao, P. Ding, C. Cui, S. Lyu, Y. Fan, and D. Wang. Germ: A generalist robotic model with mixture-of-experts for quadruped robot. arXiv preprint arXiv:2403.13358 , 2024. - · arXiv
  19. [24][24] M. Shafiee, G. Bellegarda, and A. Ijspeert. Manyquadrupeds: Learning a single locomotion policy for diverse quadruped robots. arXiv preprint arXiv:2310.10486 , 2023. - · arXiv
  20. [25][25] B. Trabucco, M. Phielipp, and G. Berseth. Anymorph: Learning transferable polices by inferring agent morphology. In International Conference on Machine Learning , pages 2167721691. PMLR, 2022. -
  21. [26][26] C. D'Eramo, D. Tateo, A. Bonarini, M. Restelli, J. Peters, et al. Sharing knowledge in multi-task deep reinforcement learning. In 8th International Conference on Learning Representations, { ICLR } 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , pages 1-11. International Conference on Learning Representations, ICLR, 2020. -
  22. [27][27] A. Hendawy, J. Peters, and C. D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In The Twelfth International Conference on Learning Representations , 2023. -
  23. [28][28] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR, 2020. -
  24. [29][29] C. L. Nehaniv and K. Dautenhahn. The correspondence problem. In Imitation in animals and artifacts , pages 41-61. MIT Press, 2002. -
  25. [30][30] T. Wang, R. Liao, J. Ba, and S. Fidler. Nervenet: Learning structured policy with graph neural networks. In International conference on learning representations , 2018. -
  26. [31][31] W. Huang, I. Mordatch, and D. Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning , pages 4455-4464. PMLR, 2020. -
  27. [32][32] J. Whitman, M. Travers, and H. Choset. Learning modular robot control policies. IEEE Transactions on Robotics , 2023. -
  28. [33][33] V. Kurin, M. Igl, T. Rockt¨ aschel, J. B¨ ohmer, and S. Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations (ICLR) , 2021. -
  29. [34][34] A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: learning universal controllers with transformers. In International Conference on Learning Representations . ICLR, 2022. -
  30. [35][35] B. Li, H. Li, Y. Zhu, and D. Zhao. Mat: Morphological adaptive transformer for universal morphology policy learning. IEEE Transactions on Cognitive and Developmental Systems , 2024. -
  31. [36][36] G. Feng, H. Zhang, Z. Li, X. B. Peng, B. Basireddy, L. Yue, Z. Song, L. Yang, Y. Liu, K. Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference on Robot Learning , pages 1893-1903. PMLR, 2023. -
  32. [37][37] Z. Luo, Y. Dong, X. Li, R. Huang, Z. Shu, E. Xiao, and P. Lu. Moral: Learning morphologically adaptive locomotion controller for quadrupedal robots on challenging terrains. IEEE Robotics and Automation Letters , 2024. -
  33. [38][38] D. Bahdanau, K. H. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015 , 2015. -
  34. [39][39] J. Lin, X. Sun, X. Ren, M. Li, and Q. Su. Learning when to concentrate or divert attention: Self-adaptive attention temperature for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2985-2990. Association for Computational Linguistics, Oct.-Nov. 2018. -
  35. [40][40] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. - · arXiv
  36. [41][41] A. Maurer, M. Pontil, and B. Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research , 17(81):1-32, 2016. -
  37. [42][42] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002. -
  38. [43][43] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012. -
  39. [44][44] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. - · arXiv
  40. [45][45] N. Bohlinger and K. Dorer. Rl-x: A deep reinforcement learning library (not only) for robocup. In Robot World Cup , pages 228-239. Springer, 2023. -
  41. [46][46] G. Ji, J. Mun, H. Kim, and J. Hwangbo. Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters , 7(2):4630-4637, 2022.

Cited by

Sorted by citations

  • Towards Embodiment Scaling Laws in Robot Locomotion

    Ai, B., Dai, L., Bohlinger, N., Li, D., Mu, T., Wu, Z., Fay, K., Christensen, H. I., Peters, J., & Su, H. (2025) · arXiv · doi:10.48550/arxiv.2505.05753

    “To train a policy that can control ∼ 1000 different embodi-ments with different state and action spaces, we use URMA, an embodiment-aware architecture for robots with arbitrary numbers of joints [8].”
    InfluentialSelf-citation

Authors
Bohlinger, N., Czechmanowski, G., Krupka, M., Kicki, P., Walas, K., Peters, J., & Tateo, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2409.06366
arXiv
2409.06366
OpenAlex
W4403885084
Cited by
4
References
0
Citations by year
2025: 3 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
Keywords
legged robot locomotion, multi-task reinforcement learning, end-to-end learning, transfer learning, foundation models, quadruped robot, humanoid robots, hexapod robot

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review