CoChat
Sign Up
PreprintCited by 33Reinforcement Learning in RoboticsRobot Manipulation and LearningAI-based Problem Solving and Planning

ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

Open access
Xia, Fei, Li, Chengshu, Martín-Martín, Roberto, Litany, Or, Toshev, Alexander, Savarese, Silvio·2020·arXiv
DOI 10.48550/arxiv.2008.07792arXiv 2008.07792
Open this paper in CoChat — free accountOpen PDF

Abstract

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous control tasks. We propose to lift the action space to a higher level in the form of subgoals for a motion generator (a combination of motion planner and trajectory executor). We argue that, by lifting the action space and by leveraging sampling-based motion planners, we can efficiently use RL to solve complex, long-horizon tasks that could not be solved with existing RL methods in the original action space. We propose ReLMoGen -- a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base. These problems are challenging because they are usually long-horizon, hard to explore during training, and comprise alternating phases of navigation and interaction. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. In all settings, ReLMoGen outperforms state-of-the-art Reinforcement Learning and Hierarchical Reinforcement Learning baselines. ReLMoGen also shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2008.07792

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some form of action-space abstraction must be necessary to solve long-horizon activities of BEHAVIOR-1K.

    “This supports the idea that some form of action-space abstraction must be necessary to solve long-horizon activities of BEHA VIOR-1K, as others reported [27, 26, 69].”

    From BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation (2024) · cites this paper as [69]

    The Introduction (p. 1) says mobile-manipulation tasks require long sequences of low-level actions that challenge reinforcement learning, and the Analysis (p. 5) reports that ReLMoGen’s subgoal-based method outperformed joint-velocity baselines on harder tasks. This supports the value of action-space abstraction, but the paper does not establish that it is necessary or report results for BEHAVIOR-1K activities. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 57 · Cited by 33

Research works printed but not linked: 57

  1. [1]S. M. LaValle, Planning algorithms . Cambridge university press, 2006.
  2. [2]B. Siciliano and O. Khatib, Springer Handbook of Robotics . Berlin, Heidelberg: Springer-Verlag, 2007.
  3. [3]R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018.
  4. [4]K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, 'Deep reinforcement learning: A brief survey,' IEEE Signal Processing Magazine , vol. 34, no. 6, pp. 26-38, 2017.
  5. [5]Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, 'Target-driven visual navigation in indoor scenes using deep reinforcement learning,' in 2017 IEEE international conference on robotics and automation (ICRA) . IEEE, 2017, pp. 3357-3364.
  6. [6]H. Quan, Y. Li, and Y. Zhang, 'A novel mobile robot navigation method based on deep reinforcement learning,' International Journal of Advanced Robotic Systems , vol. 17, no. 3, p. 1729881420921672, 2020.
  7. [7]A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, 'Tossingbot: Learning to throw arbitrary objects with residual physics,' IEEE Transactions on Robotics , 2020.
  8. [8]D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke et al. , 'Scalable deep reinforcement learning for vision-based robotic manipulation,' in Conference on Robot Learning , 2018, pp. 651-673.
  9. [9]J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, 'Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,' arXiv preprint arXiv:1703.09312 , 2017. · arXiv
  10. [10]S. Levine, C. Finn, T. Darrell, and P. Abbeel, 'End-toend training of deep visuomotor policies,' The Journal of Machine Learning Research , vol. 17, no. 1, pp. 1334-1373, 2016.
  11. [11]A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser, 'Learning synergies between pushing and grasping with self-supervised deep reinforcement learning,' in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2018, pp. 4238-4245.
  12. [12]I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, 'Deep exploration via bootstrapped dqn,' in Advances in neural information processing systems , 2016, pp. 4026-4034.
  13. [13]I. Osband, B. Van Roy, D. J. Russo, and Z. Wen, 'Deep exploration via randomized value functions.' Journal of Machine Learning Research , vol. 20, no. 124, pp. 1-62, 2019.
  14. [14]A. Levy, G. Konidaris, R. Platt, and K. Saenko, 'Learning multi-level hierarchies with hindsight,' International Conference on Learning Representations , 2019.
  15. [15]O. Nachum, S. S. Gu, H. Lee, and S. Levine, 'Dataefficient hierarchical reinforcement learning,' in Advances in Neural Information Processing Systems , 2018, pp. 3303-3313.
  16. [16]O. Nachum, H. Tang, X. Lu, S. Gu, H. Lee, and S. Levine, 'Why does hierarchy (sometimes) work so well in reinforcement learning?' arXiv preprint arXiv:1909.10618 , 2019. · arXiv
  17. [17]C. Li, F. Xia, R. Mart´ ın-Mart´ ın, and S. Savarese, 'Hrl4in: Hierarchical reinforcement learning for interactive navigation with mobile manipulators,' in Conference on Robot Learning , 2020, pp. 603-616.
  18. [18]V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, 'Playing atari with deep reinforcement learning,' arXiv preprint arXiv:1312.5602 , 2013. · arXiv
  19. [20]J. J. Kuffner and S. M. LaValle, 'Rrt-connect: An efficient approach to single-query path planning,' in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065) , vol. 2. IEEE, 2000, pp. 995-1001.
  20. [21]R. Bohlin and L. E. Kavraki, 'Path planning using lazy prm,' in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065) , vol. 1. IEEE, 2000, pp. 521-528.
  21. [22]M. M¨ uller, A. Dosovitskiy, B. Ghanem, and V. Koltun, 'Driving policy transfer via modularity and abstraction,' arXiv preprint arXiv:1804.09364 , 2018. · arXiv
  22. [23]E. Kaufmann, M. Gehrig, P. Foehn, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, 'Beauty and the beast: Optimal methods meet learning for drone racing,' in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 690-696.
  23. [24]T. Jurgenson and A. Tamar, 'Harnessing reinforcement learning for neural motion planning,' in Proceedings of Robotics: Science and Systems , Freiburg im Breisgau, Germany, June 2019.
  24. [25]A. H. Qureshi, A. Simeonov, M. J. Bency, and M. C. Yip, 'Motion planning networks,' in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 2118-2124.
  25. [26]S. Bansal, V. Tolani, S. Gupta, J. Malik, and C. Tomlin, 'Combining optimal control and learning for visual navigation in novel environments,' in Conference on Robot Learning (CoRL) , 2019.
  26. [27]S. Levine and V. Koltun, 'Guided policy search,' in International Conference on Machine Learning , 2013, pp. 1-9.
  27. [28]N. Jetchev and M. Toussaint, 'Trajectory prediction in cluttered voxel environments,' in 2010 IEEE International Conference on Robotics and Automation . IEEE, 2010, pp. 2523-2528.
  28. [30]K. Ota, Y. Sasaki, D. K. Jha, Y. Yoshiyasu, and A. Kanezaki, 'Efficient exploration in constrained environments with goal-oriented reference path,' arXiv preprint arXiv:2003.01641 , 2020. · arXiv
  29. [31]Y. Jiang, F. Yang, S. Zhang, and P. Stone, 'Integrating task-motion planning with reinforcement learning for robust decision making in mobile robots,' in In Proceedings of the AAMAS , 2019.
  30. [32]A. Dragan, G. J. Gordon, and S. Srinivasa, 'Learning from experience in manipulation planning: Setting the right goals,' in In Proceedings of the ISRR , 2011.
  31. [33]J. Yamada, G. Salhotra, Y. Lee, M. Pflueger, K. Pertsch, P. Englert, G. S. Sukhatme, and J. J. Lim, 'Motion planner augmented action spaces for reinforcement learning,' RSS Workshop on Action Representations for Learning in Continuous Control , 2020.
  32. [34]J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser, 'Spatial Action Maps for Mobile Manipulation,' in Proceedings of Robotics: Science and Systems , Corvalis, Oregon, USA, July 2020.
  33. [35]N. Heess, G. Wayne, Y. Tassa, T. Lillicrap, M. Riedmiller, and D. Silver, 'Learning and transfer of modulated locomotor controllers,' arXiv preprint arXiv:1610.05182 , 2016. · arXiv
  34. [36]T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum, 'Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation,' in Advances in neural information processing systems , 2016, pp. 3675-3683.
  35. [37]A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu, 'Feudal networks for hierarchical reinforcement learning,' in Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 3540-3549.
  36. [38]O. Nachum, S. Gu, H. Lee, and S. Levine, 'Nearoptimal representation learning for hierarchical reinforcement learning,' International Conference on Learning Representations , 2018.
  37. [39]G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto, 'Autonomous skill acquisition on a mobile manipulator,' in Twenty-Fifth AAAI Conference on Artificial Intelligence , 2011.
  38. [40]I. Osband, J. Aslanides, and A. Cassirer, 'Randomized prior functions for deep reinforcement learning,' in Advances in Neural Information Processing Systems , 2018, pp. 8617-8629.
  39. [41]K. Ciosek, Q. Vuong, R. Loftin, and K. Hofmann, 'Better exploration with optimistic actor critic,' in Advances in Neural Information Processing Systems , 2019, pp. 1787-1798.
  40. [42]P. Anderson et al. , 'On evaluation of embodied navigation agents,' arXiv preprint arXiv:1807.06757 , 2018. · arXiv
  41. [43]Manolis Savva, Abhishek Kadian, Oleksandr Maksymets et al. , 'Habitat: A Platform for Embodied AI Research,' in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.
  42. [44]I. Zamora, N. G. Lopez, V. M. Vilches, and A. H. Cordero, 'Extending the openai gym for robotics: a toolkit for reinforcement learning using ros and gazebo,' arXiv preprint arXiv:1608.05742 , 2016. · arXiv
  43. [45]F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. Mart´ ın-Mart´ ın, and S. Savarese, 'Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments,' IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 713-720, April 2020.
  44. [46]D. Berenson, J. Kuffner, and H. Choset, 'An optimization approach to planning for mobile manipulation,' in 2008 IEEE International Conference on Robotics and Automation . IEEE, 2008, pp. 1187-1192.
  45. [47]E. Klingbeil, A. Saxena, and A. Y. Ng, 'Learning to open new doors,' in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2010, pp. 2751-2757.
  46. [48]C. Wang, Q. Zhang, Q. Tian, S. Li, X. Wang, D. Lane, Y. Petillot, and S. Wang, 'Learning mobile manipulation through deep reinforcement learning,' Sensors , vol. 20, no. 3, p. 939, 2020.
  47. [50]A. X. Chang et al. , 'Shapenet: An information-rich 3d model repository,' arXiv preprint arXiv:1512.03012 , 2015. · arXiv
  48. [51]D. Hernandez, 'How to survive a robot apocalypse: Just close the door,' The Wall Street Journal , p. 10, 2017.
  49. [53]A. Stooke and P. Abbeel, 'rlpyt: A research code base for deep reinforcement learning in pytorch,' arXiv preprint arXiv:1909.01500 , 2019. · arXiv
  50. [55]K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige et al. , 'Using simulation and domain adaptation to improve efficiency of deep robotic grasping,' in 2018 IEEE international conference on robotics and automation (ICRA) . IEEE, 2018, pp. 4243-4250.
  51. [56]K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, 'Rl-cyclegan: Reinforcement learning aware simulation-to-real,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 11 157-11 166.
  52. [57]F. Ramos, R. C. Possas, and D. Fox, 'Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators,' arXiv preprint arXiv:1906.01728 , 2019. · arXiv
  53. [58]Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, 'Closing the sim-toreal loop: Adapting simulation randomization with real world experience,' in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 8973-8979.
  54. [59]K. Kang, S. Belkhale, G. Kahn, P. Abbeel, and S. Levine, 'Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight,' International Conference on Robotics and Automation (ICRA) , 2019.
  55. [60]X. Meng, N. Ratliff, Y. Xiang, and D. Fox, 'Neural autonomous navigation with riemannian motion policy,' in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 8860-8866.
  56. [61]F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, 'Gibson env: Real-world perception for embodied agents,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 9068-9079.
  57. [62]J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, 'Sim-to-real: Learning agile locomotion for quadruped robots,' arXiv preprint arXiv:1804.10332 , 2018. · arXiv

Authors
Xia, F., Li, C., Martín-Martín, R., Litany, O., Toshev, A., & Savarese, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2008.07792
arXiv
2008.07792
OpenAlex
W3069663381
Cited by
33
References
57
Citations by year
2025: 1 · 2024: 2 · 2023: 9 · 2022: 5 · 2021: 16
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
motion generation, mobile manipulation, hierarchical reinforcement learning, sampling-based motion planning, interactive navigation, long-horizon tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review