CoChat
Sign Up
PreprintCited by 153Reinforcement Learning in RoboticsRobot Manipulation and LearningAI-based Problem Solving and Planning

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

Open access
Gupta, Abhishek, Kumar, Vikash, Lynch, Corey, Levine, Sergey, Hausman, Karol·2019·arXiv
DOI 10.48550/arxiv.1910.11956arXiv 1910.11956
Open this paper in CoChat — free accountOpen PDF

Abstract

We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learning phase that finetunes these policies for task performance. Our method, while not necessarily perfect at imitation learning, is very amenable to further improvement via environment interaction, allowing it to scale to challenging long-horizon tasks. We simplify the long-horizon policy learning problem by using a novel data-relabeling algorithm for learning goal-conditioned hierarchical policies, where the low-level only acts for a fixed number of steps, regardless of the goal achieved. While we rely on demonstration data to bootstrap policy learning, we do not assume access to demonstrations of every specific tasks that is being solved, and instead leverage unstructured and unsegmented demonstrations of semantically meaningful behaviors that are not only less burdensome to provide, but also can greatly facilitate further improvement using reinforcement learning. We demonstrate the effectiveness of our method on a number of multi-stage, long-horizon manipulation tasks in a challenging kitchen simulation environment. Videos are available at https://relay-policy-learning.github.io/

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.11956

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Franka Kitchen contains 7 objects for interaction and comes with a human demonstration dataset of 566 demonstrations, each completing 4 tasks in arbitrary order.

    “the Franka Kitchen environment contains 7 objects for interaction and comes with a human demonstration dataset of 566 demonstrations, each completing 4 tasks in arbitrary order.”

    From Diffusion policy: Visuomotor policy learning via action diffusion (2025) · cites this paper as Gupta et al. (2019)

    The Environment Setup (p. 6) describes a Franka robot in a kitchen scene and says the human-demonstration sequences each manipulate four different elements. But it reports 400 sequences, not 566, and does not establish a count of seven interactive objects or say the four elements are completed in arbitrary order. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 32 · Cited by 153

Research works printed but not linked: 27

  1. [1][1] V. Kumar, E. Todorov, and S. Levine. Optimal control with learned local models: Application to dexterous manipulation. In ICRA 2016 . -
  2. [3][3] S. Gu, E. Holly, T. P. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In ICRA 2017 . -
  3. [5][5] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NeurIPS 2016 . -
  4. [7][7] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. CoRR , abs/1802.06070, 2018. URL http://arxiv.org/abs/ 1802.06070 . - · arXiv
  5. [8][8] S. Krishnan, A. Garg, R. Liaw, B. Thananjeyan, L. Miller, F. T. Pokorny, and K. Goldberg. SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards. I. J. Robotics Res. , 38(2-3), 2019. doi:10.1177/0278364918784350. URL https://doi.org/10.1177/0278364918784350 . - · DOI
  6. [9][9] R. Fox, S. Krishnan, I. Stoica, and K. Goldberg. Multi-level discovery of deep options. CoRR , abs/1703.08294, 2017. URL http://arxiv.org/abs/1703.08294 . - · arXiv
  7. [10][10] P. Bacon, J. Harb, and D. Precup. The option-critic architecture. CoRR , abs/1609.05140, 2016. URL http://arxiv.org/abs/1609.05140 . - · arXiv
  8. [11][11] R. Parr and S. J. Russell. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems 10, [NIPS Conference, Denver, Colorado, USA, 1997] , pages 1043-1049, 1997. URL http://papers.nips.cc/paper/ 1384-reinforcement-learning-with-hierarchies-of-machines . -
  9. [12][12] T. G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. J. Artif. Intell. Res. , 13:227-303, 2000. doi:10.1613/jair.639. URL https: //doi.org/10.1613/jair.639 . - · DOI
  10. [13][13] P. Dayan and G. E. Hinton. Feudal reinforcement learning. In Advances in Neural Information Processing Systems 5, [NIPS Conference, Denver, Colorado, USA, November 30 - December 3, 1992] , pages 271-278, 1992. URL http://papers.nips.cc/paper/ 714-feudal-reinforcement-learning . -
  11. [14][14] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. CoRR , abs/1703.01161, 2017. URL http://arxiv.org/abs/1703.01161 . - · arXiv
  12. [16][16] A. Levy, R. P. Jr., and K. Saenko. Hierarchical actor-critic. CoRR , abs/1712.00948, 2017. URL http://arxiv.org/abs/1712.00948 . - · arXiv
  13. [17][17] L. P. Kaelbling. Learning to achieve goals. In Proceedings of the 13th International Joint Conference on Artificial Intelligence. Chambéry, France, August 28 - September 3, 1993 , pages 1094-1099, 1993. -
  14. [18][18] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. CoRR , abs/1707.01495, 2017. URL http://arxiv.org/abs/1707.01495 . - · arXiv
  15. [19][19] V. Pong, S. Gu, M. Dalal, and S. Levine. Temporal difference models: Model-free deep RL for model-based control. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018. URL https://openreview.net/forum?id=Skw0n-W0Z . -
  16. [20][20] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , pages 1312-1320, 2015. URL http://proceedings.mlr.press/ v37/schaul15.html . -
  17. [21][21] A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In NeurIPS 2018 , pages 9209-9220, 2018. URL http://papers.nips.cc/ paper/8132-visual-reinforcement-learning-with-imagined-goals . -
  18. [22][22] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. CoRR , abs/1903.01973, 2019. - · arXiv
  19. [23][23] K. Hausman, Y. Chebotar, S. Schaal, G. S. Sukhatme, and J. J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In NeurIPS 2017 . -
  20. [24][24] P. Henderson, W. Chang, P. Bacon, D. Meger, J. Pineau, and D. Precup. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. In AAAI 2018 . -
  21. [25][25] A. Sharma, M. Sharma, N. Rhinehart, and K. M. Kitani. Directed-info GAIL: learning hierarchical policies from unsegmented demonstrations using directed information. CoRR , abs/1810.01266, 2018. - · arXiv
  22. [26][26] T. Kipf, Y. Li, H. Dai, V. F. Zambaldi, A. Sanchez-Gonzalez, E. Grefenstette, P. Kohli, and P. Battaglia. Compile: Compositional imitation learning and execution. In ICML 2019 . -
  23. [31][31] S. Ross, G. J. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS 2011 . -
  24. [32][32] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust region policy optimization. CoRR , abs/1502.05477, 2015. URL http://arxiv.org/abs/1502.05477 . - · arXiv
  25. [33][33] D. Ghosh, A. Singh, A. Rajeswaran, V. Kumar, and S. Levine. Divide-and-conquer reinforcement learning. CoRR , abs/1711.09874, 2017. URL http://arxiv.org/abs/1711.09874 . - · arXiv
  26. [34][34] A. A. Rusu, S. G. Colmenarejo, Ç. Gülçehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. In ICLR 2016 . -
  27. [35][35] V. Kumar and E. Todorov. Mujoco haptix: A virtual reality system for hand manipulation. In Humanoid Robots (Humanoids), 2015 IEEE-RAS 15th International Conference on , pages 657-663. IEEE, 2015.

Authors
Gupta, A., Kumar, V., Lynch, C., Levine, S., & Hausman, K.
Affiliations
  • University of California, Berkeley (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.11956
arXiv
1910.11956
OpenAlex
W2981701630
Cited by
153
References
32
Citations by year
2026: 1 · 2025: 10 · 2024: 38 · 2023: 30 · 2022: 22 · 2021: 32 · 2020: 19 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
long-horizon tasks, imitation learning, reinforcement learning, demonstration-based learning, robotic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review