CoChat
Sign Up
PreprintCited by 503Reinforcement Learning in RoboticsAdvanced Bandit Algorithms ResearchDomain Adaptation and Few-Shot Learning

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Open access
Duan, Yan, Schulman, John, Chen, Xi, Bartlett, Peter L., Sutskever, Ilya, Abbeel, Pieter·2016·arXiv
DOI 10.48550/arxiv.1611.02779arXiv 1611.02779
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL$^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL$^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL$^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL$^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1611.02779

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The same ideas behind meta-learning can be readily extended to the reinforcement learning setting.

    “While our exposition in this section focused on the supervised learning case, the same ideas can also be readily extended to the reinforcement learning setting (Duan et al., 2016; Wang et al., 2016).”

    From Meta-learned models of cognition (2024) · cites this paper as Duan et al., 2016

    The Related Work section (p. 9) says the paper draws on supervised meta-learning ideas and applies them to reinforcement learning. It also says the reinforcement learning setting is more challenging, so the claim that the extension is ‘readily’ made overstates what the source says. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • It is straightforward to meta-learn an approximation to the Bayes-optimal policy.

    “Even though finding an analytical expression of the Bayes-optimal policy is often impossible, it is straightforward to meta-learn an approximation to it (Duan et al., 2016; Wang et al., 2016).”

    From Meta-learned models of cognition (2024) · cites this paper as Duan et al., 2016

    The Introduction (p. 2) describes training a recurrent policy to maximize performance across MDPs drawn from a prior, and Policy Optimization (p. 3) says standard RL algorithms can readily optimize that policy. The paper reports performance comparable to algorithms with optimality guarantees, but does not establish that the learned policy approximates the Bayes-optimal policy or that doing so is generally straightforward. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Meta-RL approaches pre-train adaptive agents on diverse task distributions to significantly increase sample efficiency on new problems.

    “One way to address these shortcomings are meta-RL approaches, where adaptive agents are pre-trained on diverse task distributions to significantly increase sample efficiency on new problems”

    From XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (2024) · cites this paper as Duan et al., 2016

  • The implemented baselines are near-single-file implementations of recurrent PPO for single-task environments and its extension to RL^2 for meta-learning.

    “we are providing near-single-file implementations of recurrent PPO (Schulman et al., 2017) for single-task environments and its extension to RL 2 (Duan et al., 2016; Wang et al., 2016) for meta-learning as baselines.”

    From XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (2024) · cites this paper as Duan et al., 2016

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. arXiv preprint arXiv:1606.04474 , 2016. · arXiv
  2. Jean-Yves Audibert and R´ emi Munos. Introduction to bandits: Algorithms and theory. ICML Tutorial on bandits , 2011.
  3. Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.
  4. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks , 5(2):157-166, 1994.
  5. S´ ebastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multiarmed bandit problems. arXiv preprint arXiv:1204.5721 , 2012. · arXiv
  6. Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems , pp. 2249-2257, 2011.
  7. Kyunghyun Cho, Bart Van Merri¨ enboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 , 2014. · arXiv
  8. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014. · arXiv
  9. Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11) , pp. 465-472, 2011.
  10. Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. arXiv preprint arXiv:1609.07088 , 2016. · arXiv
  11. Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence , 28(4):594-611, 2006.
  12. AA Feldbaum. Dual control theory. i. Avtomatika i Telemekhanika , 21(9):1240-1249, 1960.
  13. Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. arXiv preprint arXiv:1509.06841 , 2015. · arXiv
  14. Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: a survey . World Scientific, 2015.
  15. John Gittins, Kevin Glazebrook, and Richard Weber. Multi-armed bandit allocation indices . John Wiley & Sons, 2011.
  16. John C Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B (Methodological) , pp. 148-177, 1979.
  17. Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning. In Advances in neural information processing systems , pp. 3338-3346, 2014.
  18. Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems , pp. 2944-2952, 2015.
  19. Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks , pp. 87-94. Springer, 2001.
  20. Shin Ishii, Wako Yoshida, and Junichiro Yoshimoto. Control of exploitation-exploration metaparameter in reinforcement learning. Neural networks , 15(4):665-687, 2002.
  21. Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research , 11(Apr):1563-1600, 2010.
  22. Rafal J´ ozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , pp. 2342-2350, 2015. URL http: //jmlr.org/proceedings/papers/v37/jozefowicz15.html .
  23. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Ja´ skowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. arXiv preprint arXiv:1605.02097 , 2016. · arXiv
  24. Gregory Koch. Siamese neural networks for one-shot image recognition . PhD thesis, University of Toronto, 2015.
  25. J Zico Kolter and Andrew Y Ng. Near-bayesian exploration in polynomial time. In Proceedings of the 26th Annual International Conference on Machine Learning , pp. 513-520. ACM, 2009.
  26. Brenden M Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the 33rd Annual Conference of the Cognitive Science Society , volume 172, pp. 2, 2011.
  27. Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI , volume 1, pp. 3, 2008.
  28. Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems , pp. 1071-1079, 2014.
  29. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 17(39):1-40, 2016.
  30. Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885 , 2016. · arXiv
  31. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015. · arXiv
  32. Benedict C May, Nathan Korda, Anthony Lee, and David S Leslie. Optimistic bayesian sampling in contextual-bandit problems. Journal of Machine Learning Research , 13(Jun):2069-2106, 2012.
  33. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.
  34. Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv:1605.09128 , 2016. · arXiv
  35. Sylvie CW Ong, Shao Wei Png, David Hsu, and Wee Sun Lee. Planning under uncertainty for robotic tasks with mixed observability. The International Journal of Robotics Research , 29(8): 1053-1068, 2010.
  36. Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning. arXiv preprint arXiv:1607.00215 , 2016. · arXiv
  37. Ian Osband, Dan Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems , pp. 3003-3011, 2013.
  38. Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342 , 2015. · arXiv
  39. Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295 , 2015. · arXiv
  40. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016a. · arXiv
  41. Andrei A Rusu, Matej Vecerik, Thomas Roth¨ orl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610.04286 , 2016b. · arXiv
  42. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Oneshot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065 , 2016. · arXiv
  43. John Schulman, Sergey Levine, Philipp Moritz, Michael I Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477 , 2015. · arXiv
  44. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR2016) , 2016.
  45. Nicolas Schweighofer and Kenji Doya. Meta-learning in reinforcement learning. Neural Networks , 16(1):5-9, 2003.
  46. Satinder Pal Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning , 8(3-4):323-339, 1992.
  47. Malcolm Strens. A bayesian framework for reinforcement learning. In ICML , pp. 943-950, 2000.
  48. Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(Jul):1633-1685, 2009.
  49. Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence Review , 18(2):77-95, 2002.
  50. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. arXiv preprint arXiv:1606.04080 , 2016. · arXiv
  51. Niklas Wahlstr¨ om, Thomas B Sch¨ on, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251 , 2015. · arXiv
  52. Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems , pp. 2746-2754, 2015.
  53. Peter Whittle. Optimization over time . John Wiley & Sons, Inc., 1982.
  54. Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning , pp. 1015-1022. ACM, 2007.
  55. A Steven Younger, Sepp Hochreiter, and Peter R Conwell. Meta-learning with backpropagation. In Neural Networks, 2001. Proceedings. IJCNN'01. International Joint Conference on , volume 3. IEEE, 2001.

Cited by

Sorted by citations

Authors
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., & Abbeel, P.
Affiliations
  • University of California, Berkeley (US)
  • OpenAI (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1611.02779
arXiv
1611.02779
OpenAlex
W2578206533
Cited by
503
References
1
Citations by year
2026: 5 · 2025: 13 · 2024: 45 · 2023: 60 · 2022: 68 · 2021: 81 · 2020: 104 · 2019: 77 · 2018: 31 · 2017: 17 · 2016: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1619362)
  • Office of Naval Research
  • Australian Research Council Centre of Excellence for Mathematical and Statistical Frontiers
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
meta-learning, recurrent neural networks, multi-armed bandits, finite Markov decision processes, visual navigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review