CoChat
Sign Up
PreprintCited by 590Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningMachine Learning and Algorithms

Learning to reinforcement learn

Open access
Wang, Jane X, Kurth-Nelson, Zeb, Tirumala, Dhruva, Soyer, Hubert, Leibo, Joel Z, Munos, Remi, Blundell, Charles, Kumaran, Dharshan +1 more·2016·arXiv
DOI 10.48550/arxiv.1611.05763arXiv 1611.05763
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1611.05763

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • It is straightforward to meta-learn an approximation to the Bayes-optimal policy.

    “Even though finding an analytical expression of the Bayes-optimal policy is often impossible, it is straightforward to meta-learn an approximation to it (Duan et al., 2016; Wang et al., 2016).”

    From Meta-learned models of cognition (2024) · cites this paper as Wang et al., 2016

    Section 3.1 (p. 5) says Bayesian-optimal strategies are generally intractable to compute and shows that meta-RL can learn a prior-dependent bandit algorithm that performs well. It does not establish that approximating the Bayes-optimal policy is straightforward in general; in the independent-arm experiment (Section 3.1.1, p. 5), the learned policy performs less well than a Bayesian-optimal comparator. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Curriculum learning, incremental environment complexity, and continual learning and policy distillation for multiple tasks are methods of particular interest for achieving successful sim-to-real transfer in navigation.

    “Of particular interest due to their potential and novelty are the following methods: curriculum learning [37], incremental environment complexity [39], and continual learning and policy distillation for multiple tasks [12].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [37]

    The Formalism section (p. 3) describes training an agent on a sequence of tasks, and One-Shot Navigation (pp. 12–13) discusses adaptation in a navigation task. The paper does not describe policy distillation or present continual learning and policy distillation as methods for sim-to-real transfer; those parts of the claim are unsupported. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Custom simulators have often been used for navigation tasks, or simulation worlds have been created using Unity, Unreal Engine, or Gazebo.

    “Owing to the more specific environment and sensor suites that are required for different navigation tasks, custom simulators have often been used [36], [37], or simulation worlds have been created using Unity, Unreal Engine, or Gazebo [39], [42].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [37]

    Section 3.2.3 (pp. 12–13) discusses a navigation task in the Labyrinth environment, supporting the use of a simulated navigation environment as an example. It does not say that custom simulators are often used, attribute their use to task-specific sensor suites, or identify Unity, Unreal Engine, or Gazebo; those engines are attached to other citations in the citing sentence. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • The same ideas behind meta-learning can be readily extended to the reinforcement learning setting.

    “While our exposition in this section focused on the supervised learning case, the same ideas can also be readily extended to the reinforcement learning setting (Duan et al., 2016; Wang et al., 2016).”

    From Meta-learned models of cognition (2024) · cites this paper as Wang et al., 2016

  • Meta-RL approaches pre-train adaptive agents on diverse task distributions to significantly increase sample efficiency on new problems.

    “One way to address these shortcomings are meta-RL approaches, where adaptive agents are pre-trained on diverse task distributions to significantly increase sample efficiency on new problems”

    From XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (2024) · cites this paper as Wang et al., 2016

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Marcos Economides, Zeb Kurth-Nelson, Annika Lübbert, Marc Guitart-Masip, and Raymond Dolan. Modelbased reasoning in humans becomes automatic with training. PLoS Computational Biology , 11(9):e1004463, 2015.
  2. Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Proc. of Int'l Conf. on Artificial Intelligence and Statistics, AISTATS , 2012a.
  3. Tor Lattimore and Rémi Munos. Bounded regret for finite-armed structured bandits. In Advances in Neural Information Processing Systems 27 , pages 550-558, 2014.
  4. Dan Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems 27 , pages 1583-1591, 2014.
  5. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In Proceedings of The 33rd International Conference on Machine Learning , pages 1842-1850, 2016.

Cited by

Sorted by citations

Related

Authors
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M.
Affiliations
Google DeepMind (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1611.05763
arXiv
1611.05763
OpenAlex
W2550182557
Cited by
579
References
32
Citations by year
2026: 2 · 2025: 17 · 2024: 47 · 2023: 59 · 2022: 79 · 2021: 104 · 2020: 117 · 2019: 69 · 2018: 61 · 2017: 22 · 2016: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
meta-learning, recurrent neural networks, training data efficiency, learned dynamics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review