CoChat
Sign Up
PreprintCited by 47Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningMachine Learning and Algorithms

Meta-learning of Sequential Strategies

Open access
Ortega, Pedro A., Wang, Jane X., Rowland, Mark, Genewein, Tim, Kurth-Nelson, Zeb, Pascanu, Razvan, Heess, Nicolas, Veness, Joel +16 more·2019·arXiv
DOI 10.48550/arxiv.1905.03030arXiv 1905.03030
Open this paper in CoChat — free accountOpen PDF

Abstract

In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class. Our goal is to equip the reader with the conceptual foundations of this tool for building new, scalable agents that operate on broad domains. To do so, we present basic algorithmic templates for building near-optimal predictors and reinforcement learners which behave as if they had a probabilistic model that allowed them to efficiently exploit task structure. Furthermore, we recast memory-based meta-learning within a Bayesian framework, showing that the meta-learned strategies are near-optimal because they amortize Bayes-filtered data, where the adaptation is implemented in the memory dynamics as a state-machine of sufficient statistics. Essentially, memory-based meta-learning translates the hard problem of probabilistic sequential inference into a regression problem.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1905.03030

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These neurons have specialised roles required for solving a new task online via cultural transmission, a subset of the sufficient statistics which drive the state-machine.

    “These neurons have specialised roles required for solving a new task online via cultural transmission, a subset of the sufficient statistics which drive the statemachine 46 .”

    From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [46]

    In “D. Anatomy of meta-learned agents” (p. 6), the paper says a state machine’s state captures sufficient statistics of past inputs; the Conclusions (p. 12) likewise describe memory states as sufficient statistics for predicting future interactions. Our copy does not establish that particular neurons have specialised roles required for solving new tasks through cultural transmission, or that those neurons constitute a subset of the sufficient statistics. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Meta-learning can be used to construct Bayes-optimal learning algorithms.

    “We accomplish this by pointing out that meta-learning can be used to construct Bayes-optimal learning algorithms, allowing us to draw strong connections to the rational analysis of cognition.”

    From Meta-learned models of cognition (2024) · cites this paper as Ortega et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 71

  1. Felix Abramovich and Ya'acov Ritov. Statistical theory: a concise introduction . CRC Press, 2013.
  2. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems , pages 3981-3989, 2016.
  3. Jonathan Baxter. Theoretical models of learning to learn. In Learning to learn , pages 71-94. Springer, 1998.
  4. Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research , 12:149-198, 2000.
  5. Dimitri P Bertsekas. Neuro-dynamic programming. In Encyclopedia of optimization , pages 2555-2560. Springer, 2008.
  6. C.M. Bishop. Pattern Recognition and Machine Learning . Information Science and Statistics. Springer New York, 2016.
  7. Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge University Press, 2006.
  8. Yutian Chen, Yannis Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis C Cobo, Andrew Trask, Ben Laurie, et al. Sample efficient adaptive text-to-speech. arXiv preprint arXiv:1809.10460 , 2018a. · arXiv
  9. Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162 , 2019. · arXiv
  10. A Philip Dawid, Vladimir G Vovk, et al. Prequential probability: Principles and properties. Bernoulli , 5(1):125-162, 1999.
  11. Misha Denil, Pulkit Agrawal, Tejas D Kulkarni, Tom Erez, Peter Battaglia, and Nando de Freitas. Learning to perform physics experiments via deep reinforcement learning. arXiv preprint arXiv:1611.01843 , 2016. · arXiv
  12. Persi Diaconis. Sufficiency as statistical symmetry. In Proceedings of the AMS Centennial Symposium , pages 1526, 1988.
  13. Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016. · arXiv
  14. Jeffrey L Elman. Finding structure in time. Cognitive science , 14(2):179-211, 1990.
  15. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Modelagnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 , 2017. · arXiv
  16. Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning , pages 1690-1699, 2018.
  17. Ian J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradient-based neural networks. ICLR , 2014.
  18. Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta-learning probabilistic inference for prediction. arXiv preprint arXiv:1805.09921 , 2018. · arXiv
  19. Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.08930 , · arXiv
  20. Alex Graves. Supervised sequence labelling. In Supervised sequence labelling with recurrent neural networks , pages 5-13. Springer, 2012.
  21. Arthur Guez, David Silver, and Peter Dayan. Efficient bayesadaptive reinforcement learning using sample-based search. In Advances in Neural Information Processing Systems , pages 1025-1033, 2012.
  22. Lars Peter Hansen and Thomas J Sargent. Robustness . Princeton university press, 2008.
  23. Jose Hernandez-Orallo. Beyond the turing test. Journal of Logic, Language and Information , 9(4):447-466, 2000.
  24. Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks , pages 87-94. Springer, 2001.
  25. Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability . Springer Science & Business Media, 2004.
  26. Edwin T Jaynes. Probability theory: The logic of science . Cambridge university press, 2003.
  27. Michael I Jordan. Serial order: A parallel distributed processing approach. In Advances in psychology , volume 121, pages 471-495. Elsevier, 1997.
  28. Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , pages 2502-2511, 2018.
  29. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 2017.
  30. Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi. Optimization by simulated annealing. science , 220(4598): 671-680, 1983.
  31. John F Kolen. Fool's gold: Extracting finite state machines from recurrent network dynamics. In Advances in neural information processing systems , pages 501-508, 1994.
  32. John F Kolen and Stefan C Kremer. A field guide to dynamical recurrent networks . John Wiley & Sons, 2001.
  33. Bernard Osgood Koopman. On distributions admitting a sufficient statistic. Transactions of the American Mathematical society , 39(3):399-409, 1936.
  34. Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines , 17(4):391-444, 2007.
  35. Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885 , 2016. · arXiv
  36. Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine Learning Proceedings 1994 , pages 157-163. Elsevier, 1994.
  37. Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior , 10(1):6-38, 1995.
  38. Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for extensive form games. Experimental economics , 1(1):9-41, 1998.
  39. Debasis Mitra, Fabio Romeo, and Alberto SangiovanniVincentelli. Convergence and finite-time behavior of simulated annealing. Advances in applied probability , 18(3): 747-771, 1986.
  40. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013. · arXiv
  41. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937, 2016.
  42. Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based RL. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=HyxAfnA5tm.
  43. Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 , 2018. · arXiv
  44. Yaghout Nourani and Bjarne Andresen. A comparison of simulated annealing cooling strategies. Journal of Physics A: Mathematical and General , 31(41):8373, 1998.
  45. Pedro A Ortega and Daniel A Braun. A minimum relative entropy principle for learning and acting. Journal of Artificial Intelligence Research , 38:475-511, 2010.
  46. Pedro A Ortega and Shane Legg. Modeling friends and foes. arXiv preprint arXiv:1807.00196 , 2018. · arXiv
  47. Ian Osband and Benjamin Van Roy. Posterior sampling for reinforcement learning without episodes. arXiv preprint arXiv:1608.02731 , 2016. · arXiv
  48. Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, and Rahul Jain. Learning unknown markov decision processes: A thompson sampling approach. In Advances in Neural Information Processing Systems , pages 1333-1342, 2017.
  49. German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review, 2018.
  50. Judea Pearl. Causality . Cambridge university press, 2009.
  51. Scott Reed and Nando De Freitas. Neural programmerinterpreters. arXiv preprint arXiv:1511.06279 , 2015. · arXiv
  52. Daniel Ritchie, Paul Horsfall, and Noah D Goodman. Deep amortized inference for probabilistic programs. arXiv preprint arXiv:1610.05735 , 2016. · arXiv
  53. Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends R © in Machine Learning , 11(1):196, 2018.
  54. Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295 , 2015. · arXiv
  55. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850, 2016.
  56. David Silver and Joel Veness. Monte-carlo planning in large pomdps. In Advances in neural information processing systems , pages 2164-2172, 2010.
  57. David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. arXiv:1612.08810 , 2016. · arXiv
  58. Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , pages 4077-4087, 2017.
  59. Charles Spearman. "general intelligence," objectively determined and measured. The American Journal of Psychology , 15(2):201-292, 1904.
  60. Richard S Sutton, Andrew G Barto, Francis Bach, et al. Reinforcement learning: An introduction . MIT press, 1998.
  61. Richard S Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael P Bowling. Dyna-style planning with linear function approximation and prioritized sweeping. arXiv preprint arXiv:1206.3285 , 2012. · arXiv
  62. Sebastian Thrun and Lorien Pratt. Learning to learn: Intro- duction and overview. In Learning to learn , pages 3-17. Springer, 1998.
  63. Susana Urbina. Tests of intelligence. The Cambridge handbook of intelligence , pages 20-38, 2011.
  64. Bart van den Broek, Wim Wiegerinck, and Bert Kappen. Risk sensitive path integral control. In Proceedings of the TwentySixth Conference on Uncertainty in Artificial Intelligence , pages 615-622. AUAI Press, 2010.
  65. Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems , pages 36303638, 2016.
  66. Paul MB Vitanyi and Ming Li. An introduction to Kolmogorov complexity and its applications , volume 34. Springer Heidelberg, 1997.
  67. Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems , pages 5320-5329, 2017.
  68. Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems , pages 2746-2754, 2015.
  69. Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv:1703.04813 , 2017. · arXiv
  70. Zhongwen Xu, Hado van Hasselt, and David Silver. Metagradient reinforcement learning. Advances in neural information processing systems , 2018.
  71. Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Caml: Fast context adaptation via meta-learning. arXiv preprint arXiv:1810.03642 , 2018. · arXiv

Cited by

Sorted by citations

Authors
Ortega, P. A., Wang, J. X., Rowland, M., Genewein, T., Kurth-Nelson, Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., Jayakumar, S. M., McGrath, T., Miller, K., Azar, M., Osband, I., Rabinowitz, N., György, A., Chiappa, S., Osindero, S., Teh, Y. W., van Hasselt, H., de Freitas, N., Botvinick, M., & Legg, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1905.03030
arXiv
1905.03030
OpenAlex
W2944299231
Cited by
47
References
5
Citations by year
2026: 1 · 2025: 1 · 2024: 8 · 2023: 4 · 2022: 4 · 2021: 17 · 2020: 7 · 2019: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
meta-learning, sample efficiency, Bayesian meta-learning, amortized inference, sufficient statistics, state machines, reinforcement learning, probabilistic models, regression analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review