CoChat
Sign Up
PreprintCited by 28Advanced Bandit Algorithms ResearchReinforcement Learning in RoboticsDomain Adaptation and Few-Shot Learning

Meta-trained agents implement Bayes-optimal agents

Open access
Mikulik, Vladimir, Delétang, Grégoire, McGrath, Tom, Genewein, Tim, Martic, Miljan, Legg, Shane, Ortega, Pedro A.·2020·arXiv
DOI 10.48550/arxiv.2010.11223arXiv 2010.11223
Open this paper in CoChat — free accountOpen PDF

Abstract

Memory-based meta-learning is a powerful technique to build agents that adapt fast to any task within a target distribution. A previous theoretical study has argued that this remarkable performance is because the meta-training protocol incentivises agents to behave Bayes-optimally. We empirically investigate this claim on a number of prediction and bandit tasks. Inspired by ideas from theoretical computer science, we show that meta-learned and Bayes-optimal agents not only behave alike, but they even share a similar computational structure, in the sense that one agent system can approximately simulate the other. Furthermore, we show that Bayes-optimal agents are fixed points of the meta-learning dynamics. Our results suggest that memory-based meta-learning might serve as a general technique for numerically approximating Bayes-optimal agents - that is, even for task distributions for which we currently don't possess tractable models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.11223

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Training agents to imitate via meta-reinforcement learning embeds the logic for a state-machine capable of approximately Bayes-optimal cultural transmission into the neural network's weights.

    “Training agents to imitate via meta-reinforcement learning embeds the logic for a state-machine capable of approximately Bayes-optimal cultural transmission into the neural network's weights 45 .”

    From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [45]

    The Introduction (p. 2) says meta-learned agents approximately simulate Bayes-optimal agents using computational structures corresponding to state machines. The Preliminaries (p. 2) describe RNN weights defining a state machine. But the paper studies prediction and bandit tasks; our copy does not support the claim that agents were trained to imitate or that the result applies to cultural transmission. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

  • Meta-learning procedures similar to the one discussed often lead to networks that closely approximate Bayesian inference.

    “in practice, it has been observed that meta-learning procedures similar to the one discussed here often lead to networks that closely approximate Bayesian inference (Mikulik, Delétang, et al., 2020; Rabinowitz, 2019).”

    From Meta-learned models of cognition (2024) · cites this paper as Mikulik, Delétang, et al., 2020

  • Meta-learning can be used to construct Bayes-optimal learning algorithms.

    “We accomplish this by pointing out that meta-learning can be used to construct Bayes-optimal learning algorithms, allowing us to draw strong connections to the rational analysis of cognition.”

    From Meta-learned models of cognition (2024) · cites this paper as Mikulik, Delétang, et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 60

  1. [1][1] Y Bengio, S Bengio, and J Cloutier. Learning a synaptic learning rule. In IJCNN-91-Seattle International Joint Conference on Neural Networks , volume 2, pages 969-vol. IEEE, 1991. -
  2. [3][3] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn , pages 3-17. Springer, 1998. -
  3. [4][4] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks , pages 87-94. Springer, 2001. -
  4. [5][5] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning , pages 1842-1850, 2016. -
  5. [6][6] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 , 2016. - · arXiv
  6. [7][7] Eliza Strickland. Yoshua Bengio, revered architect of AI, has some ideas about what to build next. IEEE Spectrum , December 2019. -
  7. [8][8] Daniel A Braun, Ad Aertsen, Daniel M Wolpert, and Carsten Mehring. Motor task variation induces structural learning. Current Biology , 19(4):352-357, 2009. -
  8. [9][9] Daniel A Braun, Carsten Mehring, and Daniel M Wolpert. Structure learning in action. Behavioural Brain Research , 206(2):157-165, 2010. -
  9. [10][10] Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv:1905.03030 , 2019. - · arXiv
  10. [11][11] Michael O'Gordon Duff and Andrew Barto. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes . PhD thesis, University of Massachusetts at Amherst, 2002. -
  11. [12][12] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. In Advances in Neural Information Processing Systems , pages 15670-15679, 2019. -
  12. [13][13] Niru Maheswaranathan and David Sussillo. How recurrent networks implement contextual processing in sentiment analysis. arXiv preprint arXiv:2004.08013 , 2020. - · arXiv
  13. [14][14] Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. In Advances in Neural Information Processing Systems , pages 8535-8545, 2019. -
  14. [15][15] Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations , 2019. -
  15. [18][18] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019. -
  16. [19][19] David Marr. Vision: A computational investigation into the human representation and processing of visual information . MIT Press, 2010. -
  17. [20][20] John F Kolen and Stefan C Kremer. A field guide to dynamical recurrent networks . John Wiley &Sons, 2001. -
  18. [22][22] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014. - · arXiv
  19. [23][23] AJ Robinson and Frank Fallside. The utility driven dynamic error propagation network . University of Cambridge Department of Engineering Cambridge, 1987. -
  20. [24][24] Paul J Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural networks , 1(4):339-356, 1988. -
  21. [25][25] George H Mealy. A method for synthesizing sequential circuits. The Bell System Technical Journal , 34(5):1045-1079, 1955. -
  22. [26][26] Michael Sipser. Introduction to the theory of computation. ACM Sigact News , 27(1):27-29, 1996. -
  23. [28][28] Edmund M Clarke Jr, Orna Grumberg, Daniel Kroening, Doron Peled, and Helmut Veith. Model checking . MIT press, 2018. -
  24. [30][30] Jos C.M. Baeten and Davide Sangiorgi. Concurrency theory: A historical perspective on coinduction and process calculi. In Jörg H. Siekmann, editor, Computational Logic , volume 9 of Handbook of the History of Logic , pages 399 - 442. North-Holland, 2014. -
  25. [33][33] Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis . CRC press, 2013. -
  26. [34][34] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. preprint , page 28, 2018. -
  27. [35][35] John C Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society: Series B (Methodological) , 41(2):148-164, 1979. -
  28. [36][36] Harrison Edwards and Amos Storkey. Towards a neural statistician. arXiv preprint arXiv:1606.02185 , 2016. - · arXiv
  29. [37][37] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT Press, 2018. -
  30. [38][38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. - · arXiv
  31. [39][39] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deepRL with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018. - · arXiv
  32. [40][40] Ingwer Borg and Patrick JF Groenen. Modern multidimensional scaling: Theory and applications . Springer Science & Business Media, 2005. -
  33. [41][41] Antoine Girard and George J Pappas. Approximate bisimulations for nonlinear dynamical systems. In Proceedings of the 44th IEEE Conference on Decision and Control , pages 684-689. IEEE, 2005. -
  34. [42][42] Neil C Rabinowitz. Meta-learners' learning dynamics are unlike learners'. arXiv preprint arXiv:1905.01320 , 2019. - · arXiv
  35. [43][43] Wolfgang Maass and Henry Markram. On the computational power of circuits of spiking neurons. Journal of Computer and System Sciences , 69(4):593-616, 2004. -
  36. [44][44] Bryan Lim, Stefan Zohren, and Stephen Roberts. Recurrent neural filters: Learning independent bayesian filtering steps for time series prediction. arXiv preprint arXiv:1901.08096 , 2019. - · arXiv
  37. [45][45] Rahul G Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence , pages 2101-2109, 2017. -
  38. [46][46] Huseyin Coskun, Felix Achilles, Robert DiPietro, Nassir Navab, and Federico Tombari. Long short-term memory kalman filters: Recurrent neural estimators for pose regularization. In Proceedings of the IEEE International Conference on Computer Vision , pages 5524-5532, 2017. -
  39. [47][47] Tsendsuren Munkhdalai and Hong Yu. Meta networks. Proceedings of Machine Learning Research , 70:2554, 2017. -
  40. [48][48] Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. In Advances in Neural Information Processing Systems , pages 13331-13342, 2019. -
  41. [49][49] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1126-1135. JMLR. org, 2017. -
  42. [50][50] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. In International Conference on Learning Representations , 2018. -
  43. [51][51] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In International Conference on Learning Representations , 2020. -
  44. [52][52] Cosma Rohilla Shalizi and James P Crutchfield. Computational mechanics: Pattern and prediction, structure and simplicity. Journal of Statistical Physics , 104(3-4):817-879, 2001. -
  45. [53][53] Amy Zhang, Zachary C Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, and Tommaso Furlanello. Learning causal state representations of partially observable environments. arXiv preprint arXiv:1906.10437 , 2019. - · arXiv
  46. [54][54] Leslie Pack Kaelbling and Stanley J Rosenschein. Action and planning in embedded agents. Robotics and Autonomous Systems , 6(1-2):35-48, 1990. -
  47. [55][55] Stanley J Rosenschein and Leslie Pack Kaelbling. A situated view of representation and control. Artificial Intelligence , 73(1-2):149-173, 1995. -
  48. [56][56] Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics , 7:49-72, 2019. -
  49. [57][57] Hansem Sohn, Devika Narain, Nicolas Meirhaeghe, and Mehrdad Jazayeri. Bayesian computation through cortical latent dynamics. Neuron , 103(5):934-947, 2019. -
  50. [58][58] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. In Advances in Neural Information Processing Systems , pages 15603-15615, 2019. -
  51. [59][59] Ishita Dasgupta, Eric Schulz, Joshua B. Tenenbaum, and Samuel J. Gershman. A theory of learning to infer. bioRxiv , 2019. -
  52. [60][60] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding DQNs. In International Conference on Machine Learning , pages 1899-1908, 2016. -
  53. [61][61] Anurag Koul, Sam Greydanus, and Alan Fern. Learning finite state representations of recurrent policy networks. arXiv preprint arXiv:1811.12530 , 2018. - · arXiv
  54. [63][63] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 , 2013. - · arXiv
  55. [64][64] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision , pages 818-833. Springer, 2014. -
  56. [65][65] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713 , 2016. - · arXiv
  57. [66][66] Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450 , 2016. - · arXiv
  58. [67][67] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017. - · arXiv
  59. [68][68] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences , 2020. -
  60. [69][69] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144, 2016.

Cited by

Sorted by citations

Related

Authors
Mikulik, V., Delétang, G., McGrath, T., Genewein, T., Martic, M., Legg, S., & Ortega, P. A.
Affiliations
  • University of Oxford (GB)
  • Imperial College London (GB)
  • Robert Bosch (Germany) (DE)
  • Consejo Superior de Investigaciones Científicas (ES)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
18691-18703
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.11223
arXiv
2010.11223
OpenAlex
W3104803801
Cited by
24
References
0
Citations by year
2025: 1 · 2024: 2 · 2023: 4 · 2022: 14 · 2021: 1 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
meta-training, agent-based modeling, fixed points, bandit problems, prediction tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review