CoChat
Sign Up
PreprintCited by 13Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningAI-based Problem Solving and Planning

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents

Open access
Wang, Jane X., King, Michael, Porcel, Nicolas, Kurth-Nelson, Zeb, Zhu, Tina, Deck, Charlie, Choy, Peter, Cassin, Mary, Reynolds, Malcolm, Song, Francis, Buttimore, Gavin, Reichert, David P., Rabinowitz, Neil, Matthey, Loic, Hassabis, Demis, Lerchner, Alexander, Botvinick, Matthew·2021·arXiv
DOI 10.48550/arxiv.2102.02926arXiv 2102.02926
Open this paper in CoChat — free accountOpen PDF

Abstract

There has been rapidly growing interest in meta-learning as a method for increasing the flexibility and sample efficiency of reinforcement learning. One problem in this area of research, however, has been a scarcity of adequate benchmark tasks. In general, the structure underlying past benchmarks has either been too simple to be inherently interesting, or too ill-defined to support principled analysis. In the present work, we introduce a new benchmark for meta-RL research, emphasizing transparency and potential for in-depth analysis as well as structural richness. Alchemy is a 3D video game, implemented in Unity, which involves a latent causal structure that is resampled procedurally from episode to episode, affording structure learning, online inference, hypothesis testing and action sequencing based on abstract domain knowledge. We evaluate a pair of powerful RL agents on Alchemy and present an in-depth analysis of one of these agents. Results clearly indicate a frank and specific failure of meta-learning, providing validation for Alchemy as a challenging benchmark for meta-RL. Concurrent with this report, we are releasing Alchemy as public resource, together with a suite of analysis tools and sample agent trajectories.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.02926

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most recent works on adaptive agents avoid complex environments in favor of more simplistic ones.

    “most recent works on adaptive agents (Laskin et al., 2022; Lee et al., 2023; Lu et al., 2023; Norman & Clune, 2023) avoid complex environments in favor of more simplistic ones”

    From XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (2024) · cites this paper as Wang et al., 2021

    Related work (p. 3) says the most popular meta-RL benchmarks use relatively simple task parameterizations, supporting the claim's general contrast. It also describes environments designed for greater complexity, so it does not establish that most recent work on adaptive agents avoids complex environments. The 2021 paper cannot assess the cited 2022–2023 works. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Meta-RL lacked environments that were both non-trivial and computationally efficient.

    “Unlike other RL subfields, meta-RL lacked environments that were both non-trivial and computationally efficient.”

    From XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (2024) · cites this paper as Wang et al., 2021

    The Introduction (p. 2) says previous meta-RL environments tended to trade off accessibility and interestingness, and Related work (p. 3) contrasts simple benchmarks with more complex ones. But the paper defines accessibility as the ability to analyze a task distribution, not computational efficiency; it also does not establish the comparison with other RL subfields. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • There are reported cases in which meta-learning failed to find the Bayes-optimal solution.

    “Indeed, there are reported cases in which meta-learning failed to find the Bayes-optimal solution (Wang et al., 2021).”

    From Meta-learned models of cognition (2024) · cites this paper as Wang et al., 2021

  • Contemporary meta-learning algorithms are able to find approximately Bayes-optimal solutions to simple problems but sometimes struggle to do so on more complex ones.

    “While contemporary meta-learning algorithms are able to find approximately Bayes-optimal solutions to simple problems, they sometimes struggle to do so on more complex ones (e.g. as in the earlier discussed work of Wang et al. (2021)).”

    From Meta-learned models of cognition (2024) · cites this paper as Wang et al. (2021)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 64

  1. [1][1] Jonathan Baxter. Theoretical models of learning to learn. In Learning to learn , pages 71-94. Springer, 1998. -
  2. [2][2] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801 , 2016. - · arXiv
  3. [3][3] Timothy EJ Behrens, Mark W Woolrich, Mark E Walton, and Matthew FS Rushworth. Learning the value of information in an uncertain world. Nature neuroscience , 10(9):1214-1221, 2007. -
  4. [4][4] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013. -
  5. [5][5] Y Bengio, S Bengio, and J Cloutier. Learning a synaptic learning rule. In Neural Networks, 1991., IJCNN-91-Seattle International Joint Conference on , volume 2, pages 969-vol. IEEE, 1991. -
  6. [6][6] Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences , 23(5):408-422, 2019. -
  7. [8][8] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning , pages 2048-2056. PMLR, 2020. -
  8. [9][9] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning , pages 1282-1289. PMLR, 2019. -
  9. [11][11] Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162 , 2019. - · arXiv
  10. [12][12] Nathaniel D Daw et al. Trial-by-trial data analysis using computational models. Decision making, affect, and learning: Attention and performance XXIII , 23(1), 2011. -
  11. [13][13] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. arXiv preprint arXiv:2012.02096 , 2020. - · arXiv
  12. [14][14] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016. - · arXiv
  13. [15][15] Michael O Duff. Design for an optimal probe. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 131-138, 2003. -
  14. [16][16] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018. - · arXiv
  15. [17][17] Simon Farrell and Stephan Lewandowsky. Computational modeling of cognition and behavior . Cambridge University Press, 2018. -
  16. [18][18] Valerii Vadimovich Fedorov. Theory of optimal experiments . Elsevier, 2013. -
  17. [19][19] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 , 2017. - · arXiv
  18. [20][20] Wilson S Geisler. Ideal observer analysis. The visual neurosciences , 10(7):12-12, 2003. -
  19. [21][21] John C Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B (Methodological) , pages 148-177, 1979. -
  20. [22][22] Alison Gopnik, Clark Glymour, David M Sobel, Laura E Schulz, Tamar Kushnir, and David Danks. A theory of causal learning in children: causal maps and bayes nets. Psychological review , 111(1):3, 2004. -
  21. [23][23] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.08930 , 2018. - · arXiv
  22. [24][24] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Metareinforcement learning of structured exploration strategies. arXiv preprint arXiv:1802.07245 , 2018. - · arXiv
  23. [25][25] Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.01719 , 2020. - · arXiv
  24. [26][26] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput. , 9(8):17351780, 1997. -
  25. [27][27] Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424 , 2019. - · arXiv
  26. [28][28] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019. -
  27. [30][30] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations , 2017. -
  28. [31][31] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI , pages 4246-4247. Citeseer, 2016. -
  29. [33][33] Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. arXiv preprint arXiv:1809.02627 , 2018. - · arXiv
  30. [34][34] Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. arXiv preprint arXiv:1902.01378 , 2019. - · arXiv
  31. [35][35] Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. arXiv preprint arXiv:2006.13760 , 2020. - · arXiv
  32. [37][37] Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In International Conference on Machine Learning , pages 6925-6935. PMLR, 2021. -
  33. [38][38] Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. arXiv preprint arXiv:1904.00956 , 2019. - · arXiv
  34. [39][39] Vladimir Mikulik, Grégoire Delétang, Tom McGrath, Tim Genewein, Miljan Martic, Shane Legg, and Pedro A Ortega. Meta-trained agents implement bayes-optimal agents. arXiv preprint arXiv:2010.11223 , 2020. - · arXiv
  35. [40][40] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. International Conference on Learning Representations , 2017. -
  36. [41][41] Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:1804.03720 , 2018. - · arXiv
  37. [43][43] Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv:1905.03030 , 2019. - · arXiv
  38. [44][44] Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568 , 2019. - · arXiv
  39. [45][45] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342 , 2015. - · arXiv
  40. [46][46] Emilio Parisotto, H Francis Song, Jack W Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning , 2019. -
  41. [47][47] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning , pages 5331-5340. PMLR, 2019. -
  42. [48][48] Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence , 2(8):428-436, 2020. -
  43. [49][49] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016. - · arXiv
  44. [50][50] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850, 2016. -
  45. [51][51] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook . PhD thesis, Technische Universität München, 1987. -
  46. [52][52] Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835 , 2018. - · arXiv
  47. [53][53] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020. -
  48. [54][54] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10740-10749, 2020. -
  49. [55][55] H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: on-policy maximum a posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations , 2019. -
  50. [56][56] Bradly Stadie, Ge Yang, Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. The importance of sampling inmeta-reinforcement learning. Advances in Neural Information Processing Systems , 31:9280-9290, 2018. -
  51. [58][58] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning . MIT Press, 1998. -
  52. [59][59] Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science , 331(6022):1279-1285, 2011. -
  53. [60][60] Sebastian Thrun and Lorien Pratt. Learning to learn . Springer Science & Business Media, 1998. -
  54. [61][61] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012. -
  55. [62][62] Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J Gershman, and Joshua B Tenenbaum. Human-level reinforcement learning through theory-based modeling, exploration, and planning. arXiv preprint arXiv:2107.12544 , 2021. - · arXiv
  56. [64][64] Joaquin Vanschoren. Meta-learning. In Automated Machine Learning , pages 35-61. Springer, Cham, 2019. -
  57. [65][65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017. -
  58. [66][66] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019. -
  59. [67][67] Jane X Wang. Meta-learning in natural and artificial intelligence. arXiv preprint arXiv:2011.13464 , 2020. - · arXiv
  60. [69][69] Marek Wydmuch, Michał Kempka, and Wojciech Ja´ skowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games , 2018. -
  61. [70][70] Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Halma: Humanlike abstraction learning meets affordance in rapid problem solving. arXiv preprint arXiv:2102.11344 , 2021. - · arXiv
  62. [71][71] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Advances in neural information processing systems , 31:2396-2407, 2018. -
  63. [72][72] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning , pages 1094-1100. PMLR, 2020. -
  64. [73][73] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via metalearning. arXiv preprint arXiv:1910.08348 , 2019. · arXiv

Cited by

Sorted by citations

Related

Metadata

Authors
Wang, J. X., King, M., Porcel, N., Kurth-Nelson, Z., Zhu, T., Deck, C., Choy, P., Cassin, M., Reynolds, M., Song, F., Buttimore, G., Reichert, D. P., Rabinowitz, N., Matthey, L., Hassabis, D., Lerchner, A., & Botvinick, M.
Affiliations
  • Google (United States) (US)
  • University of Massachusetts Chan Medical School (US)
  • University College London (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2102.02926
arXiv
2102.02926
OpenAlex
W3206344027
Cited by
10
References
0
Citations by year
2024: 5 · 2023: 3 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
meta-reinforcement learning, benchmark, alchemy, 3D games, structure learning, online inference, hypothesis testing, action sequencing, sample efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review