CoChat
Sign Up
PreprintCited by 44Reinforcement Learning in RoboticsArtificial Intelligence in GamesAI-based Problem Solving and Planning

The NetHack Learning Environment

Open access
Küttler, Heinrich, Nardelli, Nantas, Miller, Alexander H., Raileanu, Roberta, Selvatici, Marco, Grefenstette, Edward, Rocktäschel, Tim·2020·arXiv
DOI 10.48550/arxiv.2006.13760arXiv 2006.13760
Open this paper in CoChat — free accountOpen PDF

Abstract

Progress in Reinforcement Learning (RL) algorithms goes hand-in-hand with the development of challenging environments that test the limits of current methods. While existing RL environments are either sufficiently complex or based on fast simulation, they are rarely both. Here, we present the NetHack Learning Environment (NLE), a scalable, procedurally generated, stochastic, rich, and challenging environment for RL research based on the popular single-player terminal-based roguelike game, NetHack. We argue that NetHack is sufficiently complex to drive long-term research on problems such as exploration, planning, skill acquisition, and language-conditioned RL, while dramatically reducing the computational resources required to gather a large amount of experience. We compare NLE and its task suite to existing alternatives, and discuss why it is an ideal medium for testing the robustness and systematic generalization of RL agents. We demonstrate empirical success for early stages of the game using a distributed Deep RL baseline and Random Network Distillation exploration, alongside qualitative analysis of various agents trained in the environment. NLE is open source at https://github.com/facebookresearch/nle.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.13760

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • NetHack typically takes years of practice to achieve an ascension.

    “The game is known to be incredibly hard, with it typically taking years of practice to achieve an ascension (Küttler et al., 2020).”

    From Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (2024) · cites this paper as Küttler et al., 2020

    The Introduction (p. 2) says NetHack is difficult for humans to master, and Appendix A (p. 15) calls winning by ascension immensely challenging. Neither passage, nor the passages retrieved for years of practice and time to ascend, says that achieving an ascension typically takes years of practice. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • It is unreasonable to expect a standard RL policy trained tabula rasa on a single level of NLE to generalise to dissimilar levels, as it might encounter entirely unseen entities or very out-of-distribution combinations of entities and level layouts.

    “For example, it is unreasonable to expect a standard RL policy trained tabula rasa on a single level of NLE (Küttler et al., 2020) to generalise to dissimilar levels, as it might encounter entirely unseen entities or very out-of-distribution combinations of entities and level layouts.”

    From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Küttler et al., 2020

    Section 3.1 (p. 7) reports that agents trained on a limited set of NLE seeds achieved high training performance but generalized poorly to held-out seeds. This supports the concern about training on too little variation, but the paper does not establish the specific single-level scenario or attribute poor generalization to unseen entities and combinations of entities and layouts. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • MiniHack and NLE contain enough complexity to test for many types and strengths of non-visual generalisation but save on computation due to the lack of complex visual processing required.

    “These environments contain enough complexity to test for many types and strengths of non-visual generalisation but save on computation due to the lack of complex visual processing required.”

    From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Küttler et al., 2020

    The Introduction (p. 1) presents NLE as a fast, complex environment for testing systematic generalization; §2.2 (p. 4) describes its symbolic observations, and Broader Impact (p. 10) contrasts its low computational cost with visually sophisticated environments. The source supports the claim for NLE, but does not address MiniHack, so it cannot establish the claim for both environments. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The NetHack Learning Environment suggests using 1 billion environment interactions.

    “while The NetHack Learning Environment suggests using 1 billion (Küttler et al., 2020).”

    From Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (2024) · cites this paper as Küttler et al., 2020

  • A controllable environment with a structured context space is required to test these forms of compositional generalisation, and to ensure that the agent is seeing truly novel combinations at test time.

    “Crucially, a controllable environment with a structured context space is required to test these forms of compositional generalisation, and to ensure that the agent is seeing truly novel combinations at test time; this is difficult to validate in PCG environments like OpenAI Procgen (Cobbe et al., 2020a) or NLE (Küttler et al., 2020).”

    From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Küttler et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 56 · Cited by 44

Research works printed but not linked: 58

  1. [1][1] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML , 2004. -
  2. [4][4] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine , 34(6):26-38, 2017. -
  3. [5][5] Andrea Asperti, Daniele Cortesi, Carlo De Pieri, Gianmaria Pedrini, and Francesco Sovrano. Crawling in rogue's dungeons with deep reinforcement techniques. IEEE Transactions on Games , 2019. -
  4. [6][6] Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In NeurIPS , 2018. -
  5. [7][7] Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR , abs/1612.03801, 2016. - · arXiv
  6. [8][8] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NeurIPS , 2016. -
  7. [9][9] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 2013. -
  8. [10][10] S. R. K. Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte-carlo framework. In ACL , 2011. -
  9. [11][11] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. CoRR , abs/1606.01540, 2016. - · arXiv
  10. [12][12] Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In ICLR , 2019. -
  11. [13][13] Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR , 2019. -
  12. [15][15] Jonathan Campbell and Clark Verbrugge. Exploration in NetHack with secret discovery. IEEE Transactions on Games , 2018. -
  13. [16][16] Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In ICLR , 2018. -
  14. [18][18] Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv:1912.01588 , 2019. - · arXiv
  15. [19][19] Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In ICML , 2019. -
  16. [20][20] Dustin Dannenhauer, Michael W Floyd, Jonathan Decker, and David W Aha. Dungeon crawl stone soup as an evaluation domain for artificial intelligence. Workshop on Games and Simulations for Artificial Intelligence, AAAI , 2019. -
  17. [21][21] Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In NeurIPS , 1992. -
  18. [22][22] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-Explore: A New Approach for Hard-exploration Problems. arXiv preprint arXiv:1901.10995 , 2019. - · arXiv
  19. [23][23] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return then explore. CoRR , abs/2004.12919, 2020. - · arXiv
  20. [24][24] Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML , 2018. -
  21. [27][27] Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip H.S. Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In ICML , 2017. -
  22. [28][28] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 2015. -
  23. [29][29] William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. The MineRL competition on sample efficient reinforcement learning using human priors. NeurIPS Competition Track , 2019. -
  24. [30][30] Luke Harries, Sebastian Lee, Jaroslaw Rzepecki, Katja Hofmann, and Sam Devlin. Mazeexplorer: A customisable 3d benchmark for assessing generalisation in reinforcement learning. In IEEE Conference on Games , 2019. -
  25. [31][31] Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI , 2017. -
  26. [32][32] Felix Hill, Andrew K. Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClelland, and Adam Santoro. Emergent systematic generalization in a situated agent. In ICLR , 2020. -
  27. [33][33] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9 (8):1735-1780, 1997. doi: 10.1162/neco.1997.9.8.1735. - · DOI · Paper page
  28. [34][34] International Roguelike Development Conference. Berlin Interpretation. http://www. roguebasin.com/index.php?title=Berlin\_Interpretation , 2008. Accessed: 202001-08. -
  29. [35][35] Yacine Jernite, Kavya Srinet, Jonathan Gray, and Arthur Szlam. Craftassist instruction parsing: Semantic parsing for a minecraft assistant. CoRR , abs/1905.01978, 2019. - · arXiv
  30. [36][36] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. In NeurIPS , 2019. -
  31. [37][37] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI , 2016. -
  32. [38][38] Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning. In IJCAI , 2019. -
  33. [39][39] Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv:1806.10729 , 2018. - · arXiv
  34. [40][40] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research , 1996. -
  35. [41][41] Yuji Kanagawa and Tomoyuki Kaneko. Rogue-gym: A new challenge for generalization in reinforcement learning. In IEEE Conference on Games , 2019. -
  36. [42][42] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Ja´ skowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games , 2016. -
  37. [44][44] Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Distributed RL. arXiv , abs/1910.03552, 2019. - · arXiv
  38. [45][45] David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurIPS , 2017. -
  39. [46][46] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In IJCAI , 2019. -
  40. [48][48] Daniel J. Mankowitz, Augustin Zídek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, Hado van Hasselt, David Silver, and Tom Schaul. Unicorn: Continual learning with a universal, off-policy agent. arXiv , abs/1802.08294, 2018. - · arXiv
  41. [49][49] Nantas Nardelli, Gabriel Synnaeve, Zeming Lin, Pushmeet Kohli, Philip H.S. Torr, and Nicolas Usunier. Value propagation networks. In ICLR , 2019. -
  42. [51][51] Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:1804.03720 , 2018. - · arXiv
  43. [52][52] Santiago Ontanón, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, and Mike Preuss. A survey of real-time strategy game ai research and competition in starcraft. IEEE Transactions on Computational Intelligence and AI in Games , 2013. -
  44. [53][53] Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In ICML , 2017. -
  45. [54][54] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks , 2019. -
  46. [55][55] Ronald Parr and Stuart J. Russell. Reinforcement learning with hierarchies of machines. In NeurIPS , 1997. -
  47. [56][56] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML , 2017. -
  48. [57][57] Roberta Raileanu and Tim Rocktäschel. RIDE: Rewarding impact-driven exploration for procedurally-generated environments. In ICLR , 2020. -
  49. [60][60] Sebastian Risi and Julian Togelius. Procedural content generation: From automatically generating game levels to increasing generality in machine learning. CoRR , abs/1911.13071, 2019. - · arXiv
  50. [61][61] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NeurIPS , 2019. -
  51. [62][62] Tim Salimans and Richard Chen. Learning montezuma's revenge from a single demonstration. CoRR , abs/1812.03381, 2018. - · arXiv
  52. [64][64] Gabriel Synnaeve, Nantas Nardelli, Alex Auvolat, Soumith Chintala, Timothée Lacroix, Zeming Lin, Florian Richoux, and Nicolas Usunier. TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games. arXiv preprint arXiv:1611.00625 , 2016. - · arXiv
  53. [66][66] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # Exploration: A study of count-based exploration for deep reinforcement learning. In NeurIPS , 2017. -
  54. [67][67] Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal Machine Learning Research , 2009. -
  55. [68][68] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Manfred Otto Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In ICML , 2017. -
  56. [69][69] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. StarCraft II: A New Challenge for Reinforcement Learning. arXiv preprint arXiv:1708.04782 , 2017. - · arXiv
  57. [71][71] Chang Ye, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Rotation, translation, and cropping for zero-shot generalization. CoRR , abs/2001.09908, 2020. - · arXiv
  58. [72][72] Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. RTFM: Generalising to new environment dynamics via reading. In ICLR , 2020.

Authors
Küttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Selvatici, M., Grefenstette, E., & Rocktäschel, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
7671-7684
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.13760
arXiv
2006.13760
OpenAlex
W3037871539
Cited by
44
References
56
Citations by year
2025: 1 · 2024: 5 · 2023: 12 · 2022: 6 · 2021: 13 · 2020: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
Engineering and Physical Sciences Research Council (EP/N019474/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
NetHack, roguelike games, reinforcement learning environment, exploration, planning, skill acquisition, deep reinforcement learning, random network distillation, agent robustness, generalization, distributed training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review