Benchmarking the Spectrum of Agent Capabilities
Open accessAbstract
Evaluating the general abilities of intelligent agents requires complex simulation environments. Existing benchmarks typically evaluate only one narrow task per environment, requiring researchers to perform expensive training runs on many different environments. We introduce Crafter, an open world survival game with visual inputs that evaluates a wide range of general abilities within a single environment. Agents either learn from the provided reward signal or through intrinsic objectives and are evaluated by semantically meaningful achievements that can be unlocked during each episode, such as discovering resources and crafting tools. Consistently unlocking all achievements requires strong generalization, deep exploration, and long-term reasoning. We experimentally verify that Crafter is of appropriate difficulty to drive future research and provide baselines scores of reward agents and unsupervised agents. Furthermore, we observe sophisticated behaviors emerging from maximizing the reward signal, such as building tunnel systems, bridges, houses, and plantations. We hope that Crafter will accelerate research progress by quickly evaluating a wide spectrum of abilities.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Crafter is an open-source 2D survival game designed specifically for research on strong generalization, deep exploration, and long-term reasoning in reinforcement learning.
“Crafter (Hafner, 2021) is an open-source 2D survival game designed specifically for research on strong generalization, deep exploration, and long-term reasoning in reinforcement learning.”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Hafner, 2021
The Introduction (p. 1) describes Crafter as a survival game for reinforcement-learning research with randomly generated 2D worlds, and identifies strong generalization, deep exploration, and long-term reasoning as design challenges. It provides installation instructions, but the passages found do not establish that Crafter is open-source. (LLM, conf=0.84)
Experimental — accuracy not yet measuredCrafter's evaluation protocol allocates algorithms only 1 million environment interactions.
“While Crafter has become a popular benchmark, the evaluation protocol proposed allocates algorithms only 1 million environment interactions, a very limiting constraint when compared to other RL benchmarks.”
From Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (2024) · cites this paper as Hafner, 2021
Crafter was proposed to evaluate strong generalization, deep exploration, and long-term reasoning.
“Crafter was proposed to evaluate “strong generalization, deep exploration, and long-term reasoning” (Hafner, 2021).”
From Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (2024) · cites this paper as Hafner, 2021
Crafter is a Minecraft-inspired, procedurally generated environment that combines resource gathering, crafting, and combat elements.
“It is a Minecraft-inspired, procedurally generated environment that combines resource gathering, crafting, and combat elements.”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Hafner, 2021
Crafter includes a comprehensive set of tasks and achievements, enabling researchers to evaluate agent performance across multiple objectives and time scales.
“Additionally, the game includes a comprehensive set of tasks and achievements, enabling researchers to evaluate agent performance across multiple objectives and time scales.”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Hafner, 2021
Games have long been identified as go-to benchmarks for intelligent generalist agents.
“games have long been identified as go-to benchmarks for intelligent generalist agents (Pell, 2011; Genesereth et al., 2005; Whiteson et al., 2010; Schaul et al., 2011; Bellemare et al., 2013; Cˆ ot´ e et al., 2019; Hafner, 2021; Guss et al., 2021; Fan et al., 2022).”
From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hafner, 2021
The games in SmartPlay range in complexity, from requiring simple one-step reasoning and rule-following in Bandits, to intricate long-term planning, multi-hop dependencies, and learning from interactions in Crafter and Hanoi.
“The games range in complexity, from requiring simple one-step reasoning and rule-following in Bandits, to intricate long-term planning, multi-hop dependencies, and learning from interactions in Crafter (Hafner, 2021) and Hanoi.”
From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hafner, 2021
SmartPlay takes a suite of benchmarks developed over different times to best represent a broad range of difficulties and skills.
“SmartPlay takes a suite of benchmarks (Brockman et al., 2016; Hafner, 2021; Fan et al., 2022) developed over different times to best represent a broad range of difficulties and skills.”
From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hafner, 2021
To succeed in Crafter, the agent needs to collect different resources and craft more advanced weapons/tools to unlock more skills and achievements, while balancing crafting goals with survival goals like maintaining health, thirst, food, and rest.
“The agent also needs to collect different resources and craft more advanced weapons/tools to unlock more skills and achievements, while at the same time balance crafting goals with survival goals like maintaining health, thirst, food, and rest (Hafner, 2021).”
From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hafner, 2021
To ensure compatibility with prior works, reward aligns with the score/reward definition in games originally designed for RL.
“To ensure compatibility with prior works, reward aligns with the score/reward definition in games originally designed for RL (i.e., Bandits, Rock Paper Scissors, Messenger, Crafter (Hanjie et al., 2021; Hafner, 2021)).”
From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hafner, 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · see citing works
Unity: A General Platform for Intelligent Agents
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.02627 · Cited by 626 · see citing works
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaśkowski, W. (2016) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 550 · see citing works
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527 · see citing works
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · see citing works
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401 · see citing works
Rainbow: Combining Improvements in Deep Reinforcement LearningPreprint
Hessel, M., Modayil, J., van Hasselt, H. P., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., & Silver, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.02298 · Cited by 377 · see citing works
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359 · see citing works
Mastering Atari with Discrete World Models
Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020) · arXiv · doi:10.48550/arxiv.2010.02193 · Cited by 247 · see citing works
Self-citationAgent57: Outperforming the Atari Human BenchmarkPreprint
Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, D., & Blundell, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13350 · Cited by 216 · see citing works
The Malmo platform for artificial intelligence experimentation
Johnson, M., Hofmann, K., Hutton, T. J., & Bignell, D. E. (2016) · International Joint Conference on Artificial Intelligence · Cited by 205 · see citing works
Dopamine: A Research Framework for Deep Reinforcement LearningPreprint
Castro, P. S., Moitra, S., Gelada, C., Kumar, S., & Bellemare, M. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06110 · Cited by 155 · see citing works
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129 · see citing works
Self-citationWhen to use parametric models in reinforcement learning?Preprint
van Hasselt, H. P., Hessel, M., & Aslanides, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05243 · Cited by 101 · see citing works
The NetHack Learning Environment
Küttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Selvatici, M., Grefenstette, E., & Rocktäschel, T. (2020) · arXiv · doi:10.48550/arxiv.2006.13760 · Cited by 44 · see citing works
Leveraging Procedural Generation to Benchmark Reinforcement Learning
Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2020) · International Conference on Machine Learning · Cited by 40 · see citing works
The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human PriorsPreprint
Guss, W. H., Castro, M. Y., Devlin, S., Houghton, B., Kuno, N. S., Loomis, C., Milani, S., Mohanty, S. P., Nakata, K., Salakhutdinov, R., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11071 · Cited by 13 · see citing works
Flagged first, then by citations
1 of the 20 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801 , 2016. · arXiv
- Stephanie Milani, Nicholay Topin, Brandon Houghton, William H Guss, Sharada P Mohanty, Keisuke Nakata, Oriol Vinyals, and Noboru Sean Kuno. Retrospective analysis of the 2019 minerl competition on sample efficient reinforcement learning. In NeurIPS 2019 Competition and Demonstration Track , pp. 203-214. PMLR, 2020.
- Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568 , 2019. · arXiv
Cited by
Sorted by citations
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124 · see citing works
“Scaling properties To investigate whether Dreamer can scale robustly, we train 6 model sizes ranging from 12M to 400M parameters, as well as different replay ratios on Crafter 50 and a DMLab task 43 .”
MethodSelf-citationA Survey of Generalisation in Deep Reinforcement LearningPreprint
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv (Cornell University) · Cited by 63 · see citing works
Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19 · see citing works
“Indeed, in the past few years a series of challenging PCG environments have been proposed (Juliani et al., 2019; Cobbe et al., 2020; Küttler et al., 2020; Samvelyan et al., 2021; Chevalier-Boisvert et al., 2018; Hafner, 2022; Deitke et al., 2022), mostly focusing on testing and improving generalisation in RL (Kirk et…”
BackgroundOut of the Cage: How Stochastic Parrots Win in Cyber Security Environments
Rigaki, M., Lukáš, O., Catania, C., & Garcia, S. (2024) · Proceedings of the 16th International Conference on Agents and Artificial Intelligence · doi:10.5220/0012391800003636 · Cited by 17 · see citing works
Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning
Matthews, M., Beukman, M., Ellis, B., Samvelyan, M., Jackson, M., Coward, S., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2402.16801 · Cited by 4 · see citing works
“Con-cretely, we first propose Craftax-Classic , a reimple-mentation of Crafter (Hafner, 2021) in JAX that runs 250 times faster than the Python-native original.”
BackgroundMethodInfluentialSurvival games for humans and machines
Strannegård, C., Engsner, N., Ulfsbäcker, S., Andreasson, S., Endler, J. A., & Nordgren, A. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101235 · Cited by 4 · see citing works
XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX
Nikulin, A., Kurenkov, V., Zisman, I., Agarkov, A., Sinii, V., & Kolesnikov, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1390 · Cited by 3 · see citing works
“This hierarchical structure is very similar to the way tasks are organized in the famous Minecraft (Guss et al., 2021) or the simpler Crafter (Hafner, 2021) benchmarks.”
MethodAdaptive Reinforcement Learning with LLM-augmented Reward FunctionsPreprint
Place, A. (2023) · doi:10.36227/techrxiv.24708198 · Cited by 3 · see citing works
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.13543 · Cited by 2 · see citing works
“( Hafner, 2021) A Minecraft-inspired grid environment where the player has to explore, gather resources and craft items to ensure their survival.”
Learning to Generalize With Object-Centric Agents in the Open World Survival Game Crafter
Stanić, A., Tang, Y., Ha, D., & Schmidhuber, J. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2023.3276849 · Cited by 3 · see citing works
A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1 · see citing works
“…(Chevalier-Boisvert et al., 2018; Samvelyan et al., 2021; K¨uttler et al., 2020; Juliani et al., 2019; Cobbe et al., 2020; Beattie et al., 2016; Hafner, 2021; Petrenko et al., 2021) or embodied AI tasks where the agent must generalize across different physical spaces (Savva et al., 2019; Shen…”
BackgroundSmartPlay: A Benchmark for LLMs as Intelligent Agents
Wu, Y., Tang, X., Mitchell, T. M., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.01557 · Cited by 6 · see citing works
“The agent also needs to collect different resources and craft more advanced weapons/tools to unlock more skills and achievements, while at the same time balance crafting goals with survival goals like maintaining health, thirst, food, and rest (Hafner, 2021).”
BackgroundMethodInfluentialKinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
Matthews, M., Beukman, M., Lu, C., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2410.23208 · Cited by 1 · see citing works
“…issues in agent training such as network capacity (Obando-Ceron et al., 2024), plasticity loss (Igl et al., 2020; Berariu et al., 2021; Sokar et al., 2023), lifelong learning (Kirkpatrick et al., 2017) and multi-task learning (Sodhani et al., 2021; Hafner, 2021; Benjamins et al., 2023).”
Task Relabelling for Multi-task Transfer using Successor Features
Balla, M., & Pérez-Liébana, D. (2022) · IEEE Conference on Games (CoG) · doi:10.1109/cog51982.2022.9893550 · Cited by 1 · see citing works
Policy Optimization with Augmented Value Targets for Generalization in Reinforcement Learning
Nafi, N. M., Poggi-Corradini, G., & Hsu, W. W. Y. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191507 · Cited by 1 · see citing works
A World Model for Actor–Critic in Reinforcement Learning
Panov, A. I., & Ugadiarov, L. (2023) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661823030379 · Cited by 1 · see citing works
Peng, S., Hu, X., Yi, Q., Zhang, R., Guo, J., Huang, D., Tian, Z., Chen, R., Du, Z., Guo, Q., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29376 · Cited by 5 · see citing works
Analyzing the Sensitivity to Policy-Value Decoupling in Deep Reinforcement Learning Generalization
Nafi, N. M., Ali, R. F., & Hsu, W. W. Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651435 · Cited by 1 · see citing works
Adaptive Reinforcement Learning with LLM-augmented Reward FunctionsPreprint
Place, A. (2023) · doi:10.36227/techrxiv.24708198.v1 · Cited by 1 · see citing works
The AI Hippocampus: How Far are We From Human Memory?
Jia, Z., Li, J., Kang, Y., Wang, Y., Wu, T., Wang, Q., Wang, X., Zhang, S., Shen, J., Li, Q., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.09113
Model Based Reinforcement Learning Pre-Trained with Various State Data
Ono, M., & Ichise, R. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00169
Environments for Automatic Curriculum Learning: A Short Survey
Nesterova, M. V., Skrynnik, A., & Panov, A. I. (2024) · Doklady Mathematics · doi:10.1134/s1064562424602099
A rational model of innovation by recombinationPreprint
Zhao, B., Vélez, N., & Griffiths, T. L. (2024) · doi:10.31234/osf.io/tzr9a
HierarchyCraft: A Benchmark Builder for Hierarchical Reasoning
Federico, M., Wang, S., Li, Y., & Taylor, M. E. (2026) · The Journal of Open Source Software · doi:10.21105/joss.06468
- Authors
- Hafner, D.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Hafner, Danijar
- ORCID
- Hafner, Danijar
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2109.06780
- OpenAlex
- W3201036908
- Cited by
- 16
- References
- 20
- Citations by year
- 2026: 1 · 2024: 7 · 2023: 6 · 2022: 1 · 2021: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- crafters, intrinsic rewards, unsupervised learning, long-term reasoning, exploration, visual input, reward maximization, agent capabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
