PreprintCited by 33Artificial Intelligence in GamesReinforcement Learning in RoboticsAI-based Problem Solving and Planning

Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

Open access
Zhu, Xizhou, Chen, Yuntao, Tian, Hao, Tao, Chenxin, Su, Weijie, Yang, Chenyu, Huang, Gao, Li, Bin +5 more·2023·arXiv

Abstract

The captivating realm of Minecraft has attracted substantial research interest in recent years, serving as a rich platform for developing intelligent agents capable of functioning in open-world environments. However, the current research landscape predominantly focuses on specific objectives, such as the popular "ObtainDiamond" task, and has not yet shown effective generalization to a broader spectrum of tasks. Furthermore, the current leading success rate for the "ObtainDiamond" task stands at around 20%, highlighting the limitations of Reinforcement Learning (RL) based controllers used in existing methods. To tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel framework integrates Large Language Models (LLMs) with text-based knowledge and memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These agents, equipped with the logic and common sense capabilities of LLMs, can skillfully navigate complex, sparse-reward environments with text-based interactions. We develop a set of structured actions and leverage LLMs to generate action plans for the agents to execute. The resulting LLM-based agent markedly surpasses previous methods, achieving a remarkable improvement of +47.5% in success rate on the "ObtainDiamond" task, demonstrating superior robustness compared to traditional RL-based controllers. Notably, our agent is the first to procure all items in the Minecraft Overworld technology tree, demonstrating its extensive capabilities. GITM does not need any GPU for training, but a single CPU node with 32 CPU cores is enough. This research shows the potential of LLMs in developing capable agents for handling long-horizon, complex tasks and adapting to uncertainties in open-world environments. See the project website at https://github.com/OpenGVLab/GITM.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.17144

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent works have developed more efficient reinforcement learning agents for robotics and embodied AI, focusing on enhancing autonomous agents' abilities for planning, reasoning, and collaboration in embodied environments.

    “Recent works have developed more efficient reinforcement learning agents for robotics and embodied artificial intelligence [16,38,78,131-138].The focus is on enhancing autonomous agents' abilities for planning, reasoning, and collaboration in embodied environments.”

    From A survey on large language model based autonomous agents (2024) · cites this paper as [16]

    The citation is attached to the claim about developing more efficient reinforcement-learning agents. The Introduction (pp. 2–3) instead describes RL agents as inefficient in Minecraft and proposes an LLM-based agent as an alternative; the Abstract (p. 1) likewise identifies GITM as an LLM-based framework, not an RL agent. (LLM, conf=0.87)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured
  • The reflection process can occur hierarchically, meaning that insights can be generated based on existing insights.

    “In addition, the reflection process can occur hierarchically, meaning that the insights can be generated based on existing insights.”

    From A survey on large language model based autonomous agents (2024) · cites this paper as [16]

    Section 3.2 (p. 6) says the planner creates new plans from summarized reference plans and adds successful sequences for future summarization. Section A.4.1 (p. 17) describes repeatedly summarizing sequences held in memory. This supports building on earlier summaries, but the paper does not describe that process as hierarchical reflection or explicitly say that insights are generated from existing insights. (LLM, conf=0.78)

    Experimental — accuracy not yet measured
  • Realworld simulation evaluates agents within immersive environments like games and interactive simulators, using metrics like task success rate and human similarity based on trajectories and completed objectives.

    “In this method, the agents are evaluated within immersive environments like games and interactive simulators.The agents are required to perform tasks autonomously, and then metrics like task success rate and human similarity are leveraged to evaluate the capability of the agents based on their trajectories and completed objectives [16,22,33,38,59,80,122,162,163,164].”

    From A survey on large language model based autonomous agents (2024) · cites this paper as [16]

    The Experiments section (p. 7) evaluates a Minecraft agent on item-collection tasks using success rate and coverage of the Overworld technology tree. It does not describe human similarity or trajectory-based metrics, or evaluation in interactive simulators beyond Minecraft. (LLM, conf=0.87)

    Experimental — accuracy not yet measured
  • By remembering historical information, memory can help to better decide when and how to make explorations, for example, focusing more on previously failed trials or actions with lower exploring frequencies.

    “By remembering historical information, the memory can help to better decide when and how to make explorations, for example, focusing more on previously failed trials or actions with lower exploring frequencies [93].”

    From A Survey on the Memory Mechanism of Large Language Model-based Agents (2025) · cites this paper as [93]

    Section 3.2 (p. 6) says text-based memory retains reference plans from prior experience to help the planner handle similar tasks more efficiently. But A.4.1 (p. 17) specifies that memory stores successful action sequences, not failed trials, and the paper does not describe using exploration frequency to choose actions. Section 4.3 (p. 9) attributes correction of previous mistakes to feedback rather than memory. (LLM, conf=0.88)

    Experimental — accuracy not yet measured
  • GITM intends to design agents in Minecraft, which can explore in complex and sparse-reward environments; the agents draw from the online Minecraft Wiki and craft recipes to provide an infinite source of knowledge for their navigation.

    “GITM [93] intends to design agents in Minecraft, which can explore in complex and sparse-reward environments. The agents draw from the online Minecraft Wiki and craft recipes to provide an infinite source of knowledge for their navigation.”

    From A Survey on the Memory Mechanism of Large Language Model-based Agents (2025) · cites this paper as [93]

    The Abstract (p. 1) says GITM aims to create Minecraft agents capable of navigating complex, sparse-reward environments. Section 3.1 (p. 5) says it uses the online Minecraft Wiki and crafting recipes as an exhaustive knowledge source for goal decomposition, but does not describe that source as infinite. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 19

  1. [1][1] A. Amiranashvili, N. Dorka, W. Burgard, V. Koltun, and T. Brox. Scaling imitation learning in minecraft. arXiv preprint arXiv:2007.02701 , 2020. - · arXiv
  2. [2][2] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems , 35:24639-24654, 2022. -
  3. [3][3] S. Cai, Z. Wang, X. Ma, A. Liu, and Y. Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. arXiv preprint arXiv:2301.10034 , 2023. - · arXiv
  4. [5][5] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.08853 , 2022. - · arXiv
  5. [6][6] W. H. Guss, S. Milani, N. Topin, B. Houghton, S. Mohanty, A. Melnik, A. Harter, B. Buschmaas, B. Jaster, C. Berganski, et al. Towards robust and domain agnostic reinforcement learning competitions: Minerl 2020. In NeurIPS 2020 Competition and Demonstration Track , pages 233-252. PMLR, 2021. -
  6. [9][9] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022. - · arXiv
  7. [11][11] I. Kanitscheider, J. Huizinga, D. Farhi, W. H. Guss, B. Houghton, R. Sampedro, P. Zhokhov, B. Baker, A. Ecoffet, J. Tang, et al. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2106.14876 , 2021. - · arXiv
  8. [14][14] Z. Lin, J. Li, J. Shi, D. Ye, Q. Fu, and W. Yang. Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:2112.04907 , 2021. - · arXiv
  9. [15][15] H. Mao, C. Wang, X. Hao, Y. Mao, Y. Lu, C. Wu, J. Hao, D. Li, and P. Tang. Seihai: A sample-efficient hierarchical ai for the minerl competition. In Distributed Artificial Intelligence: Third International Conference, DAI 2021, Shanghai, China, December 17-18, 2021, Proceedings 3 , pages 38-51. Springer, 2022. -
  10. [17][17] S. Milani, N. Topin, B. Houghton, W. H. Guss, S. P. Mohanty, K. Nakata, O. Vinyals, and N. S. Kuno. Retrospective analysis of the 2019 minerl competition on sample efficient reinforcement learning. In NeurIPS 2019 Competition and Demonstration Track , pages 203-214. PMLR, 2020. -
  11. [18][18] S. Milani, A. Kanervisto, K. Ramanauskas, S. Schulhoff, B. Houghton, S. Mohanty, B. Galbraith, K. Chen, Y. Song, T. Zhou, et al. Towards solving fuzzy tasks with human feedback: A retrospective of the minerl basalt 2022 competition. arXiv preprint arXiv:2303.13512 , 2023. - · arXiv
  12. [19][19] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023. - · arXiv
  13. [20][20] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 , 2023. - · arXiv
  14. [21][21] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302 , 2022. - · arXiv
  15. [23][23] O. E. L. Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg, M. Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808 , 2021. - · arXiv
  16. [24][24] C. Tessler, S. Givony, T. Zahavy, D. Mankowitz, and S. Mannor. A deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI conference on artificial intelligence , 2017. -
  17. [25][25] Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560 , 2023. - · arXiv
  18. [26][26] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. - · arXiv
  19. [28][28] H. Yuan, C. Zhang, H. Wang, F. Xie, P. Cai, H. Dong, and Z. Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16563 , 2023. · arXiv

Cited by

Sorted by citations

Authors
Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., Qiao, Y., Zhang, Z., & Dai, J.
Affiliations
  • Sensetime (China) (CN)
  • University of Science and Technology of China (CN)
  • Tsinghua University (CN)
  • Chinese University of Hong Kong (HK)
  • Shanghai Artificial Intelligence Laboratory (CN)
  • Chinese Academy of Sciences (CN)
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4378768661
Cited by
33
References
6
Citations by year
2026: 1 · 2025: 16 · 2024: 14 · 2023: 2
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
Minecraft, large language models, open-world environments, action planning, long-horizon tasks, LLM agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review