The DOI matches its DataCite record.
Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions
Open accessAbstract
Large language models (LLMs) such as ChatGPT and GPT-4 have recently demonstrated their remarkable abilities of communicating with human users. In this technical report, we take an initiative to investigate their capacities of playing text games, in which a player has to understand the environment and respond to situations by having dialogues with the game world. Our experiments show that ChatGPT performs competitively compared to all the existing systems but still exhibits a low level of intelligence. Precisely, ChatGPT can not construct the world model by playing the game or even reading the game manual; it may fail to leverage the world knowledge that it already has; it cannot infer the goal of each step as the game progresses. Our results open up new research questions at the intersection of artificial intelligence, machine learning, and natural language processing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 14
- Ammanabrolu, P. and Hausknecht, M. Graph constrained reinforcement learning for natural language action spaces. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
- Arulkumaran, K., Cully, A., and Togelius, J. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion, 2019. · DOI
- Bender, E. M. and Koller, A. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5185-5198, 2020.
- Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
- Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. · arXiv
- Brown, N. and Sandholm, T. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 2018. · DOI
- Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
- Guo, X., Yu, M., Gao, Y., Gan, C., Campbell, M., and Chang, S. Interactive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
- Ha, D. and Schmidhuber, J. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
- Hausknecht, M., Loynd, R., Yang, G., Swaminathan, A., and Williams, J. D. Nail: A general interactive fiction agent. arXiv preprint arXiv:1902.04259, 2019. · arXiv
- Hausknecht, M., Ammanabrolu, P., Côté, M.-A., and Yuan, X. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
- He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- Moravčík, M., Schmid, M., Burch, N., Lisỳ, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowling, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 2017. · DOI
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
Cited by
Sorted by citations
Player-Driven Emergence in LLM-Driven Game Narrative
Peng, X., Quaye, J., Rao, S., Xu, W., Botchway, P., Brockett, C., Jojic, N., DesGarennes, G., Lobb, K., Xu, M., et al. (2024) · 2024 IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645607 · Cited by 27
AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents
Anokhin, P., Semenov, N., Sorokin, A., Evseev, D., Kravchenko, A., Burtsev, M., & Burnaev, E. (2024) · arXiv · doi:10.48550/arxiv.2407.04363 · Cited by 8
Metadata
- Authors
- Tsai, C. F., Zhou, X., Liu, S. S., Li, J., Yu, M., & Mei, H.
- Affiliations
- Toyota Technological Institute at Chicago (US)
- New Jersey Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.02868
- OpenAlex
- W4362731048
- Cited by
- 9
- References
- 0
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, game playing, ChatGPT, GPT-4, goal inference, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions. https://app.cochat.ai/papers/10.48550/arxiv.2304.02868
- The paper
- Tsai, C. F., Zhou, X., Liu, S. S., Li, J., Yu, M., & Mei, H. (2023). Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions. arXiv. https://doi.org/10.48550/arxiv.2304.02868
