CoChat

Large Language Models and Video Games: A Preliminary Scoping Review

Open access
Sweetser, Penny · 2024 · ACM Conversational User Interfaces 2024
Cited by 50

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) hold interesting potential for the design, development, and research of video games. Building on the decades of prior research on generative AI in games, many researchers have sped to investigate the power and potential of LLMs for games. Given the recent spike in LLM-related research in games, there is already a wealth of relevant research to survey. In order to capture a snapshot of the state of LLM research in games, and to help lay the foundation for future work, we carried out an initial scoping review of relevant papers published so far. In this paper, we review 76 papers published between 2022 to early 2024 on LLMs and video games, with key focus areas in game AI, game development, narrative, and game research and reviews. Our paper provides an early state of the field and lays the groundwork for future research and reviews on this topic.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3640794.3665582

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 73

  1. 10.18653/v1/2023.findings-emnlp.151 · DOI
  2. 10.34190/ecgbl.17.1.1709 · DOI
  3. 10.1145/3544548.3581441 · DOI · Paper page
  4. Maren Awiszus, Frederik Schubert, and Bodo Rosenhahn. 2022. Wor (l) d-GAN: Towards Natural Language Based PCG in Minecraft. IEEE Transactions on Games (2022).
  5. Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yancheng Luo, Fuli Feng, Xiangnaan He, and Qi Tian. 2023. A bi-step grounding paradigm for large language models in recommendation systems. arXiv preprint arXiv:2308.08434 (2023). · arXiv
  6. 10.1007/s10846-023-01991-3 · DOI
  7. Jake Brawer, Kayleigh Bishop, Bradley Hayes, and Alessandro Roncone. 2023. Towards A Natural Language Interface for Flexible Multi-Agent Task Assignment. arXiv preprint arXiv:2311.00153 (2023). · arXiv
  8. M Charity, Yash Bhartia, Daniel Zhang, Ahmed Khalifa, and Julian Togelius. 2023. A Preliminary Study on a Conceptual Game Feature Generation and Recommendation System. arXiv preprint arXiv:2308.13538 (2023). · arXiv
  9. 10.1109/FIE58773.2023.10343021 · DOI
  10. Cédric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Côté. 2023. Augmenting Autotelic Agents with Large Language Models. arXiv preprint arXiv:2305.12487 (2023). · arXiv
  11. Maximilian Croissant, Madeleine Frister, Guy Schofield, and Cade McCall. 2023. An Appraisal-Based Chain-Of-Emotion Architecture for Affective Language Model Game Agents. arXiv preprint arXiv:2309.05076 (2023). · arXiv
  12. Lajos Matyas Csepregi. 2021. The Effect of Context-aware LLM-based NPC Conversations on Player Engagement in Role-playing Video Games. Unpublished manuscript (2021).
  13. Stefano De Paoli. 2023. Performing an Inductive Thematic Analysis of Semi-Structured Interviews With a Large Language Model: An Exploration and Provocation on the Limits of the Approach. Social Science Computer Review (2023), 08944393231220483.
  14. Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. 2023. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692 (2023). · arXiv
  15. 10.1145/3402942.3402945 · DOI
  16. Santiago Góngora, Luis Chiruzzo, Gonzalo Méndez, and Pablo Gervás. 2023. Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-Playing Games. In International Conference on Games and Learning Alliance. Springer, 277–288.
  17. Cong Guan, Lichao Zhang, Chunpeng Fan, Yichen Li, Feng Chen, Lihe Li, Yunjia Tian, Lei Yuan, and Yang Yu. 2023. Efficient Human-AI Coordination via Preparatory Language-based Convention. arXiv preprint arXiv:2311.00416 (2023). · arXiv
  18. Mustafa Can Gursesli, Pittawat Taveekitworachai, Febri Abdullah, Mury F Dewantoro, Antonio Lanata, Andrea Guazzini, Van Khôi Lê, Adrien Villars, and Ruck Thawonmas. 2023. The Chronicles of ChatGPT: Generating and Evaluating Visual Novel Narratives on Climate Change Through ChatGPT. In International Conference on Interactive Digital Storytelling. Springer, 181–194.
  19. 10.36227/techrxiv.23589741.v4 · DOI · Paper page
  20. 10.1145/3490100.3516458 · DOI
  21. 10.1109/CoG57401.2023.10333153 · DOI
  22. 10.17083/ijsg.v10i4.645 · DOI
  23. Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. 2023. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00166 (2023). · arXiv
  24. Vikram Kumaran, Dan Carpenter, Jonathan Rowe, Bradford Mott, and James Lester. 2023. End-to-End Procedural Level Generation in Educational Games with Natural Language Instruction. In 2023 IEEE Conference on Games (CoG). IEEE, 1–8.
  25. Michael Lankes and Andreas Stöckl. 2023. Game Reviews Reviewed: A Game Designer’s Perspective on AI-generated Game Review Analyses. In 2023 IEEE Conference on Games (CoG). IEEE, 1–8.
  26. Pier Luca Lanzi and Daniele Loiacono. 2023. Chatgpt and other large language models as evolutionary engines for online interactive collaborative game design. arXiv preprint arXiv:2303.02155 (2023). · arXiv
  27. 10.18653/v1/2023.nlp4convai-1.11 · DOI
  28. Lane Lawley and Christopher J MacLellan. 2023. VAL: Interactive Task Learning with GPT Dialog Parsing. arXiv preprint arXiv:2310.01627 (2023). · arXiv
  29. Yuxuan Lei, Jianxun Lian, Jing Yao, Xu Huang, Defu Lian, and Xing Xie. 2023. RecExplainer: Aligning Large Language Models for Recommendation Model Interpretability. arXiv preprint arXiv:2311.10947 (2023). · arXiv
  30. Yinghao Li, Haorui Wang, and Chao Zhang. 2023. Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study. arXiv preprint arXiv:2311.07387 (2023). · arXiv
  31. Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, 2023. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310.19773 (2023). · arXiv
  32. Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. 2023. LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination. arXiv preprint arXiv:2312.15224 (2023). · arXiv
  33. Weiyu Ma, Qirui Mi, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. 2023. Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach. arXiv preprint arXiv:2312.11865 (2023). · arXiv
  34. 10.1109/CoG57401.2023.10333244 · DOI
  35. Muhammad U Nasir and Julian Togelius. 2023. Practical PCG Through Large Language Models. arXiv preprint arXiv:2305.18243 (2023). · arXiv
  36. 10.1145/3628454.3629551 · DOI
  37. Kolby Nottingham, Yasaman Razeghi, Kyungmin Kim, JB Lanier, Pierre Baldi, Roy Fox, and Sameer Singh. 2023. Selective Perception: Learning Concise State Descriptions for Language Model Actors. In NeurIPS 2023 Foundation Models for Decision Making Workshop.
  38. Kolby Nottingham, Yasaman Razeghi, Kyungmin Kim, JB Lanier, Pierre Baldi, Roy Fox, and Sameer Singh. 2023. Selective perception: Optimizing state descriptions with reinforcement learning for language model actors. arXiv preprint arXiv:2307.11922 (2023). · arXiv
  39. 10.5220/0012060500003538 · doi:10.5220/0012060500003538
  40. Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. 2023. diff History for Long-Context Language Agents. arXiv preprint arXiv:2312.07540 (2023). · arXiv
  41. Angela Ramirez, Mamon Alsalihy, Kartik Aggarwal, Cecilia Li, Liren Wu, and Marilyn Walker. 2023. Controlling Personality Style in Dialogue with Zero-Shot Prompt-Based Learning. arXiv preprint arXiv:2302.03848 (2023). · arXiv
  42. Jianqiang Ren, Chao He, Lin Liu, Jiahao Chen, Yutong Wang, Yafei Song, Jianfang Li, Tangli Xue, Siqi Hu, Tao Chen, 2023. Make-A-Character: High Quality Text-to-3D Character Generation within Minutes. arXiv preprint arXiv:2312.15430 (2023). · arXiv
  43. Jasmine Roberts, Andrzej Banburski-Fahey, and Jaron Lanier. 2022. Steps towards prompt-based creation of virtual worlds. arXiv preprint arXiv:2211.05875 (2022). · arXiv
  44. Jasmine Roberts, Andrzej Banburski-Fahey, and Jaron Lanier. 2022. Surreal vr pong: Llm approach to game design. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022). https://www. microsoft. com/en-us/research/publication/surreal-vr-pong-llm-approach-to-gamedesign.
  45. Kenji Saito, Kayo Kobayashi, Waki Takekoshi, Atsuki Hashimoto, Nobukazu Hirai, Akifumi Kimura, Asuka Takahashi, Naoki Yoshioka, and Asuto Mano. 2023. Double Impact: Children’s Serious RPG Generation/Play with a Large Language Model for Their Deeper Engagement in Social Issues. In Joint International Conference on Serious Games. Springer, 274–289.
  46. Ahad Shams, Douglas Summers-Stay, Vsevolod Metelsky, Arpan Tripathi, and Karan Malhotra. 2023. Towards a Holodeck-style Simulation Game. arXiv preprint arXiv:2308.13548 (2023). · arXiv
  47. Yubo Shu, Hansu Gu, Peng Zhang, Haonan Zhang, Tun Lu, Dongsheng Li, and Ning Gu. 2023. RAH! RecSys-Assistant-Human: A Human-Central Recommendation Framework with Large Language Models. arXiv preprint arXiv:2308.09904 (2023). · arXiv
  48. 10.1145/3583133.3590656 · DOI
  49. Shyam Sudhakaran Miguel González-Duque Claire Glanois Matthias Freiberger Elias Najarro and Sebastian Risi. 2023. MarioGPT: Open-Ended Text2Level Generation through Large Language Models. arxiv:2302.05981 [cs.AI] · arXiv
  50. Yuqian Sun, Hanyi Wang, Pok Man Chan, Morteza Tabibi, Yan Zhang, Huan Lu, Yuheng Chen, Chang Hee Lee, and Ali Asadipour. 2023. Fictional Worlds, Real Connections: Developing Community Storytelling Social Chatbots through LLMs. arXiv preprint arXiv:2309.11478 (2023). · arXiv
  51. Zhu Sun, Hongyang Liu, Xinghua Qu, Kaidong Feng, Yan Wang, and Yew-Soon Ong. 2023. Large Language Models for Intent-Driven Session Recommendations. arXiv preprint arXiv:2312.07552 (2023). · arXiv
  52. Arina Svetasheva and Keeheon Lee. 2024. Harnessing Large Language Models for Effective and Efficient Hate Speech Detection. (2024). · DOI
  53. Mohammad Reza Taesiri, Tianjun Feng, Cor-Paul Bezemer, and Anh Nguyen. 2023. GlitchBench: Can large multimodal models detect video game glitches?arXiv preprint arXiv:2312.05291 (2023). · arXiv
  54. Mohammad Reza Taesiri, Finlay Macklon, Yihe Wang, Hengshuo Shen, and Cor-Paul Bezemer. 2022. Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors. arXiv preprint arXiv:2210.02506 (2022). · arXiv
  55. Pittawat Taveekitworachai Febri Abdullah Mury F. Dewantoro Ruck Thawonmas Julian Togelius and Jochen Renz. 2023. ChatGPT4PCG Competition: Character-like Level Generation for Science Birds. arxiv:2303.15662 [cs.AI] · DOI
  56. Pittawat Taveekitworachai, Febri Abdullah, Mustafa Can Gursesli, Mury F Dewantoro, Siyuan Chen, Antonio Lanata, Andrea Guazzini, and Ruck Thawonmas. 2023. What Is Waiting for Us at the End? Inherent Biases of Game Story Endings in Large Language Models. In International Conference on Interactive Digital Storytelling. Springer, 274–284.
  57. 10.1109/ICCE-Berlin58801.2023.10375663 · DOI
  58. Andrea Tinterri Marilena di Padova Francesco Palladino Giordano Vignoli Anna Dipace 2024. AI in board Game-Based Learning. In CEUR WORKSHOP PROCEEDINGS.
  59. 10.1145/3582437.3587211 · DOI · Paper page
  60. Susanna Värtinen, Perttu Hämäläinen, and Christian Guckelsberger. 2022. Generating role-playing game quests with gpt language models. IEEE Transactions on Games (2022).
  61. Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. 2023. Preference Proxies: Evaluating Large Language Models in capturing Human Preferences in Human-AI Tasks. In ICML 2023 Workshop The Many Facets of Preference-Based Learning.
  62. Markos Viggiato and Cor-Paul Bezemer. 2023. Leveraging the OPT Large Language Model for Sentiment Analysis of Game Reviews. IEEE Transactions on Games (2023).
  63. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An Open-Ended Embodied Agent with Large Language Models. arxiv:2305.16291 [cs.AI] · arXiv
  64. Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li. 2023. SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning. arXiv preprint arXiv:2305.15486 (2023). · arXiv
  65. Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2023. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940 (2023). · arXiv
  66. Xue Yan, Yan Song, Xinyu Cui, Filippos Christianos, Haifeng Zhang, David Henry Mguni, and Jun Wang. 2023. Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Models. arXiv preprint arXiv:2310.18127 (2023). · arXiv
  67. Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, 2023. Octopus: Embodied vision-language programmer from environmental feedback. arXiv preprint arXiv:2310.08588 (2023). · arXiv
  68. Zhengyi Yang, Jiancan Wu, Yanchen Luo, Jizhi Zhang, Yancheng Yuan, An Zhang, Xiang Wang, and Xiangnan He. 2023. Large language model can interpret latent space of sequential recommender. arXiv preprint arXiv:2310.20487 (2023). · arXiv
  69. 10.1007/978-3-031-47655-6_24 · DOI
  70. Lei Yue and Liang Guo. 2023. Combine DGBL With AI System: A Technical Guidance to Reduce Teacher’s Burden in Digital Game-Based Learning. In European Conference on Games Based Learning. Academic Conferences International Limited, 826–XXV.
  71. Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2023. Recommendation as instruction following: A large language model empowered recommendation approach. arXiv preprint arXiv:2305.07001 (2023). · arXiv
  72. Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Adapting large language models by integrating collaborative semantics for recommendation. arXiv preprint arXiv:2311.09049 (2023). · arXiv
  73. Andrew Zhu, Karmanya Aggarwal, Alexander Feng, Lara J Martin, and Chris Callison-Burch. 2023. Fireball: A dataset of dungeons and dragons actual-play with structured game state information. arXiv preprint arXiv:2305.01528 (2023). · arXiv

Cited by

Metadata

Authors
Sweetser, P.
Venue
ACM Conversational User Interfaces 2024
Publisher
ACM
Published
2024-07-08
Pages
1-8
OpenAlex
W4400397496
Cited by
50
References
17
FWCI
15.21
Type
conference paper
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.