CoChat
Sign Up
Cited by 137Artificial Intelligence in GamesTopic ModelingArtificial Intelligence Applications

The DOI matches its Crossref record; the references it cites are not checked yet.

Large Language Models and Games: A Survey and Roadmap

Open access
Gallotta, Roberto, Todd, Graham, Zammit, Marvin, Earle, Sam, Liapis, Antonios, Togelius, Julian, Yannakakis, Georgios N.·2024·IEEE Transactions on Games
DOI 10.1109/tg.2024.3461510
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent years have seen an explosive increase in research on large language models (LLMs), and accompanying public engagement on the topic. While starting as a niche area within natural language processing, LLMs have shown remarkable potential across a broad range of applications and domains, including games. This paper surveys the current state of the art across the various applications of LLMsinandforgames, and identifies the different roles LLMs can take within a game. Importantly, we discuss underexplored areas and promising directions for future uses of LLMs in games and we reconcile the potential and limitations of LLMs within the games domain. As the first comprehensive survey and roadmap at the intersection of LLMs and games, we are hopeful that this paper will serve as the basis for groundbreaking research and innovation in this exciting new field.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tg.2024.3461510

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs can generate dynamic and contextually rich dialogues for NPCs.

    “Roberto Gallotta et al., in “Large Language Models and Games: A Survey and Roadmap” demonstrated how LLMs can generate dynamic and contextually rich dialogues for NPCs [10].”

    From MemoryRepository for AI NPC (2024) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 137

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 167

  1. [1][1] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , 'Language models are unsupervised multitask learners,' OpenAI blog , vol. 1, no. 8, 2019. -
  2. [2][2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'BERT: Pretraining of deep bidirectional transformers for language understanding,' in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics , 2019. -
  3. [3][3] Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu, 'ERNIE: Enhanced language representation with informative entities,' in Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2019. -
  4. [4][4] T. Kaufmann, P. Weng, V. Bengs, and E. H¨ ullermeier, 'A survey of reinforcement learning from human feedback,' arXiv preprint arXiv:2312.14925 , 2023. - · arXiv
  5. [5][5] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang, 'Sparks of artificial general intelligence: Early experiments with GPT-4,' arXiv preprint arXiv:2303.12712 , 2023. - · arXiv
  6. [6][6] R. T. McCoy, S. Yao, D. Friedman, M. Hardy, and T. L. Griffiths, 'Embers of autoregression: Understanding large language models through the problem they are trained to solve,' arXiv preprint arXiv:2309.13638 , 2023. - · arXiv
  7. [7][7] M. T. Hicks, J. Humphries, and J. Slater, 'Correction: ChatGPT is bullshit,' Ethics and Information Technology , vol. 26, no. 46, 2024. -
  8. [8][8] G. N. Yannakakis and J. Togelius, Artificial Intelligence and Games . Springer, 2018. -
  9. [10][10] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, 'Large language models: A survey,' arXiv preprint arXiv:2402.06196 , 2024. - · arXiv
  10. [11][11] P. Garc´ ıa-S´ anchez, A. M. Garc´ ıa, P. A. Castillo, and I. J. P´ erez, 'A bibliometric study of the research area of videogames using Dimensions.ai database,' in Proceedings of the International Conference on Information Technology and Quantitative Management , 2019. -
  11. [12][12] A. Liapis, '10 years of the PCG workshop: Past and future trends,' in Proceedings of the FDG Workshop on Procedural Content Generation , 2020. -
  12. [13][13] A. Jordanous and B. Keller, 'Modelling creativity: Identifying key components through a corpus-based approach,' Public Library of Science , vol. 11, pp. 1-27, 2016. -
  13. [14][14] D. Yang, E. Kleinman, and C. Harteveld, 'GPT for games: A scoping review (2020-2023),' in Proceedings of the IEEE Conference on Games , 2024. -
  14. [15][15] C. E. Shannon, 'A mathematical theory of communication,' The Bell system technical journal , vol. 27, no. 3, 1948. -
  15. [16][16] J. L. Elman, 'Finding structure in time,' Cognitive science , vol. 14, no. 2, 1990. -
  16. [17][17] S. Hochreiter and J. Schmidhuber, 'Long short-term memory,' Neural Computation , vol. 9, pp. 1735-1780, 1997. -
  17. [18][18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, 'Attention is all you need,' in Proceedings of the Conference on Neural Information Processing Systems , 2017. -
  18. [21][21] Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, 'The dawn of LMMs: Preliminary explorations with GPT4V(ision),' arXiv preprint arXiv:2309.17421 , 2023. - · arXiv
  19. [22][22] H. Liu, C. Li, Q. Wu, and Y. J. Lee, 'Visual instruction tuning,' in Proceedings of the Neural Information Processing Systems Conference , 2023. -
  20. [23][23] A. Summerville, S. Snodgrass, M. Guzdial, C. Holmg˚ ard, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius, 'Procedural content generation via machine learning (PCGML),' IEEE Transactions on Games , vol. 10, no. 3, 2018. -
  21. [25][25] D. Noever, M. Ciolino, and J. Kalin, 'The chess transformer: Mastering play using generative language models,' arXiv preprint arXiv:2008.04057 , 2020. - · arXiv
  22. [26][26] A. St¨ ockl, 'Watching a language model learning chess,' in Proceedings of the Recent Advances in Natural Language Processing International Conference , 2021. -
  23. [27][27] S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel, 'Chess as a testbed for language model state tracking,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 10, 2022. -
  24. [28][28] M. Ciolino, J. Kalin, and D. Noever, 'The Go transformer: Natural language modeling for game play,' in Proceedings of the Artificial Intelligence for Industries International Conference , 2020. -
  25. [29][29] K. Li, A. K. Hopkins, D. Bau, F. Vi´ egas, H. Pfister, and M. Wattenberg, 'Emergent world representations: Exploring a sequence model trained on a synthetic task,' in Proceedings of the International Conference on Learning Representations , 2023. -
  26. [30][30] B. Bateni and J. Whitehead, 'Language-driven play: Large language models as game-playing agents in Slay the Spire,' in Proceedings of the International Conference on the Foundations of Digital Games , 2024. -
  27. [31][31] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg et al. , 'A generalist agent,' arXiv preprint arXiv:2205.06175 , 2022. - · arXiv
  28. [32][32] M. Xu, W. Dai, C. Liu, X. Gao, W. Lin, G.-J. Qi, and H. Xiong, 'Spatial-temporal transformer networks for traffic flow forecasting,' arXiv preprint arXiv:2001.02908 , 2020. - · arXiv
  29. [33][33] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, 'MaskGIT: Masked generative image transformer,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022. -
  30. [34][34] S. Yao, R. Rao, M. Hausknecht, and K. Narasimhan, 'Keep CALM and explore: Language models for action generation in text-based games,' in Proceedings of the Empirical Methods in Natural Language Processing Conference , 2020. -
  31. [35][35] C. F. Tsai, X. Zhou, S. S. Liu, J. Li, M. Yu, and H. Mei, 'Can large language models play text games well? current state-of-the-art and open questions,' arXiv preprint arXiv:2304.02868 , 2023. - · arXiv
  32. [36][36] P. Ammanabrolu, L. Jiang, M. Sap, H. Hajishirzi, and Y. Choi, 'Aligning to social norms and values in interactive narratives,' in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2022. -
  33. [37][37] M. Hausknecht, P. Ammanabrolu, M.-A. Cˆ ot´ e, and X. Yuan, 'Interactive fiction games: A colossal adventure,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2020. -
  34. [38][38] M. Klissarov, P. D'Oro, S. Sodhani, R. Raileanu, P.-L. Bacon, P. Vincent, A. Zhang, and M. Henaff, 'Motif: Intrinsic motivation from artificial intelligence feedback,' arXiv preprint arXiv:2310.00166 , 2023. - · arXiv
  35. [39][39] Meta Fundamental AI Research Diplomacy Team (FAIR), A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu et al. , 'Human-level play in the game of diplomacy by combining language models with strategic reasoning,' Science , vol. 378, no. 6624, 2022. -
  36. [40][40] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, 'Voyager: An open-ended embodied agent with large language models,' in Proceedings of the NeurIPS Workshop on Foundation Models for Decision Making , 2023. -
  37. [42][42] M. C ¸ . Uluda˘ glı and K. O˘ guz, 'Non-player character decisionmaking in computer games,' Artificial Intelligence Review , vol. 56, no. 12, 2023. -
  38. [43][43] M. Shanahan, K. McDonell, and L. Reynolds, 'Role play with large language models,' Nature , vol. 623, 2023. -
  39. [44][44] F. Gao, K. Fang, and W. K. Victor Chan, 'Chemical life: Knowledge-based personality, emotion and action cues in edu- 23. cational games,' in Proceedings of the IEEE Conference on Games , 2023. -
  40. [45][45] J. S. Park, J. O'Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, 'Generative agents: Interactive simulacra of human behavior,' in Proceedings of the Annual ACM Symposium on User Interface Software and Technology , 2023. -
  41. [46][46] M. Kreminski, 'Toward better gossip simulation in emergent narrative systems,' in Proceedings of the IEEE Conference on Games , 2023. -
  42. [47][47] M. M¨ uller-Brockhausen, G. Barbero, and M. Preuss, 'Chatter generation through language models,' in Proceedings of the IEEE Conference on Games , 2023. -
  43. [49][49] H. Warpefelt and H. Verhagen, 'A model of non-player character believability,' Journal of Gaming & Virtual Worlds , vol. 9, no. 1, 2017. -
  44. [50][50] J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu, 'Shifting attention to relevance: Towards the uncertainty estimation of large language models,' arXiv preprint arXiv:2307.01379 , 2023. - · arXiv
  45. [51][51] A. Mehta, Y. Kunjadiya, A. Kulkarni, and M. Nagar, 'Exploring the viability of conversational AI for non-playable characters: A comprehensive survey,' in Proceedings of the International Conference on Recent Trends in Computer Science and Technology , 2022. -
  46. [52][52] E. Akata, L. Schulz, J. Coda-Forno, S. J. Oh, M. Bethge, and E. Schulz, 'Playing repeated games with large language models,' arXiv preprint arXiv:2305.16867 , 2023. - · arXiv
  47. [53][53] Y. Xu, S. Wang, P. Li, F. Luo, X. Wang, W. Liu, and Y. Liu, 'Exploring large language models for communication games: An empirical study on Werewolf,' arXiv preprint arXiv:2309.04658 , 2023. - · arXiv
  48. [55][55] J. Urbanek, A. Fan, S. Karamcheti, S. Jain, S. Humeau, E. Dinan, T. Rockt¨ aschel, D. Kiela, A. Szlam, and J. Weston, 'Learning to speak and act in a fantasy text adventure game,' in Proceedings of the Empirical Methods in Natural Language Processing Conference and the Natural Language Processing International Joint Conference , 2019. -
  49. [56][56] S. Hu, Z. Huang, C. Hu, and J. Liu, '3D building generation in Minecraft via large language models,' in Proceedings of the IEEE Conference on Games , 2024. -
  50. [57][57] T. Rist, 'Using a large language model to turn explorations of virtual 3d-worlds into interactive narrative experiences,' in Proceedings of the IEEE Conference on Games , 2024. -
  51. [58][58] M. P. Eladhari, 'Re-tellings: The fourth layer of narrative as an instrument for critique,' in Proceedings of the Interactive Digital Storytelling International Conference , 2018. -
  52. [59][59] M. Guzdial, S. Shah, and M. Riedl, 'Towards automated let's play commentary,' in Proceedings of the AIIDE workshop on Experimental AI in Games , 2018. -
  53. [60][60] T. Ishigaki, G. Topi´ c, Y. Hamazono, H. Noji, I. Kobayashi, Y. Miyao, and H. Takamura, 'Generating racing game commentary from vision, language, and structured data,' in Proceedings of the Natural Language Generation International Conference , 2021. -
  54. [61][61] C. Li, S. Gandhi, and B. Harrison, 'End-to-end let's play commentary generation using multi-modal video representations,' in Proceedings of the International Conference on the Foundations of Digital Games , 2019. -
  55. [62][62] N. Renella and M. Eger, 'Towards automated video game commentary using generative AI,' in Proceedings of the AIIDE workshop on Experimental AI in Games , 2023. -
  56. [64][64] L. Cheng, X. Li, and L. Bing, 'Is GPT-4 a good data analyst?' in Findings of the Association for Computational Linguistics , 2023. -
  57. [65][65] P. Maddigan and T. Susnjak, 'Chat2VIS: Generating data visualisations via natural language using ChatGPT, Codex and GPT-3 large language models,' IEEE Access , vol. 11, 2023. -
  58. [66][66] M. S. El-Nasr, A. Drachen, and A. Canossa, Game Analytics: Maximizing the Value of Player Data 2013th Edition . Springer, 2013. -
  59. [67][67] Z. Cheng, T. Xie, P. Shi, C. Li, R. Nadkarni, Y. Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu, 'Binding language models in symbolic languages,' in Proceedings of the International Conference on Learning Representations , 2023. -
  60. [68][68] Z. Chen, H. Mao, H. Li, W. Jin, H. Wen, X. Wei, S. Wang, D. Yin, W. Fan, H. Liu, and J. Tang, 'Exploring the potential of large language models (LLMs) in learning on graph,' in Proceedings of the NeurIPS Workshop on New Frontiers in Graph Learning , 2023. -
  61. [69][69] T. Wang, M. Honari-Jahromi, S. Katsarou, O. Mikheeva, T. Panagiotakopoulos, S. Asadi, and O. Smirnov, 'player2vec: A language modeling approach to understand player behavior in games,' arXiv preprint arXiv:2404.04234 , 2024. - · arXiv
  62. [70][70] I. Beltagy, M. E. Peters, and A. Cohan, 'Longformer: The longdocument transformer,' arXiv preprint arXiv:2004.05150 , 2020. - · arXiv
  63. [71][71] A. Drachen, A. Canossa, and G. N. Yannakakis, 'Player modeling using self-organization in Tomb Raider: Underworld,' in Proceedings of the IEEE Symposium on Computational Intelligence and Games , 2009. -
  64. [72][72] A. Canossa, J. B. Martinez, and J. Togelius, 'Give me a reason to dig: Minecraft and psychology of motivation,' in Proceedings of the IEEE Conference on Computational Intelligence and Games , 2013. -
  65. [73][73] N. Rasajski, C. Trivedi, K. Makantasis, A. Liapis, and G. N. Yannakakis, 'BehAVE: Behaviour alignment of video game encodings,' in Proceedings of the ECCV Workshop on Computer Vision For Videogames , 2024. -
  66. [74][74] J. Togelius and G. N. Yannakakis, 'General general game AI,' in Proceedings of the IEEE Conference on Computational Intelligence and Games , 2016. -
  67. [75][75] A. Tychsen, M. Hitchens, T. Brolund, and M. Kavakli, 'The game master,' in Proceedings of the Australasian conference on Interactive entertainment , 2005. -
  68. [76][76] A. Liapis and A. Denisova, 'The challenge of evaluating player experience in tabletop role-playing games,' in Proceedings of the International Conference on the Foundations of Digital Games , 2023. -
  69. [77][77] M. Hua and R. Raley, 'Playing with unicorns: AI dungeon and citizen NLP,' Digital Humanities Quarterly , vol. 14, no. 4, 2020. -
  70. [78][78] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022. -
  71. [79][79] X. You, P. Taveekitworachai, S. Chen, M. C. Gursesli, X. Li, Y. Xia, and R. Thawonmas, 'Dungeons, Dragons, and Emotions: A preliminary study of player sentiment in LLM-driven TTRPGs,' in Proceedings of the International Conference on the Foundations of Digital Games , 2024. -
  72. [80][80] A. Zhu, L. Martin, A. Head, and C. Callison-Burch, 'CALYPSO: LLMs as Dungeon Masters' assistants,' in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , 2023. -
  73. [81][81] J. Kelly, M. Mateas, and N. Wardrip-Fruin, 'Towards computational support with language models for TTRPG game masters,' in Proceedings of the FDG Workshop on Human-AI Interaction through Play , 2023. -
  74. [82][82] D. Acharya, J. Kelly, W. Tate, M. Joslyn, M. Mateas, and N. Wardrip-Fruin, 'Shoelace: A storytelling assistant for GUMSHOE One-2-One,' in Proceedings of the International Conference on the Foundations of Digital Games , 2023. -
  75. [83][83] Y. Sun, Z. Li, K. Fang, C. H. Lee, and A. Asadipour, 'Language as reality: A co-creative storytelling game experience in 1001 Nights using generative AI,' in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , 2023. -
  76. [84][84] M. Treanor, A. Zook, M. P. Eladhari, J. Togelius, G. Smith, M. Cook, T. Thompson, B. Magerko, J. Levine, and A. Smith, 'AIbased game design patterns,' in Proceedings of the International Conference on the Foundations of Digital Games , 2015. -
  77. [85][85] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu, 'Jailbreaking ChatGPT via prompt engineering: An empirical study,' arXiv preprint arXiv:2305.13860 , 2023. - · arXiv
  78. [86][86] L. Huang and X. Sun, 'Create ice cream: Real-time creative element synthesis framework based on GPT3.0,' in Proceedings of the IEEE Conference on Games , 2023. -
  79. [88][88] A. J. Summerville, S. Snodgrass, M. Mateas, and S. Ontanon, 'The VGLC: The video game level corpus,' in Proceedings of the FDG Workshop on Procedural Content Generation , 2016. -
  80. [89][89] A. Summerville and M. Mateas, 'Super Mario as a string: Platformer level generation via LSTMs,' in Proceedings of the Joint Conference of DIGRA and FDG , 2016. -
  81. [90][90] Y. Zakaria, M. Fayek, and M. Hadhoud, 'Procedural level generation for Sokoban via deep learning: An experimental study,' IEEE Transactions on Games , vol. 15, no. 1, 2022. -
  82. [91][91] G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, 'Level generation through large language models,' in Proceedings of the International Conference on the Foundations of Digital Games , 2023. -
  83. [92][92] C. Bamford, 'Griddly: A platform for AI research in games,' Software Impacts , vol. 8, 2021. -
  84. [93][93] S. Sudhakaran, M. Gonz´ alez-Duque, M. Freiberger, C. Glanois, E. Najarro, and S. Risi, 'MarioGPT: Open-ended text2level generation through large language models,' Advances in Neural Information Processing Systems , vol. 36, 2024. -
  85. [94][94] B. Lyman, A. Ebrahimi, J. E. C. III, S. Chan, C. Barney, and B. D. Schutter, 'Cardistry: Exploring a GPT model workflow as an adapted method of gaminiscing,' in Proceedings of the International Conference on the Foundations of Digital Games , 2024. -
  86. [96][96] T. Merino, S. Earle, R. Sudhakaran, S. Sudhakaran, and J. Togelius, 'Making new connections: Llms as puzzle generators for the new york times' connections word game,' arXiv preprint arXiv:2407.11240 , 2024. - · arXiv
  87. [97][97] M. Zammit, A. Liapis, and G. N. Yannakakis, 'CrawLLM: Theming games with large language models,' in Proceedings of the IEEE Conference on Games , 2024. -
  88. [98][98] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford et al. , 'Mixtral of experts,' arXiv preprint arXiv:2401.04088 , 2024. - · arXiv
  89. [99][99] G. Todd, A. Padula, M. Stephenson, ´ E. Piette, D. J. Soemers, and J. Togelius, 'GAVEL: Generating games via evolution and language models,' arXiv preprint arXiv:2407.09388 , 2024. - · arXiv
  90. [100][100] C. Browne, M. Stephenson, ´ E. Piette, and D. J. N. J. Soemers, 'A practical introduction to the Ludii general game system,' in Proceedings of the International Conference on Advances in Computer Games , 2019. -
  91. [101][101] A. Liapis, 'Searching for sentient design tools for game development,' Ph.D. dissertation, Center for Computer Games, IT University of Copenhagen, Copenhagen, Denmark, 2014. -
  92. [102][102] G. Smith, J. Whitehead, and M. Mateas, 'Tanagra: Reactive planning and constraint solving for mixed-initiative level design,' IEEE Transactions on Computational Intelligence and AI in Games , vol. 3, no. 3, 2011. -
  93. [103][103] A. Liapis, G. N. Yannakakis, and J. Togelius, 'Sentient Sketchbook: Computer-aided game level authoring,' in Proceedings of the International Conference on the Foundations of Digital Games , 2013. -
  94. [104][104] P. Migkotzidis and A. Liapis, 'SuSketch: Surrogate models of gameplay as a design assistant,' IEEE Transactions on Games , vol. 14, no. 2, 2021. -
  95. [105][105] M. Charity, I. Dave, A. Khalifa, and J. Togelius, 'Baba is Y'all 2.0: Design and investigation of a collaborative mixed-initiative system,' IEEE Transactions on Games , 2022. -
  96. [106][106] R. Gallotta, K. Arulkumaran, and L. B. Soros, 'Preferencelearning emitters for mixed-initiative quality-diversity algorithms,' IEEE Transactions on Games , 2023. -
  97. [107][107] M. Charity, Y. Bhartia, D. Zhang, A. Khalifa, and J. Togelius, 'A preliminary study on a conceptual game feature generation and recommendation system,' in Proceedings of the IEEE Conference on Games , 2023. -
  98. [108][108] M. G. Torii, T. Murakami, and Y. Ochiai, 'Lottery and sprint: Generate a board game with design sprint method on AutoGPT,' in Companion Proceedings of the Annual Symposium on ComputerHuman Interaction in Play , 2023. -
  99. [109][109] D. Dhamani and M. L. Maher, 'The tyranny of possibilities in the design of task-oriented LLM systems: A scoping survey,' arXiv preprint arXiv:2312.17601 , 2023. - · arXiv
  100. [110][110] G. N. Yannakakis, A. Liapis, and C. Alexopoulos, 'Mixedinitiative co-creativity,' in Proceedings of the International Conference on the Foundations of Digital Games , 2014. -
  101. [111][111] J. Rezwana and M. L. Maher, 'Designing creative AI partners with COFI: A framework for modeling interaction in humanAI co-creative systems,' ACM Transactions on Computer-Human Interaction , vol. 30, no. 5, 2023. -
  102. [112][112] A. Liapis, G. N. Yannakakis, and J. Togelius, 'Sentient World: Human-based procedural cartography,' in Proceedings of Evolutionary and Biologically Inspired Music, Sound, Art and Design Conference , 2013. -
  103. [114][114] M. Guzdial and M. Riedl, 'An interaction framework for studying co-creative AI,' arXiv preprint arXiv:1903.09709 , 2019. - · arXiv
  104. [115][115] D. Novick and S. Sutton, 'What is mixed-initiative interaction?' in Proceedings of the AAAI Spring Symposium on Computational Models for Mixed Initiative Interaction , 1997. -
  105. [116][116] A. Liapis, G. N. Yannakakis, and J. Togelius, 'Designer modeling for personalized game content creation tools,' in Proceedings of the AIIDE Workshop on Artificial Intelligence & Game Aesthetics , 2013. -
  106. [117][117] V. Vimpari, A. Kultima, P. H¨ am¨ al¨ ainen, and C. Guckelsberger, ''An adapt-or-die type of situation': Perception, adoption, and use of text-to-image-generation AI by game industry professionals,' in Proceedings of the ACM on Human-Computer Interaction , vol. 7, 2023. -
  107. [118][118] J. Boucher, G. Smith, and Y. Telliel, 'Examining early professionals' use of generative AI in the game development process,' in Proceedings of the AIIDE Workshop on Experimental Artificial Intelligence in Games , 2023. -
  108. [119][119] A. Anjum, Y. Li, N. Law, M. Charity, and J. Togelius, 'The ink splotch effect: A case study on ChatGPT as a co-creative game designer,' in Proceedings of the International Conference on the Foundations of Digital Games , 2024. -
  109. [121][121] M. U. Nasir and J. Togelius, 'Practical PCG through large language models,' in Proceedings of the IEEE Conference on Games , 2023. -
  110. [122][122] J. Kelly, A. Calderwood, N. Wardrip-Fruin, and M. Mateas, 'There and back again: Extracting formal domains for controllable neurosymbolic story authoring,' in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , 2023. -
  111. [123][123] V. Kumaran, D. Carpenter, J. Rowe, B. Mott, and J. Lester, 'Endto-End procedural level generation in educational games with natural language instruction,' in Proceedings of the IEEE Conference on Games , 2023. -
  112. [124][124] J. Rowe, E. Lobene, B. Mott, and J. Lester, 'Play in the museum: Design and development of a game-based learning exhibit for informal science education,' International Journal of Gaming and Computer-Mediated Simulations , 2017. -
  113. [125][125] R. Gallotta, A. Liapis, and G. N. Yannakakis, 'LLMaker: A game level design interface using (only) natural language,' in Proceedings of the IEEE Conference on Games , 2024. -
  114. [126][126] --, 'Consistent game content creation via function calling for large language models,' in Proceedings of the IEEE Conference on Games , 2024. -
  115. [127][127] P. Taveekitworachai, F. Abdullah, M. F. Dewantoro, R. Thawonmas, J. Togelius, and J. Renz, 'ChatGPT4PCG competition: Character-like level generation for science birds,' in Proceedings of the IEEE Conference on Games , 2023. -
  116. [128][128] T. Brooks, A. Holynski, and A. A. Efros, 'InstructPix2Pix: Learning to follow image editing instructions,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023. -
  117. [129][129] J. Zhu, A. Liapis, S. Risi, R. Bidarra, and G. M. Youngblood, 'Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation,' in Proceedings of the IEEE Conference on Computational Intelligence and Games , 2018. -
  118. [131][131] A. Canossa and A. Drachen, 'Patterns of play: Play-personas in user-centred game development,' in Proceedings of DiGRA , 2009. -
  119. [132][132] G. N. Yannakakis and D. Melhart, 'Affective game computing: A survey,' Proceedings of the IEEE , 2023. -
  120. [133][133] G. N. Yannakakis and J. Togelius, 'Experience-driven procedural content generation,' IEEE Transactions on Affective Computing , 2011. -
  121. [134][134] K. Pinitas, D. Renaudie, M. Thomsen, M. Barthet, K. Makantasis, A. Liapis, and G. N. Yannakakis, 'Predicting player engagement in Tom Clancy's The Division 2: A multimodal approach via pixels and gamepad actions,' in Proceedings of the ACM International Conference on Multimodal Interaction , 2023. -
  122. [135][135] D. Melhart, A. Liapis, and G. N. Yannakakis, 'The Arousal video Game AnnotatIoN (AGAIN) dataset,' IEEE Transactions on Affective Computing , vol. 13, no. 4, 2022. -
  123. [136][136] J. Broekens, B. Hilpert, S. Verberne, K. Baraka, P. Gebhard, and A. Plaat, 'Fine-grained affective processing capabilities emerging from large language models,' in Proceedings of the International Conference on Affective Computing and Intelligent Interaction , 2023. -
  124. [137][137] A. Liapis, 'Artificial intelligence for designing games,' in The Handbook of Artificial Intelligence and the Arts , P. Machado, J. Romero, and G. Greenfield, Eds. Springer, 2021, in print. -
  125. [138][138] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto, 'Behavior transformers: Cloning k modes with one stone,' in Proceedings of the Neural Information Processing Systems Conference , 2022. -
  126. [139][139] J. Pfau, A. Liapis, G. N. Yannakakis, and R. Malaka, 'Dungeons & Replicants II: Automated game balancing across multiple difficulty dimensions via deep player behavior modeling,' IEEE Transactions on Games , 2023. -
  127. [140][140] S. Johnson-Bey, M. Mateas, and N. Wardrip-Fruin, 'Toward using ChatGPT to generate theme-relevant simulated storyworlds,' in Proceedings of the AIIDE Workshop on Experimental Artificial Intelligence in Games , 2023. -
  128. [141][141] M. Kreminski, D. Acharya, N. Junius, E. Oliver, K. Compton, M. Dickinson, C. Focht, S. Mason, S. Mazeika, and N. WardripFruin, 'Cozy mystery construction kit: Prototyping toward an AIassisted collaborative storytelling mystery game,' in Proceedings of the International Conference on the Foundations of Digital Games , 2019. -
  129. [142][142] C. Fern´ andez-Vara and A. Thomson, 'Procedural generation of narrative puzzles in adventure games: The Puzzle-Dice system,' in Proceedings of the Workshop on Procedural Content Generation in Games , 2012. -
  130. [143][143] J. Dormans, 'Adventures in level design: generating missions and spaces for action adventure games,' in Proceedings of the Workshop on Procedural Content Generation in Games , 2010. -
  131. [144][144] S. Al-Nassar, A. Schaap, M. V. D. Zwart, M. Preuss, and M. A. G´ omez-Maureira, 'QuestVille: Procedural quest generation using NLP models,' in Proceedings of the International Conference on the Foundations of Digital Games , 2023. -
  132. [145][145] V. Kumaran, J. Rowe, B. Mott, and J. Lester, 'SceneCraft: Automating interactive narrative scene generation in digital games with large language models,' Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , 2023. -
  133. [148][148] S. Milani et al. , 'Towards solving fuzzy tasks with human feedback: A retrospective of the MineRL BASALT 2022 competition,' in Proceedings of the NeurIPS 2022 Competitions Track , 2022. -
  134. [149][149] A. M. Smith, E. Andersen, M. Mateas, and Z. Popovic, 'A case study of expressively constrainable level design automation tools for a puzzle game,' in Proceedings of the International Conference on the Foundations of Digital Games , 2012. -
  135. [150][150] D. Karavolos, A. Liapis, and G. N. Yannakakis, 'Learning the patterns of balance in a multi-player shooter game,' in Proceedings of the FDG Workshop on Procedural Content Generation in Games , 2017. -
  136. [151][151] P. Manakul, A. Liusie, and M. J. F. Gales, 'SelfCheckGPT: Zeroresource black-box hallucination detection for generative large language models,' in Proceedings of the Empirical Methods in Natural Language Processing Conference , 2023. -
  137. [152][152] A. Strasser, 'On pitfalls (and advantages) of sophisticated large language models,' arXiv preprint arXiv:2303.17511 , 2023. - · arXiv
  138. [153][153] N. Bian, H. Lin, P. Liu, Y. Lu, C. Zhang, B. He, X. Han, and L. Sun, 'Influence of external information on large language models mirrors social cognitive patterns,' arXiv preprint arXiv:2305.04812 , 2023. - · arXiv
  139. [154][154] M. Karpinska and M. Iyyer, 'Large language models effectively leverage document-level context for literary translation, but critical errors persist,' in Proceedings of the Machine Translation Conference , 2023. -
  140. [155][155] Z. Gekhman, J. Herzig, R. Aharoni, C. Elkind, and I. Szpektor, 'TrueTeacher: Learning factual consistency evaluation with large language models,' in Proceedings of the Empirical Methods in Natural Language Processing Conference , 2023. -
  141. [156][156] J. Zhou, 'An evaluation of state-of-the-art large language models for sarcasm detection,' arXiv preprint arXiv:2312.03706 , 2023. - · arXiv
  142. [157][157] Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu, Z. Wu, L. Zhao, D. Zhu, X. Li, N. Qiang, D. Shen, T. Liu, and B. Ge, 'Summary of ChatGPT-related research and perspective towards the future of large language models,' MetaRadiology , vol. 1, no. 2, 2023. -
  143. [158][158] D. Li, A. S. Rawat, M. Zaheer, X. Wang, M. Lukasik, A. Veit, F. Yu, and S. Kumar, 'Large language models with controllable working memory,' in Findings of the Association for Computational Linguistics , 2023. -
  144. [159][159] Gemini Team, 'Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,' arXiv preprint arXiv:2403.05530 , 2024. - · arXiv
  145. [160][160] T. Munkhdalai, M. Faruqui, and S. Gopal, 'Leave no context behind: Efficient infinite context transformers with infini-attention,' arXiv preprint arXiv:2404.07143 , 2024. - · arXiv
  146. [161][161] Z. Fountas, M. A. Benfeghoul, A. Oomerjee, F. Christopoulou, G. Lampouras, H. Bou-Ammar, and J. Wang, 'Human-like episodic memory for infinite context LLMs,' arXiv preprint arXiv:2407.09450 , 2024. - · arXiv
  147. [162][162] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K¨ uttler, M. Lewis, W.-t. Yih, T. Rockt¨ aschel, S. Riedel, and D. Kiela, 'Retrieval-augmented generation for knowledgeintensive NLP tasks,' in Proceedings of the Conference on Neural Information Processing Systems , 2020. -
  148. [164][164] M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao, 'Large language model cascades with mixture of thoughts representations for cost-efficient reasoning,' arXiv preprint arXiv:2310.0309 , 2024. -
  149. [165][165] Y. Yu, Q. Zhang, J. Li, Q. Fu, and D. Ye, 'Affordable generative agents,' arXiv preprint arXiv:2402.02053 , 2024. - · arXiv
  150. [166][166] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, 'Towards efficient generative large language model serving: A survey from algorithms to systems,' arXiv preprint arXiv:2312.15234 , 2023. - · arXiv
  151. [167][167] F. de la Torre, C. M. Fang, H. Huang, A. Banburski-Fahey, J. A. Fernandez, and J. Lanier, 'LLMR: Real-time prompting of interactive worlds using large language models,' arXiv preprint arXiv:2309.12276 , 2023. - · arXiv
  152. [168][168] D. Melhart, J. Togelius, B. Mikkelsen, C. Holmg˚ ard, and G. N. Yannakakis, 'The ethics of AI in games,' IEEE Transactions on Affective Computing , vol. 15, no. 1, 2023. -
  153. [169][169] A. A. Chien, L. Lin, H. Nguyen, V. Rao, T. Sharma, and R. Wijayawardana, 'Reducing the carbon impact of generative AI inference (today and in 2035),' in Proceedings of the SCS Workshop on Sustainable Computer Systems , 2023. -
  154. [170][170] S. A. Khowaja, P. Khuwaja, and K. Dev, 'ChatGPT needs SPADE (sustainability, PrivAcy, Digital divide, and Ethics) evaluation: A review,' arXiv preprint arXiv:2305.03123 , 2023. - · arXiv
  155. [174][174] P. Zhang, G. Zeng, T. Wang, and W. Lu, 'TinyLlama: An opensource small language model,' arXiv preprint arXiv:2401.02385 , 2024. - · arXiv
  156. [175][175] U.S. Copyright Office Review Board, 'Decision affirming refusal of registration of 'A Recent Entrance to Paradise',' 2022. -
  157. [177][177] C. E. Lamb and D. G. Brown, 'Should we have seen the coming storm? Transformers, society, and CC,' in Proceedings of the International Conference on Computational Creativity , 2023. -
  158. [178][178] Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. Guo, H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li, 'Trustworthy LLMs: a survey and guideline for evaluating large language models' alignment,' in Proceedings of the NeurIPS Workshop on Socially Responsible Language Modelling Research , 2023. -
  159. [179][179] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, 'Large language models are zero-shot reasoners,' in Proceedings of the Advances in Neural Information Processing Systems Conference , 2022. -
  160. [180][180] S. Chen, B. Li, and D. Niu, 'Boosting of thoughts: Trial-and-error problem solving with large language models,' arXiv preprint arXiv:2402.11140 , 2024. - · arXiv
  161. [181][181] D. Mondal, S. Modi, S. Panda, R. Singh, and G. S. Rao, 'KAMCoT: Knowledge augmented multimodal chain-of-thoughts reasoning,' arXiv preprint arXiv:2401.12863 , 2024. - · arXiv
  162. [182][182] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, 'Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting,' in Proceedings of the Neural Information Processing Systems Conference , 2023. -
  163. [183][183] S. Frieder, L. Pinchetti, A. Chevalier, R.-R. Griffiths, T. Salvatori, T. Lukasiewicz, P. C. Petersen, and J. Berner, 'Mathematical capabilities of ChatGPT,' in Proceedings of the Neural Information Processing Systems Conference , 2023. -
  164. [184][184] A. M. Karkaj, M. J. Nelson, I. Koutis, and A. K. Hoover, 'Prompt wrangling: On replication and generalization in large language models for PCG levels,' in Proceedings of the International Conference on the Foundations of Digital Games , 2024. -
  165. [185][185] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, 'Transformers: State-of-the-art natural language processing,' in Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2020. -
  166. [186][186] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, 'Quantization and training of neural networks for efficient integer-arithmetic-only inference,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. -
  167. [187][187] Y. Jia, W. Wu, F. Cao, and S. C. Han, 'In-game toxic language detection: Shared task and attention residuals,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2022. -

Metadata

Authors
Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., & Yannakakis, G. N.
Affiliations
  • University of Malta (MT)
  • New York University (US)
ORCID
Venue
IEEE Transactions on Games
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2024
Pages
1-18
ISSN
  • 2475-1502 (ISSN-L)
  • 2475-1510
DOI
10.1109/tg.2024.3461510
OpenAlex
W4402508150
Cited by
137
References
0
Citations by year
2026: 55 · 2025: 59 · 2024: 23
FWCI
20.22
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Games (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
large language models, games, game design, game development, future research directions, LLM limitations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review