The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index; 5 of the 12 sources its claims cite were checked for support.
Game Plot Design With an LLM-Powered Assistant: An Empirical Study With Game Designers
Open accessAbstract
We introduce GamePlot, an LLM-powered assistant that supports game designers in crafting immersive narratives for turn-based games, and allows them to test these games through a collaborative game play and refine the plot throughout the process. Our user study with 14 game designers shows high levels of both satisfaction with the generated game plots and sense of ownership over the narratives, but also reconfirms that LLM are limited in their ability to generate complex and truly innovative content. We also show that diverse user populations have different expectations from AI assistants, and encourage researchers to study how tailoring assistants to diverse user groups could potentially lead to increased job satisfaction and greater creativity and innovation over time.
Overview
CoChat Summary
TL;DR
GamePlot is an LLM-powered assistant for turn-based game narrative design that supports plot generation, collaborative playtesting, live plot refinement, and Wizard-of-Oz NPC control. In a user study with 14 game designers, participants reported high satisfaction and ownership, while also confirming LLM limitations in creativity, consistency, and narrative complexity.
Key findings
- Participants rated the AI assistant’s help in improving their game story at 4/5 on average, satisfaction with the generated game plot at 4/5 on average, and perceived control/ownership over the story at 4.5/5 on average; all ratings exceeded 3.
- The game room’s usefulness for testing with players averaged 4.21, with 13 out of 14 participants giving ratings of 4 or 5.
- The most-liked feature was the ability to change the game plot during the test session, favored by 12 out of 14 participants; half also favored the NPC hidden-states dropdown and the multiplayer setup.
- Designers with different backgrounds had different expectations: game developers often preferred to offload narrative writing to the AI, while narrative writers wanted creative control but valued AI for exploring narrative paths. LLM outputs were described as lacking creativity, consistency, tension balance, and organic dialogue.
Methods
- The authors built GamePlot, a web application with a design room and a game room, using GPT-3.5-Turbo-16k with default hyperparameters, a 2,000-token limit for summarization/plot generation, a 1,000-token limit for next-turn generation, and summarization of earlier turns after 40,000 characters.
- They recruited 14 participants through Upwork and social media; 64.3% had at least one year of professional gaming-industry experience, and 8 identified as narrative-writing experts. Sessions lasted 1–2 hours, with compensation at $25/hour.
- Participants received a tutorial and template story, designed a game in the design room for 20–25 minutes, tested it in a game room with one author as a player for 15–20 minutes, then completed a post-study questionnaire; free-text responses were manually classified by the authors.
Limitations
- Stated: the study used only GPT-3.5, so results may not extend to other LLMs without empirical testing; the sample of 14 narrative designers is small and may not capture diverse designer backgrounds; and there is likely sampling bias because participants were already inclined to embrace AI in game narrative design.
- Evident: the study has no human-only or alternative-tool control condition, so it cannot empirically establish whether GamePlot improves final narrative quality over unaided design; the paper itself notes the final product may not surpass a human-written story in emotional depth or complexity.
- Evident: testing involved one of the authors joining as a player, which may have shaped the collaborative playtest experience, though the paper does not discuss this as a limitation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 1 Not valid · 5 Not checked · 71 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Consistency emerged as another problem, with participants reporting that the AI sometimes failed to maintain plot coherence”
What the sources say
- Could not verifycites Que et al., 2024 sourceExperimental — accuracy not yet measured
The Checklists for Text Completion section lists plot coherence as a criterion for evaluating AI-generated continuations, and the example text-completion prompt requires a coherent storyline. Neither passage reports participants saying that AI failed to maintain plot coherence; searches for participant reports and narrative inconsistency found no such finding in our copy. (LLM, conf=0.72)
- Partly supportedcites Lin et al., 2024 sourceExperimental — accuracy not yet measured
Section 4.1, “Manifesting of hallucination” (page unavailable), says LLM output can lack meaningful coherence or lose contextual anchoring. It does not discuss participants reporting problems or failures to maintain plot coherence specifically. (LLM, conf=0.78)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our user study with 14 game designers shows high levels of both satisfaction with the generated game plots and sense of ownership over the narratives, but also reconfirms that LLM are limited in their ability to generate complex and truly innovative content.”
- Also stated in: § 1 Introduction · p. 1 · plain
“Participants were satisfied with the generated game plots, considered the AI assistant as beneficial in enhancing the game storytelling, and also reported a sense of control and ownership over the narratives.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 1 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our user study with 14 game designers shows high levels of both satisfaction with the generated game plots and sense of ownership over the narratives, but also reconfirms that LLM are limited in their ability to generate complex and truly innovative content.”
- Also stated in: § 1 Introduction · p. 1 · plain
“Participants were satisfied with the generated game plots, considered the AI assistant as beneficial in enhancing the game storytelling, and also reported a sense of control and ownership over the narratives.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 1 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our user study with 14 game designers shows high levels of both satisfaction with the generated game plots and sense of ownership over the narratives, but also reconfirms that LLM are limited in their ability to generate complex and truly innovative content.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also show that diverse user populations have different expectations from AI assistants, and encourage researchers to study how tailoring assistants to diverse user groups could potentially lead to increased job satisfaction and greater creativity and innovation over time.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
35 more claims
Create a free account to see all 81 claims and every source verdict.
- Probably not a substantial claim. 34
- It may state two separate claims as one. 16
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A.3 Tag Inventory · p. 11
“In our experiments, we assigned different tags to NPCs ( [ID] , [Persona] , [Mood] , [Thought] , [Action] , and [Words] ) and players (only [Action] and [Words] ).”
§ 3.1 Design Room · p. 3
“[ID] Neighbor: [Backstory] She is a former spy who has gone rogue and is now trying to make amends for her past actions.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Nader Akoury, Ronan Salz, and Mohit Iyyer. 2023. Towards grounded dialogue generation in video game environments. In Creative AI Across Modalities Workshop, AAAI .
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Judith van Stegeren and Jakub My´ sliwiec. 2021. Finetuning gpt-2 on annotated rpg quests for npc dialogue generation. In Proceedings of the 16th International Conference on the Foundations of Digital Games , FDG '21, New York, NY, USA. Association for Computing Machinery.
- 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022) , pages 25-43, Seattle, United States. Association for Computational Linguistics.
Metadata
- Authors
- Alavi, S. H., Xu, W., Jojic, N., Kennett, D., Ng, R. T., Rao, S., Zhang, H., Dolan, B., & Shwartz, V.
- Affiliations
- University of British Columbia (CA)
- University of British Columbia
- Venue
- IEEE Transactions on Games
- Published
- 2026
- Pages
- 1-10
- ISSN
- 2475-1502 (ISSN-L)
- 2475-1510
- OpenAlex
- W7128620905
- Semantic Scholar
- b4f1bc92ce981a02fcf2ee5f7ed0a4a30cc3c20f
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Games (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Social Sciences › Sociology and Political Science › Digital Games and Media
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
- Keywords
- large language models, turn-based games, game narrative, collaborative gameplay, user satisfaction, sense of ownership, user heterogeneity, creativity and innovation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Game Plot Design With an LLM-Powered Assistant: An Empirical Study With Game Designers. https://app.cochat.ai/papers/10.1109/tg.2026.3663566
- The paper
- Alavi, S. H., Xu, W., Jojic, N., Kennett, D., Ng, R. T., Rao, S., Zhang, H., Dolan, B., & Shwartz, V. (2026). Game Plot Design With an LLM-Powered Assistant: An Empirical Study With Game Designers. IEEE Transactions on Games. https://doi.org/10.1109/tg.2026.3663566
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.