CoChat
Sign Up
Cited by 8Artificial Intelligence in GamesDigital Games and MediaEducational Games and Gamification

Evaluating Planning-Based Experience Managers for Agency and Fun in Text-Based Interactive Narrative

Open access
Ramirez, Alejandro, Bulitko, Vadim, Spetch, Marcia·2013·Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment
DOI 10.1609/aiide.v9i1.12684
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial intelligence (AI) techniques have been applied to video games to make the overall experience more enjoyable. In games with interactive storytelling (IS), player actions can substantially affect plot events and plot characters. Therefore, AI planning techniques have been used to shape the plot inresponse to player actions that conflict with authorial goals. While such methods are poised to increase player fun andagency, two recent implementations (ASD and PAST) have not been formally evaluated to date. In this paper we do so via a series of user studies for the first time. We show that ASD significantly enhances fun and agency, whereas PAST gets mixed results with an interaction between effects of the experience manager and player prior gaming experience in one user study, and marginally significant results for increased agency in a study with a constrained story domain.

Integrity signals

Registry

Crossref · 10.1609/aiide.v9i1.12684

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

5 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

4 of 24 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisenone given

    ASD and PAST had not been formally evaluated with a user study before this paper.

    “two recent implementations (ASD and PAST) have not been formally evaluated to date.”
    • Also stated in: § Introduction · plain
      “Despite their promise, neither of the two experience managers (ASD and PAST) has been formally evaluated with a user study.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Riedl et al. 2008

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ramirez and Bulitko 2012 source

      The Conclusions and Future Work section says the PAST prototype is operational but that its performance will be evaluated in future user studies, supporting the claim about PAST. The Introduction describes ASD, but our copy does not say whether ASD had undergone a formal evaluation. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data

    Some participants reported a pull-back effect, feeling that their choices to break the narrative were followed by a return to the original plot.

    “Some participants appear to have experienced a pull-back effect wherein they felt that upon breaking the narrative through their choices they were pulled back into the original plot.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In user study 2, experience management and gaming skills interacted significantly for fun (F(2,66)=4.73, p=0.012).

    “However, a significant interaction between these factors was found for fun (F (2, 66) = 4.73, p = 0.012).”
    Quote in the paperSubstantial claim
  • Findingown data

    In user study 2, the interaction between experience management and gaming skills for agency was not statistically significant (F(2,67)=1.93, p=0.1538).

    “Agency showed a similar interaction but it failed to reach significance (F (2, 67) = 1.93, p = 0.1538).”
    Quote in the paperNot a substantial claim
  • Findingown data

    In user study 2, players were unevenly distributed across different stories.

    “Figure 5 shows the uneven distribution of players over different stories (i.e., each story being a collection of different events).”
    Quote in the paperNot a substantial claim
  • May combine two claims. 13
  • Classified as probably not a substantial claim. 9
  • The quote only partly states this claim. 3

  • May combine two claims.

    Premisecitation

    In ASD and PAST, the AI planner can foresee ruptures and precompute narrative repairs.

    “Such ruptures can be foreseen by the AI planner and repairs to the narrative can be precomputed in response.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Riedl et al. 2008

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ramirez and Bulitko 2012 source

  • May combine two claims.

    Premisecitation

    Both ASD and PAST represent narratives as formal plans.

    “Both ASD and PAST are based on representing narratives as formal plans.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Riedl et al. 2008

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ramirez and Bulitko 2012 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    ASD selected among contingency narratives ad hoc, while PAST used a player model to select.

    “Whereas ASD used an ad hoc method of selecting between contingency narratives, PAST uses a player model to do so.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Riedl et al. 2008

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ramirez and Bulitko 2012 source

  • May combine two claims.

    Findingown data

    In user study 2, neither experience management nor gaming skills had a statistically significant main effect on fun or agency.

    “No statistically significant results were observed in the main factors of experience management and gaming skills for either fun or agency.”
    • Also stated in: § Discussion · plain
      “The results of the second user study did not show significant effects of player modelling in PAST for either player fun or agency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In user study 2, non-gamers in the control condition reported significantly more fun than non-gamers in the experimental condition, with no other post hoc comparisons significant.

    “Tukey's HSD post hoc test showed that non-gamers in the control condition had significantly higher fun ratings than non-gamers in the experimental condition; no other comparisons reached significance.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    A replication of user study 2 with 63 participants found no significant effects.

    “This time, no significant effects were found.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In user study 3, fun showed no statistically significant effect, while agency showed a marginally significant effect (F(1,67)=5.46, p=0.097).

    “No statistically significant results were found for fun (p < 0.05); however, a marginally significant result (p < 0.10) for agency (F (1, 67) = 5.46, p = 0.097) was found.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Interpretationown data

    In a constrained story domain with more even coverage, PAST increased agency for both gamers and non-gamers.

    “When using a smaller story domain with a more even coverage, agency is increased for both groups.”
    • Also stated in: § Discussion · hedged
      “The third user study, in a constrained story domain, showed marginally significant results that may indicate that agency is increased by PAST.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    In user study 2, the yoking procedure controlled only for repair types and not for the actual ruptures taken.

    “Our attempt to control variance by randomly assigning player models from the experimental participants to control condition participants (yoking) accounted only for the types of repairs, hence not controlling for the actual ruptures taken.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The balance of story content across conditions is particularly important for measuring fun and agency in experience manager evaluations.

    “This highlights that the balance of content is of particular importance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Treating interactive storytelling as AI planning is an effective way to improve player experience while giving players more choice within a narrative.

    “Results presented in this paper support previous conjectures that treating IS as AI planning is an effective way to improve player's experience while affording them more choice within a narrative.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Balancing story content across conditions is fundamental to prevent fun and agency differences caused by unequal exposure to influential story content.

    “This is fundamental to prevent fun and agency measured in the experimental condition from being affected merely by player's experience of a particular influential story content, absent from the control condition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The data met the assumptions of independent observations, normality, and unequal variance.

    “The data did not violate the assumptions of independent observations, normality (assessed by the Kolmogorov-Smirnov test) or unequal variance (assessed by the Levine's and Welch's tests).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    To reduce skimming noise, participants reading above 405 words per minute were excluded, based on a reported 450 wpm skimming speed and a 10% screen-reading slowdown.

    “Given the 10% slow down due to reading off the screen (Zelfle 1998), we eliminated data from all participants who rather than that read above 405 words per minute.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Carver 1993

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zelfle 1998

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    PAST combines ASD and PaSSAGE, and each in isolation showed positive effects.

    “This is surprising since PAST is a combination of ASD and PaSSAGE, which in isolation showed a possitive effect.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Riedl et al. 2008

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Thue et al. 2007 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A suspense manager was evaluated in a user study and significant benefits were found.

    “For instance, a suspense manager was evaluated in a user study (Cheong and Young 2008) and significant benefits were found.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Cheong and Young 2008

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Participants with outlier fun and/or agency scores according to Tukey's outlier filter were removed.

    “We then additionally removed data from all participants whose fun and/or agency score was an outlier according to Tukey's outlier filter (also known as interquartile outlier detection).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    In user study 2, the control condition used player models randomly drawn from experimental participants as part of yoking.

    “In the control condition, the player model was taken from a randomly drawn player in the experimental condition (as a part of the yoking procedure).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    PaSSAGE demonstrated significant effects of using a player model to select content, but its experience manager was manually scripted and did not use automated content generation.

    “While significant effects of using a player model to select content were demonstrated, the experience manager within PaS-SAGE was manually scripted and did not use automated content generation techniques.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Thue et al. 2007 source
    • cites Thue et al. 2010 source
    • cites Thue et al. 2011 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The player modelling approach used in PAST had previously been found effective in PaSSAGE, though without automated planning.

    “This approach was previously found effective within PaSSAGE although without any automated planning (Thue et al. 2007).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Thue et al. 2007 source

  • Classified as probably not a substantial claim.

    Premisecitation

    In PAST, the player model is a five-number vector representing inclinations toward fighter, power gamer, storyteller, method actor, and tactician styles, based on Robin Laws' player types.

    “In PAST, the player model is a vector of five numbers, each indicating an inclination towards a certain style of play: fighter, power gamer, storyteller, method actor, and tactician, based on Robin Laws' player types (Laws 2001).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Laws 2001

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    Fun and agency were measured with a previously validated questionnaire (Vermeulen et al. 2010) that had been used to evaluate experience managers (Thue et al. 2011).

    “For the survey, we used a previously validated instrument (Vermeulen et al. 2010).”
    • Also stated in: § Methodology · plain
      “The instrument has been previously used to evaluate experience managers (Thue et al. 2011).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Vermeulen et al. 2010 source
    • cites Thue et al. 2011 source
22 of 22

  • The paper relies on this without support.

    Fun and agency are important goals for interactive storytelling systems.

    “In the context of interactive narrative, fun (i.e., how entertaining and amusing an interactive experience is) and agency (i.e., the ability of the player to affect the virtual world) are frequently deemed important goals targeted by IS systems.”

  • The paper relies on this without support.

    Some experience managers improve fun and agency but increase authorial burden by relying on story annotations and extra game content.

    “While some experience managers can improve fun and agency in an interactive narrative, they do so by relying heavily on story annotations and extra game content, therefore increasing the authorial burden on the developers.”

  • The paper relies on this without support.

    The text-based Little Red Riding Hood adventure is representative of interactive storytelling experiences in general.

    “To evaluate ASD and PAST we constructed a text-based interactive adventure-inspired partly by RPG text-based adventure games-around the exemplar narrative of The Little Red Riding Hood, in line with the literature (Riedl et al. 2008;Ramirez and Bulitko 2012).”
3 of 3

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • Inconsistent p-value across alpha = .05: F (1, 67) = 5.46, p = .097

    Major

    The paper reports p = .097 here; recomputing from the test statistic and its degrees of freedom gives p = .022. The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .097 · recomputed p = .022

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upF(1, 67) = 5.46reported p = .097 · recomputed p = .022The two fall on opposite sides of .05

    “F (1, 67) = 5.46, p = .097”

  • ConsistentF(1, 76) = 5.46reported p = .02 · recomputed p = .022

    “F (1, 76) = 5.46, p = .02”

  • ConsistentF(1, 77) = 20.46reported p = .001 · recomputed p < .001

    “F (1, 77) = 20.46,p = .001”

    The paper prints p at a floor (such as p < .001)

  • ConsistentF(2, 66) = 4.73reported p = .012 · recomputed p = .012

    “F (2, 66) = 4.73, p = .012”

  • ConsistentF(2, 67) = 1.93reported p = .1538 · recomputed p = .153

    “F (2, 67) = 1.93, p = .1538”

5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Study 3 reports more error degrees of freedom than participants

    Major

    Study 3 reports 34 participants, but its agency test has 67 error degrees of freedom. A test based on those 34 participants cannot have 67 residual degrees of freedom; the reported test statistic or sample size is wrong.

  • Study 2 agency test has an incompatible error degree of freedom

    Minor

    Study 2 has 72 participants distributed across all six combinations of its three management conditions and two gaming-skill groups. A two-way ANOVA using those groups can have at most 66 residual degrees of freedom, not the reported 67.

2 of 2

Cites

Flagged first, then by citations

1 of the 14 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. Riedl, M, O.; Stern, A.; Dini, D.; and Alderman, J. 2008. Dynamic experience management in virtual worlds for en- tertainment, education, and training. International Transac- tions on Systems Science and Applications 4(2):23-42.
  2. Sharma, M.; Mehta, M.; Ontanón, S.; and Ram, A. 2007. Player modeling evaluation for interactive fiction. In Pro- ceedings of the AIIDE Workshop on Optimizing Player Sat- isfaction, 19-24.

Cited by

Sorted by citations

Related

Authors
Ramirez, A., Bulitko, V., & Spetch, M.
Affiliations
University of Alberta (CA)
ORCID
Spetch, Marcia
Venue
Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2013-10-19
Volume
9
Issue
1
Pages
65-71
ISSN
  • 2326-909X (ISSN-L)
  • 2334-0924
DOI
10.1609/aiide.v9i1.12684
OpenAlex
W2225093001
Cited by
8
References
14
Citations by year
2017: 2 · 2015: 1 · 2014: 5
FWCI
1.37
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (publishedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Social Sciences › Sociology and Political Science › Digital Games and Media
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
Keywords
interactive storytelling, player agency, past

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-05.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-05.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review