CoChat
Sign Up
Cited by 240Artificial Intelligence in GamesArtistic and Creative ResearchDigital Media and Visual Art

Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals

Open access
Mirowski, Piotr, Mathewson, Kory W., Pittman, Jaylen, Evans, Richard·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3581225
Open this paper in CoChat — free accountOpen PDF

Abstract

Language models are increasingly attracting interest from writers. However, such models lack long-range semantic coherence, limiting their usefulness for longform creative writing. We address this limitation by applying language models hierarchically, in a system we call Dramatron. By building structural context via prompt chaining, Dramatron can generate coherent scripts and screenplays complete with title, characters, story beats, location descriptions, and dialogue. We illustrate Dramatron’s usefulness as an interactive co-creative system with a user study of 15 theatre and film industry professionals. Participants co-wrote theatre scripts and screenplays with Dramatron and engaged in open-ended interviews. We report reflections both from our interviewees and from independent reviewers who critiqued performances of several of the scripts to illustrate how both Dramatron and hierarchical text generation could be useful for human-machine co-creativity. Finally, we discuss the suitability of Dramatron for co-creativity, ethical considerations—including plagiarism and bias—and participatory models for the design and deployment of such tools.

Integrity signals

Registry

Crossref · 10.1145/3544548.3581225

Cites

106 of 106 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

20 of 32 cited sources checked · 8 supported · 2 flagged · accuracy not yet measured

  • Premisereasoning

    The current state-of-the-art method for longer text generation incorporates a human-in-the-loop who selects from various generations sampled from the model.

    “Because these models do not achieve long-term semantic coherence, the current state-of-the-art method for longer text generation incorporates a human-in-the-loop who selects from various generations sampled from the model [10,67,75].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [10] 10. Boyd Branch, Piotr Mirowski, and Kory W Mathewson. 2021. Collaborative Storytelling with Human Actors and AI Narrators. Proceedings of the 12th International Conference on Computational Creativity (2021). https://arxiv.org/ abs/2109.14728 source

      In “Interactive live AI narration on stage” (p. 2), the paper describes an operator choosing among three GPT-3-generated sets of sentences for a long-form storytelling performance. It supports the human-selection method as an example, but does not establish that this is the current state-of-the-art method for longer text generation generally. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [67] 67. Kory Mathewson and Piotr Mirowski. 2018. Improbotics: Exploring the imitation game using machine intelligence in improvised theatre. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14. source

      Section 2.2 describes a human controller choosing among model-generated dialogue suggestions, and Section 4.3 says the resulting lines can still be narratively inconsistent. The paper does not establish that this is the state-of-the-art method for longer text generation; Section 4.5 describes its dialogue system as only one component of a larger story-generation system. Page numbers were not available in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [75] 75. Piotr Mirowski and Kory Wallace Mathewson. 2019. Human improvised theatre augmented with artificial intelligence. In Proceedings of the 2019 on Creativity and Cognition. 527-530. source
  • Premisecitation

    The context window of state-of-the-art large language models is limited to at most 2048 tokens.

    “This limitation stems from the context window of LLMs, which today is limited to at most 2048 tokens (i.e. about 1500 words) in state-of-the-art models [78,86].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [78] 78. OpenAI. 2021. Pricing. https://openai.com/api/pricing/ source

      Section C, “Additional model details” (page not specified), says the paper’s own language models and RL policies have a context length of 2k tokens. It does not establish that all state-of-the-art LLMs are limited to at most 2048 tokens, or equate that length with about 1500 words. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [86] 86. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoff- mann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Su- sannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saf- fron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Lau- rent source

      The Training section (p. 6) says the authors trained their models with a 2048-token context window. That does not establish a 2048-token maximum for all state-of-the-art LLMs; D.6 (p. 62) also reports extending Gopher’s context length at evaluation time beyond its training boundary. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    Chinchilla 70B outperformed other large language models on a large collection of language tasks.

    “We conducted our study using Chinchilla 70B because it outperformed other LLMs on a large collection of language tasks [50].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [50] 50. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556 (2022). source
  • Findingown data

    Many participants noticed gender biases and ageism in the system outputs.

    “many participants noticed gender biases and ageism in the system outputs.”
    Quote in the paperSubstantial claim
  • Findingown data

    All participants found Dramatron useful for getting inspiration.

    “All participants found Dramatron useful for getting inspiration”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 85
  • May combine two claims. 34
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    The study does not evaluate Dramatron outputs for grammatical correctness because the few errors made by the LLM can be fixed by the human co-writer.

    “We do not evaluate Dramatron outputs for grammatical correctness as [80], as the few errors made by the LLM can be fixed by the human co-writer.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [80] 80. Vishakh Padmakumar and He He. 2022. Machine-in-the-Loop Rewriting for Creative Image Captioning. In Proceedings of the 20th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). source

      Section 5.1, “User Evaluation of Suggestion Quality,” explicitly asks users how grammatically correct the model suggestions were; the Results report grammaticality ratings. The source therefore does evaluate grammatical correctness, contrary to the practice the citing sentence attributes to it. Its study concerns creative image captions, not Dramatron, and the cited passages do not give human correction of a few errors as a reason to omit evaluation. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • Rests on an assumption: The few grammatical errors made by the LLM can be fixed by the human co-writer.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The first mention and use of hierarchical generation for a theatre play was in [93,95] for THEaiTRE's 2021 production AI: Can a Robot Write a Play? in Prague.

    “Hierarchical generation of a theatre play was first mentioned and used in [93,95] for the production of AI: Can a Robot Write a Play? by company THEaiTRE in 2021 in Prague 9 .”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [93] 93. Rudolf Rosa, Ondřej Dušek, Tom Kocmi, David Mareček, Tomáš Musil, Patrícia Schmidtová, Dominik Jurko, Ondřej Bojar, Daniel Hrbek, David Košt'ák, et al. 2020. THEaiTRE: Artificial intelligence to write a theatre play. arXiv preprint arXiv:2006.14668 (2020). source

      The paper mentions hierarchical generation in “Unresolved issues and future plans” (p. 3), but says the authors intended to use it in a future version—not that they used it for the 2021 play. The Introduction (p. 1) instead says they generated individual scenes and a dramaturge joined them into a full play. The copy does not establish that this was the first mention of hierarchical theatre-play generation. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [95] 95. Rudolf Rosa, Patrícia Schmidtová, Ondřej Dušek, Tomáš Musil, David Mareček, Saad Obaid, Marie Nováková, Klára Vosecká, and Josef Doležal. 2022. GPT-2- based Human-in-the-loop Theatre Play Script Generation. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022). 29-37. source

  • May combine two claims.

    Findingcitation

    Characters, plot, and location descriptions all meet the specification in the log line, in addition to causal dependencies, enabled by prompt chaining.

    “Characters, plot, and location descriptions all meet the specification in the log line, in addition to causal dependencies, enabled by prompt chaining [121] and explained on the diagram of Figure 1.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [121] 121. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. arXiv:2109.10862 [cs.CL] source

      The Framework (p. 2) describes a sequence of connected steps that generates each story sentence and then repeats using the expanded context. It does not describe this as prompt chaining or show that characters, plot, and location descriptions meet a log-line specification or preserve causal dependencies. Future Work (p. 8) also treats structure-level control as a potential direction rather than an established result. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several participants requested conversational agents to help them rewrite text.

    “Several participants requested conversational agents to help them rewrite text-which was prescient of recently published language agents trained from human feedback [44,79].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [44] 44. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judge- ments. arXiv preprint arXiv:2209.14375 (2022). source
    • cites [79] 79. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022). source

  • Classified as probably not a substantial claim.

    Premisecitation

    Crowdworkers' personal opinions, demographic characteristics, and cognitive biases can affect the quality of crowdsourced annotations in fact-checking or subjective assessments.

    “it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [36] 36. Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. In Proceedings of the eleventh ACM international conference on web search and data mining. 162-170. source
    • cites [30] 30. Tim Draws, David La Barbera, Michael Soprano, Kevin Roitero, Davide Ceolin, Alessandro Checco, and Stefano Mizzaro. 2022. The Effects of Crowd Worker Bi- ases in Fact-Checking Tasks. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2114-2124. source
    • cites [53] 53. Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-12. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The survey questions were slightly adapted from Yuan et al. [125] and Stevenson et al. [108].

    “These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Yuan et al. [125]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Stevenson et al. [108] 108. Claire Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. 2022. Putting GPT-3's Creativity to the (Alternative Uses) Test. (2022). source

      The Alternative Uses Task sections (p. 2) provide a question posed to GPT-3 and instructions given to human participants, so the paper contains wording that could be adapted into questions. Our copy does not show the citing document’s survey questions, however, and cannot establish that they were only slightly adapted from this wording. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Similar risks from LLMs have been noted before.

    “Similar LLM risks have been previously mentioned in [119].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [119] 119. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The rules are currently unclear for machine learning systems trained on copyright-protected material.

    “Currently, the rules are unclear for machine learning-based systems trained on copyright-protected material [9].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] 9. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). https://arXiv.org/abs/2108.07258 source

      The Data overview (p. 18) mentions legal regulation of training data, but does not say that copyright rules for training machine-learning systems are unclear. Searches for copyright, intellectual property, fair use, and copyrighted training data found no passage addressing that point in our copy; the copy is also marked as truncated. (LLM, conf=0.70)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior work has argued for involving subject matter experts, literary scholars, and industry professionals in developing machine learning models.

    “Previous work has argued for engagement with subject matter experts, literary scholars, and industry professionals in the development of machine learning models [112].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [112] 112. Imke Van Heerden and Anil Bas. 2021. AI as Author-Bridging the Gap Between Machine Learning and Literary Theory. Journal of Artificial Intelligence Research 71 (2021), 175-189. source

      The Introduction explicitly argues for involving literary scholars in developing machine-learning models, supporting engagement with those subject-matter experts. Section 5 discusses how scholars and professionals might respond to changes in the literary industry, but does not argue that industry professionals should participate in model development. Page numbers are unavailable in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Decomposing a log line into a synopsis can be viewed as a narrative analogue of Chain of Thought prompting, and it uses the idea of feeding LLM output as the prompt for the next generation stage, also known as prompt chaining or language engineering.

    “Our work of decomposing a log line into a synopsis can be seen as a narrative equivalent of Chain of Thought prompting for reasoning tasks [118], and uses the idea of using LLM output as prompt for the next stage of the generation-also called prompt chaining [121] or language engineering [24].”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [118] 118. Stephen G Ware and Robert Michael Young. 2011. CPOCL: A Narrative Planner Supporting Conflict.. In AIIDE. source

      Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [121] 121. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. arXiv:2109.10862 [cs.CL] source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [24] 24. Antonia Creswell and Murray Shanahan. 2022. Faithful Reasoning Using Large Language Models. arXiv preprint arXiv:2208.14271 (2022). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A short plot has been written using prompt engineering from two characters, a genre, and a theme.

    “Prompt engineering has been used to write a short plot from two characters, a genre and a theme in [54].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [54] 54. Yiping Jin, Vishakha Kadam, and Dittaya Wanvarie. 2022. Plot Writing From Pre-Trained Language Models. arXiv preprint arXiv:2206.03021 (2022). https: //arXiv.org/abs/2206.03021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Lee et al. distinguish verbatim, paraphrase, and idea plagiarism and propose automatically evaluating verbatim plagiarism using BM25-based retrieval of queries in a large reference document set along with a text alignment tool.

    “Lee et al. [61] distinguish between verbatim, paraphrase, and idea plagiarism and suggest to automatically evaluate the first one using BM25-based retrieval of queries in large set of reference documents, along with a text alignment tool.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [61] 61. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2022. Do Language Models Plagiarize? arXiv e-prints (2022), arXiv-2203. source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Most participants were able to co-write a full script and complete the open discussion interview within the allotted two hours.

    “Most were able to co-write a full script and the open discussion interview in the allotted 2-hours.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The show's run was titled Plays By Bots and ran 7 performances over two weeks.

    “The show's run was titled Plays By Bots and ran 7 performances over two weeks”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    All 15 participants had prior exposure to AI writing tools.

    “All participants had prior exposure to AI writing tools.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    More than providing a creative spark, the model can be employed to populate the universe of the story.

    “More than merely providing a creative spark for the main story, the model can be employed to populate the universe of the story”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Participant p10 stated that iterative rewriting was a desired workflow.

    “As p10 reiterates, iterative rewriting was a desired workflow”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participant 12 found it difficult to come up with a log line, and the process seemed precious.

    “For participant 12, it was difficult to come up with a log line, and the process seemed precious.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several participants wanted to be able to query and dialogue with the writing model.

    “Several participants wanted to be able to query and dialogue with the writing model”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some participants suggested the tool could be employed in a TV writers' room to help with writing formulaic scripts.

    “Some participants suggested this tool could be employed in a TV writers' room, to help with writing formulaic scripts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The problem of bias raised the issue of coping strategies or cultural appropriation.

    “This problem raised the issue of coping strategies or cultural appropriation”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Findingown data

    Participation of experts is critical in tool development.

    “Widely relevant feedback from the interviews included: participation of experts is critical in tool development, different use-cases demand iterative improvement and development, and different LLMs satisfy different needs from participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participant p1 tried three modes of script generation: interactive co-authorship, modifying output from one fully automated generation, and curating and modifying outputs from 3-4 generations.

    “Participant p1 tried and discussed three different modes of script generation: 1) interactive co-authorship, 2) modifying the output from one fully automated generation, and 3) curating and modifying outputs from 3-4 generations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Learning could also happen by analyzing how to improve Dramatron's outputs.

    “Learning could also happen by analysing how to improve Dramatron's outputs”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The method makes it possible for elements in the final scene to provide dramatic closure on elements introduced in the opening scene, and for generated stories to follow narrative arcs.

    “Our method makes it possible for elements in the final scene to provide dramatic closure on elements introduced in the opening scene 6 , and for generated stories to follow narrative arcs (see Section 3.3).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingown data

    Different use-cases demand iterative improvement and development.

    “Widely relevant feedback from the interviews included: participation of experts is critical in tool development, different use-cases demand iterative improvement and development, and different LLMs satisfy different needs from participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Three participants (p8, p13, p15) suggested fitting a narrative arc within each scene.

    “three participants (p8, p13, p15) suggested fitting a narrative arc within each scene.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The scripts were performed by a cast of 4-6 experienced improvisers and actors.

    “The scripts were brought to life by a cast of 4-6 experienced improvisers and actors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    There were two reviews written about the production of Plays By Bots at the festival.

    “There were two reviews written about the production of Plays By Bots at the festival.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Several participants were interested by the co-writing potential of Dramatron and thought it could provide them with material.

    “Beyond inspiration, several participants were interested by the co-writing potential of Dramatron, and thought it could provide them with material.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Glitches and language model limitations can be subverted for inspiration, particularly when the script is staged.

    “Glitches and language model limitations can be subverted for inspiration, in particular when the script is staged”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The co-writing approach allows both human and Dramatron to contribute to authorship of a script.

    “This co-writing approach allows the human and Dramatron to both contribute to the authorship of a script.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The participants were engaged and provided constructive feedback about Dramatron.

    “As detailed in Section 5.7, the participants were engaged and provided constructive feedback about Dramatron.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The system positively benefited and evolved over the course of the participant study.

    “In this way, the system positively benefited and evolved over the course of the participant study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Participant p13 liked the user experience of interactive, step-by-step generation of title, characters, and plot.

    “Participant p13 liked the user experience of interactive, step-by-step generation of title, characters and plot”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several participants (p6, p9, p11, p13, p15) highlighted the potential for the script to be staged after editing.

    “Several participants (p6, p9, p11, p13, p15) highlighted the potential for the script to be staged after editing”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Dramatron was designed for interactive co-writing as an augmentative tool for human writers.

    “That said, we designed Dramatron for interactive co-writing, as an augmentative tool for human writers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Participant p1 staged several scripts generated with Dramatron.

    “Participant p1 staged several scripts generated with Dramatron (see Section 5.9).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Modeling characters and their relationships was a recurrent theme in suggested improvements.

    “Modeling characters and their relationships was a recurrent theme”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    One participant had 22.5 hours of interaction over 5 sessions and co-wrote 5 scripts that were later staged.

    “Another participant had 22.5 hours of interaction in total, split over 5 sessions and spread over two months, where they co-wrote 5 scripts that were later staged.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Four scripts, co-written by a human playwright and Dramatron, were staged and shown at a large theatre festival.

    “These 4 were produced and and presented at a large theatre festival, as described in Section 5.9.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The reviewer expressed surprise at the ability of the system to create a play that hangs together and creates a world.

    “The reviewer also expressed surprise at the ability of the system to create a play that hangs together and creates a world.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Participants 4 and 5 argued against Dramatron's constrained top-down hierarchy.

    “In fact, they argued against Dramatron's constrained top-down hierarchy:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Four key themes emerged from post-show discussions with the creative team.

    “Four key themes emerged through these discussions which echo the themes presented earlier in Section 5.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In co-authorship, participants often removed longer location descriptions and portions of dialogue, particularly loops, and rewrote character descriptions.

    “During co-authorship we observed that participants tended to delete longer location descriptions and parts of dialogue (especially loops), and rewrite the character descriptions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The system has a distinct glitch style, and generated text can be repetitive and fun to work with.

    “Specifically, the system has a distinct glitch style, generated text can be repetitive and fun to work with.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    One participant commented that the generated script was only a starting point provided by the AI and that the writer still needed to create their own script.

    “One participant commented that the generated script was only a starting point provided by the AI and that the writer still needed to create their own script.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The reviewer complimented the abilities of Dramatron.

    “With a hint of incredulity, the reviewer complimented the abilities of Dramatron.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Participants would skim the text and remark on specific inspiring lines, but were less focused on the overall coherence of the discourse.

    “We observed that participants would skim the text and remark on specific inspiring lines, but were less focused on the overall coherence of the discourse.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The answers about ease of use and pride were highly correlated (r = 0.9).

    “We note that these two answers were highly correlated (𝑟 = 0.9)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The majority of performers reported that participating in the production was fun.

    “Finally, the prevailing feedback from the majority of performers was that participating in the production was fun.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Participants 4 and 5, who identified as playwrights for the stage, reported that they would never start a work with a story already in mind and might instead take an investigative approach centered on a theme.

    “Participants 4 and 5, who self-identified as playwrights for the stage,”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some lines from Dramatron were so funny they were reprised later in the show once the human actors were improvising.

    “They further noted that some lines from Dramatron are so funny they were reprised later in the show once the human actors were improvising.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the total 15 study participants, 13 provided responses on the postsession feedback form.

    “Of the total 15 study participants, 13 provided responses on our postsession feedback form.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Four participants tended to add material to Dramatron's generated outputs, while eight participants tended to remove and curate the outputs.

    “We notice that four participants tended to add to generated outputs (three of them, p6, p13, and p15 expressed interest in staging the resulting scripts) while eight others tended to remove from and curate the outputs (including p1 who staged productions).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Participants were split about the ease of use of the tool (61% agreed) and whether they felt proud about the output (46% agreed), with one participant strongly disagreeing.

    “Participants were more split about the ease of use of the tool (61% agreed) and whether they felt proud about the output of the system (46% agreed), with one participant strongly disagreeing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Generated text is sometimes repetitive.

    “Secondly, the generated text can sometimes be repetitive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several questions had a relatively strong correlation (r ≥ 0.66): helpful, collaboration, ease, enjoyment, expression, and ease.

    “and that several questions had a relatively strong correlation (𝑟 ≥ 0.66): helpful, collaboration, ease, enjoyment, expression and ease.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Dramatron's output has a distinctive glitch aesthetic that can be nonsensical, vague, or passive at times.

    “First, the system has a distinct glitch style that can sometimes be nonsensical, vague, or passive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Participants noticed some biases in the LLM outputs, particularly around gender.

    “some biases-in particular around gender-were noticed by the participants in the LLM outputs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some participants raised a third concern about the potential impact of generative tools on creative economies.

    “The third concern raised by some of the participants was about the potential impact of generative tools on creative economies:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The statements that received the strongest positive response were surprise (92% agreed, 46% strongly agreed), enjoyment (77% agreed, 69% strongly agreed), and uniqueness (69% agreed, 46% strongly agreed).

    “The statements that received the strongest positive response were I was surprised by the responses from the AI system (92% participants agreed, 46% strongly agreed) followed by I enjoyed writing with the AI system (77% agreed, 69% strongly agreed) and The script(s) written with the AI system feel unique (69% agreed, 46% strongly agreed)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    This expert expressed enthusiasm to write alongside Dramatron again for future productions.

    “This expert also expressed enthusiasm to write alongside Dramatron again for future productions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Participants did notice if the dialogue of a scene was unrelated to that scene's beat or log line, and most noticed when the generation had excessive repetition.

    “That said, participants did notice if the dialogue of a scene was unrelated to that scene's beat or log line, and most noticed when the generation had excessive repetition.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Several participants observed that Dramatron's top-down hierarchical writing style is somewhat more aligned with screenwriting than with playwriting.

    “several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Study participants raised concerns about where the dataset used to train the system came from.

    “Our study participants raised concerns the source of the dataset:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The survey responses for collaboration, helpfulness, and expression of creative goals were 77% agreed (54% strongly), 84% agreed (38% strongly), and 77% agreed (23% strongly), respectively.

    “Positive responses to statements I felt like I was collaborating with the AI system, (77% agreed, 54% strongly agreed), I found the AI system helpful (84% agreed, 38% strongly agreed) and I was able to express my creative goals with the AI system (77% agreed, 23% strongly agreed)”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Dramatron is an interactive co-writing tool that allows writers to generate scripts from a provided log line.

    “We present Dramatron: an interactive co-writing tool which allows writers to generate scripts from a provided log line.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The reviewer noted that the success of the performance was due to both the Dramatron system and the human actors.

    “The reviewer also noted that the success of the performance was due to both the Dramatron system and the human actors”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Playwrights often discover what a play is about only after finishing it, so the log line can be seen as a post-hoc summary.

    “The log line itself could be seen as a post-hoc summary, as”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The creative team attributed agency to the system and had expectations of its capabilities.

    “As well, the team attributed agency to the system, and had expectations of the systems capabilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The second review was also positive.

    “The second review was also positive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Log lines typically contain setting, protagonist, antagonist, conflict or goal, and sometimes inciting incident.

    “Log lines typically contain the setting, protagonist, antagonist, a conflict or goal, and sometimes the inciting incident [8]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] 8. Lane Shefter Bishop. 2016. Sell Your Story in A Single Sentence: Advice from the Front Lines of Hollywood. The Countryman Press.

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Large language models are limited in modeling long-range dependencies, which restricts their usefulness for longform creative writing.

    “However, such models lack long-range semantic coherence, limiting their usefulness for longform creative writing.”
    • Also stated in: § 1 INTRODUCTION · plain
      “This is a difficult task for LLMs because the narrative of a script or screenplay must exhibit long-term coherence and reincorporation, and LLMs are limited in their ability to model long-range dependencies (e.g., to reincorporate information from many pages ago).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The study engaged 15 theatre and film industry professionals in two-hour sessions to co-write scripts with Dramatron.

    “we engaged 15-experts in two-hour long user study sessions to co-write a script alongside Dramatron.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The hierarchical prompt structure and few-shot examples were engineered through iterative prompting and testing.

    “We engineered the hierarchical prompt structure and the few-shot examples through iterative prompting and testing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The study design and data collection process was validated by an ethical review board external to DeepMind.

    “Our study design and data collection process was validated by an ethical review board external to DeepMind.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The log line is a common seed of narrative inspiration for screenplays and theatre scripts.

    “Oftentimes the seed of narrative inspiration for a screenplay or theatre script is a log line [101].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [101] 101. Industrial Scripts. 2019. How to write outstanding TV & movie loglines: The ultimate guide. https://industrialscripts.com/loglines-guide/

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The incorporation of the middle layer allows the entire plot to fit within the language model's context window, overcoming prior limitations on long-term semantic coherence.

    “The incorporation of the middle layer, where the plot is summarised as a sequence of abstract scene descriptions, allows the entire plot to fit within the language models' context window.This overcomes prior limitations on long-term semantic coherence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    This work represents the largest expert user study conducted on co-creative authorship to date.

    “To the best of our knowledge, this work represents the largest expert user study conducted on co-creative authorship to date.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Large language models can give the impression of coherence within and between paragraphs but have difficulty with long-term semantic coherence due to restricted context windows.

    “LLMs can give the impression of coherence within and between paragraphs [7], but have difficulty with long-term semantic coherence due to the restricted size of their context windows.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7] 7. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021), 610-623. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study and data collection process received validation and approval from HuBREC, an ethics committee at DeepMind with external academic members and chair.

    “Our study and data collection process was validated and approved by HuBREC (Human Behavioral Research Ethics Committee), which is a research ethics committee run within Deepmind which includes and is chaired by academics from outside the company.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Interview feedback was first analyzed separately by two investigators to extract themes, and their analyses were then merged and jointly summarized into seven major themes.

    “Feedback was analysed first separately by two investigators to extract different themes, then their respective analyses were merged and jointly summarised into seven major themes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Over the course of the interviews, the authors incorporated feedback by making small, incremental changes to the prompt prefix sets of Dramatron.

    “Over the course of the interviews, we incorporate the feedback we could by making small, incremental changes to the prompt prefix sets of Dramatron.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Prompts were chosen and reworded so that LLM outputs could be parsed into components with high probability of success.

    “We chose and reworded the prompts so that the outputs of the LLM could be parsed into various components (title, character description, etc.) with high probability of success.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Chung et al. discuss how AI-based creativity support tools augment and automate parts of an artist's support network.

    “Along similar lines, Chung et al. [20] discuss how AI-based creativity support tools augment and automate portions of an artist's support network.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] 20. John Joon Young Chung, Shiqing He, and Eytan Adar. 2022. Artist Support Networks: Implications for Future Creativity Support Tools. Proceedings of Designing Interactive Systems: DIS'22 (2022). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    This reinforces the observations during interviews about criticism of log line-based conditioning for generating theatre scripts.

    “This reinforces our observations during interviews (see the criticism of log line-based conditioning for generating theatre scripts in Section 5.6).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The usefulness of LLMs as writing tools grows as they produce more coherent outputs.

    “their utility as writing tools increases as they generate more coherent outputs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    This finding aligns with observations in the interactive sessions where participants tended to delete degenerate repetitions.

    “This finding aligns with our observations in the interactive sessions where participants tended to delete degenerate repetitions”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Our distance measures are treated as signs of how much participants engaged with Dramatron.

    “We interpret our distance measures as indicators of engagement with Dramatron.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Tale Brush graphical tool introduced story arc generation, drawing on Kurt Vonnegut's theory about the protagonist's fortune over the course of a story.

    “Story arc generation was introduced in the Tale Brush graphical tool [21], using Kurt Vonnegut's theory about the fortune of the protagonist as the story progresses.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] 21. John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. 1-19. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The sentiment partially modulates how generalizable the findings are over longer term engagements.

    “While this sentiment partially modulates how generalizable the findings are over longer term engagements”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: A single engagement of co-writing alongside Dramatron might exhibit novelty effects.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Common sense and logical consistency remain elusive goals for LLMs.

    “While common sense and logical consistency is an elusive goal for LLMs [7]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] 7. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021), 610-623. source

  • May combine two claims.

    Premisecitation

    Unlike our approach, that work did not generate characters alongside a synopsis and started flat dialogue generation from manually input two-line exchanges between characters.

    “In contrast to our approach, they did not generate characters alongside synopsis and would start the”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    Positive relative Levenshtein distance scores indicate active writing with Dramatron, while negative scores indicate passive acceptance of Dramatron.

    “Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron (choosing among generated text seeds is another aspect of interaction not accounted for with this metric).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The creative team's reflections indicate that co-written scripts are useful.

    “These reflections discussed by the creative team reflect the usefulness of co-written scripts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    That work only served the production of one specific theatrical play and was not evaluated within a diverse community of writers.

    “Their work also only served the production of a specific theatrical play rather than being evaluated within a diverse community of writers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The distance measures are interpreted as indicators of engagement with Dramatron.

    “We interpret our distance measures as indicators of engagement with Dramatron.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The co-creativity of a system like Dramatron extends beyond the playwright and the system to include the performer on stage who works with the generated and edited text.

    “The co-creativity of a system such as Dramatron extends beyond the playwright and Dramatron, and to the performer on the stage working with the generated and edited text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    One interpretation is that Dramatron is seen by industry experts not as a full script writing tool but rather as a learning tool to practice writing and a source of inspiration.

    “One interpretation is that Dramatron is seen by the industry experts not as much as a full script writing tool but rather as a learning tool to practice writing and a source of inspiration”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Hierarchical generation can be used in various ways to improve the long-range coherence of long texts.

    “Hierarchical generation can be leveraged in various ways to improve the long-range coherence of long texts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    AI-based creativity support tools may need to match artists' expectations of collaboration, similar to their interactions with human collaborators, such as sub-contracting, co-creation, or inspiration.

    “These tools may need to conform to the artists' expectations of collaboration, similarly to the types of interactions they have with human collaborators: for example, sub-contracting, co-creation, or inspiration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Cultural and economic factors contribute to differences between writing screenplays and writing theatre scripts.

    “Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts:”
    Quote in the paperNot a substantial claim
103 of 103

  • The paper relies on this without support.

    The few grammatical errors made by the LLM can be fixed by the human co-writer.

    “We do not evaluate Dramatron outputs for grammatical correctness as [80], as the few errors made by the LLM can be fixed by the human co-writer.”

  • The authors state this as an assumption.

    A single engagement of co-writing alongside Dramatron might exhibit novelty effects.

    “One potentially confounding factor is that a single engagement of co-writing alongside Dramatron might exhibit some sort of novelty effects.”

  • The paper relies on this without support.

    Creative writing for theatre is fundamentally interactive between storytellers and the audience.

    “Creative writing for theatre is fundamentally interactive: not just between collaborating storytellers, but between storytellers and the audience.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Pride in the outcome was lower for wr-AI-ter (46%) than for Dramatron.

    “However, pride of outcome was lower for wr-AI-ter (46%) compared to Dramatron.”

    From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]

    Section 6.1.4 assigns the 46% pride figure to participants evaluating Dramatron, not wr-AI-ter. The paper does not make the claimed comparison between the two tools. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Uniqueness was reported by 14% of wr-AI-ter users and 69% of Dramatron users.

    “The perception of uniqueness presents a signifcant divergence, with Dramatron's outputs being considered unique by 69% of participants, in contrast to wr-AI-ter's 14%.”

    From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]

    Section 6.1.1 reports that 69% of Dramatron survey respondents agreed the scripts felt unique. Our copy does not report a 14% uniqueness rating for wr-AI-ter or a comparison between the two tools (searched: “wr-AI-ter 14% uniqueness” and “unique 69% 14% comparison”). No page number was available in the indexed passages. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Helpfulness approval was 71% for wr-AI-ter and 84% for Dramatron.

    “Both tools were seen as helpful, with Dramatron receiving an 84% approval rating in this regard, compared to wr-AI-ter's substantial 71%.”

    From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]

    Section 6.1.2 reports that 84% of respondents agreed Dramatron was helpful. The cited paper's survey results do not provide the claimed 71% helpfulness rating for wr-AI-ter, so only the Dramatron part is supported. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • wr-AI-ter produced a higher sense of ownership (57%) than Dramatron.

    “Our results showed a higher sense of ownership (57%) for the results generated by wr-AI-ter”

    From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]

    Section 6.1.5 reports low ownership of scripts co-written with Dramatron: more than half of respondents felt they did not own the result. The paper does not evaluate wr-AI-ter or report its 57% figure, which the citing sentence presents as the citing authors’ own result. It supports the Dramatron side of the comparison, but not the full comparative claim. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Ease-of-use agreement was lower for wr-AI-ter (48%) than for Dramatron (61%).

    “Ease of use was a split opinion for Dramatron (61% agreement), while wr-AI-ter experienced a lower agreement (48%).”

    From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]

    Section 6.1.4 reports that opinion on Dramatron’s ease of use was split, with 61% agreeing. The cited paper’s survey concerns Dramatron; it does not report the claimed 48% agreement for wr-AI-ter or establish the comparison. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Jaccard calculation reverses the stated meaning of similarity

    Minor

    Intersection divided by union measures similarity, not distance. Subtracting that ratio from 1 would make identical sets score 0 and disjoint sets score 1, the reverse of the paper's stated interpretation; read literally as subtracting the distance by 1, the result could also be negative.

  • Ethics committee described as both external and internal

    Minor

    The paper first calls the reviewing board external to DeepMind, but later says the approving committee is run within DeepMind. Having external academics on an internal committee does not make the committee external.

  • Relative edit distance cannot have negative scores

    Minor

    The defined metric divides a nonnegative edit distance by a text length, so it cannot be negative. The later interpretation of negative relative-distance scores conflicts with that definition.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 17

  1. Lane Shefter Bishop. 2016. Sell Your Story in A Single Sentence: Advice from the Front Lines of Hollywood. The Countryman Press.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. https://arxiv.org/abs/2005.14165 · arXiv
  3. Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study.. In Proceedings of HAI-GEN+ user2agent@ IUI.
  4. Joseph Campbell. 1949. The hero with a thousand faces. Princeton, NJ: Princeton University Press.
  5. William Wallace Cook. 1928. Plotto: The Master Book of All Plots. Ellis, first edition.
  6. Barbara Hayes-Roth and Robert Van Gent. 1996. Improvisational puppets, actors, and avatars. In Proc Computer Game Developers’ Conf.
  7. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.
  8. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning. PMLR, 957–966.
  9. William Labov and Joshua Waletzky. 1967. Narrative analysis. In Essays on the Verbal and Visual Arts, ed. J. Helm. Seattle: U. of Washington Press, 12–44.
  10. 10.5555/2891460.2891543 · DOI
  11. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc ACL Wkshp. Vol 8.
  12. James R Meehan. 1977. TALE-SPIN An Interactive Program that Writes Stories.. In IJCAI Vol. 77. 91–98.
  13. Pinelopi Papalampidi, Kris Cao, and Tomas Kocisky. 2022. Towards Coherent and Consistent Use of Entities in Narrative Generation. International Conference on Macine Learning(2022).
  14. 10.5555/1946417.1946422 · DOI
  15. Melissa Roemmele, Andrew S Gordon, and Reid Swanson. 2017. Evaluating story generation systems using automated linguistic analyses. In SIGKDD 2017 Workshop on Machine Learning for Creativity. 13–17.
  16. Mariët Theune, Sander Faas, Anton Nijholt, and Dirk Heylen. 2003. The virtual storyteller: Story creation by intelligent agents. In Proceedings of the Technologies for Interactive Digital Storytelling and Entertainment (TIDSE) Conference, Vol. 204215.
  17. Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC. 2022. AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. In Joint Proceedings of the ACM IUI Workshops 2022, Vol. 10.

Cited by

Sorted by citations

Related

Authors
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • DeepMind, Canada
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-34
DOI
10.1145/3544548.3581225
OpenAlex
W4366547384
Semantic Scholar
f9b16559282e5bea0bb072f9e260a4f0af697f4a
Cited by
240
References
57
Citations by year
2026: 58 · 2025: 77 · 2024: 82 · 2023: 22 · 2022: 1
FWCI
42.24
Type
conference paper
Language
en
Open access
Licence
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion, public-domain)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Artistic and Creative Research
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
Keywords
language models, screenwriting, theatre scripts, prompt chaining, semantic coherence, creative writing, industry professionals, plagiarism, algorithmic bias, participatory design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-30.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-30.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review