The DOI matches its Crossref record; 109 of the 109 references it cites were found in a registry or an index; 20 of the 32 sources its claims cite were checked for support.
Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals
Open accessAbstract
Language models are increasingly attracting interest from writers. However, such models lack long-range semantic coherence, limiting their usefulness for longform creative writing. We address this limitation by applying language models hierarchically, in a system we call Dramatron. By building structural context via prompt chaining, Dramatron can generate coherent scripts and screenplays complete with title, characters, story beats, location descriptions, and dialogue. We illustrate Dramatron’s usefulness as an interactive co-creative system with a user study of 15 theatre and film industry professionals. Participants co-wrote theatre scripts and screenplays with Dramatron and engaged in open-ended interviews. We report reflections both from our interviewees and from independent reviewers who critiqued performances of several of the scripts to illustrate how both Dramatron and hierarchical text generation could be useful for human-machine co-creativity. Finally, we discuss the suitability of Dramatron for co-creativity, ethical considerations—including plagiarism and bias—and participatory models for the design and deployment of such tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
20 of 32 cited sources checked · 8 supported · 2 flagged · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Because these models do not achieve long-term semantic coherence, the current state-of-the-art method for longer text generation incorporates a human-in-the-loop who selects from various generations sampled from the model [10,67,75].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.2 Limited Contexts of LLMs and Need for Human-in-the-Loop Editing
What the sources say
- Partly supportedcites [10] 10. Boyd Branch, Piotr Mirowski, and Kory W Mathewson. 2021. Collaborative Storytelling with Human Actors and AI Narrators. Proceedings of the 12th International Conference on Computational Creativity (2021). https://arxiv.org/ abs/2109.14728 sourceExperimental — accuracy not yet measured
In “Interactive live AI narration on stage” (p. 2), the paper describes an operator choosing among three GPT-3-generated sets of sentences for a long-form storytelling performance. It supports the human-selection method as an example, but does not establish that this is the current state-of-the-art method for longer text generation generally. (LLM, conf=0.86)
- Partly supportedcites [67] 67. Kory Mathewson and Piotr Mirowski. 2018. Improbotics: Exploring the imitation game using machine intelligence in improvised theatre. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14. sourceExperimental — accuracy not yet measured
Section 2.2 describes a human controller choosing among model-generated dialogue suggestions, and Section 4.3 says the resulting lines can still be narratively inconsistent. The paper does not establish that this is the state-of-the-art method for longer text generation; Section 4.5 describes its dialogue system as only one component of a larger story-generation system. Page numbers were not available in our copy. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [75] 75. Piotr Mirowski and Kory Wallace Mathewson. 2019. Human improvised theatre augmented with artificial intelligence. In Proceedings of the 2019 on Creativity and Cognition. 527-530. source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This limitation stems from the context window of LLMs, which today is limited to at most 2048 tokens (i.e. about 1500 words) in state-of-the-art models [78,86].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
What the sources say
- Partly supportedcites [78] 78. OpenAI. 2021. Pricing. https://openai.com/api/pricing/ sourceExperimental — accuracy not yet measured
Section C, “Additional model details” (page not specified), says the paper’s own language models and RL policies have a context length of 2k tokens. It does not establish that all state-of-the-art LLMs are limited to at most 2048 tokens, or equate that length with about 1500 words. (LLM, conf=0.88)
- Partly supportedcites [86] 86. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoff- mann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Su- sannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saf- fron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Lau- rent sourceExperimental — accuracy not yet measured
The Training section (p. 6) says the authors trained their models with a 2048-token context window. That does not establish a 2048-token maximum for all state-of-the-art LLMs; D.6 (p. 62) also reports extending Gopher’s context length at evaluation time beyond its training boundary. (LLM, conf=0.88)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We conducted our study using Chinchilla 70B because it outperformed other LLMs on a large collection of language tasks [50].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3 METHODS: CREATIVE TEXT GENERATION USING LARGE LANGUAGE MODELS AND DRAMATRON 3.1 Language Models
What the sources say
- Supportedcites [50] 50. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556 (2022). source
The paper’s opening summary (p. 1) identifies Chinchilla as a 70B-parameter model and states that it outperforms Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG across a large range of downstream tasks. Table 5 (p. 10) shows the breadth of the language-task evaluation. (LLM, conf=0.94)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“many participants noticed gender biases and ageism in the system outputs.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.3.2
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants found Dramatron useful for getting inspiration”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.1 Inspiration for the Writer (storytelling).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants noticed how the system could enter generation loops”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.4.2
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIn the paper
“Hierarchical generation of stories can produce an entire script-sometimes tens of thousands of words-from a single user-provided summary of the central dramatic conflict, called the log line [107].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [107] 107. Josef Steiff. 2005. The complete idiot's guide to independent filmmaking. Penguin.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By prompting the system, writers could indirectly search the language model for literary styles and elements”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.3 Using the System for Learning and Analysis (storytelling)
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.3 Inspiration from Narrative Elements and Story Arcs to Structure Script Generation To circumvent the limitations of LLMs and to enable the generation of theatre and film scripts, we take a divide-and-conquer approach: we structure script generation using interdependent narrative elements.We base our approach on Aristotle's Poetics [5], which identified 1) plot (mythos) as the most important part of a tragic story-the other elements being the 2) theme of the story (dianoia), 3) its characters (ethos), 4) dialogues (lexis), as well as melody and spectacle.Thus, we start from a user-defined theme and use Dramatron to help write characters, plot, and dialogue.Aristotle introduced the well-known plot structure (beginning, middle, and end).Many other dramatic structures followed [84].For example, variations on the Freytag's pyramid [41], popular in Western storytelling 5 (Exposition, Inciting Incident, a Rising Action composed of a series of Conflicts, Complications, and Dilemmas, Climax, Falling Action, Resolution, and Dénouement) or the Hero's Journey or Monomyth [15,113].More generally, the plot is the coherent sequence of consecutive actions that shape the story.In order to achieve narrative coherence despite the limited context 3 Anecdotally, and to verify how large the LLM needs to be to work with Dramatron, we evaluated models of different sizes by prompting them with the same log line ("A morality tale where a poor old fisher living on the cold Atlantic coast catches a magical marlin fish, who promises to grant all her wishes.Prompted by her greedy husband, the fisher asks for all fish to be turned to gold").We generated a list of characters for 50 different random seeds.Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.4 For example: https://theguardian.com/commentisfree/2020/sep/08/robot-wrote-this- article-gpt-3 5 Seminal work in narrative analysis [60,85,96] suggests general but mutable structures present in storytelling processes across many linguistic and cultural traditions (e.g.abstract, orientation, complicating action, and coda narrative clauses).The finding that narratives in many societies and in various languages make use of large, modular elements has inspired the use of models of narrative in automated storytelling, as can be seen in fields such as computational narratology [55,65].The general structures found in the Structuralist narratology of Propp [85] and the Personal Experience Narratives of Labov and Waletzky [60] are aligned with Freytag's pyramid, which we choose because it is more in line with the specific discourse genre of dramatic scripts, and arguably more familiar to the playwrights we engaged with over the course of our study.However, we note that our choice of narrative structure is not universal in scope, and is, in fact, "characteristically Western" [26,55].Alternative story shapes, or "story grammars" [97] are possible [6,17]. window of LLMs, we generate a plot synopsis in one go, as a sequence of beats.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants but p4 and p5 (who preferred a nonlinear writing workflow) were enthusiastic about the interactive hierarchical generation.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.1 Praise for the interactive hierarchical generation in Dramatron (storytelling). All participants but p4 and p5 (who preferred a nonlinear writing workflow) were enthusiastic about the interactive hierarchical generation."Once I see this, I know the shape of the series.I know the way that the story unfolds.I can see the narrative more clearly [...] I like this approach of making it a log line and then packing the detail inside it.You are planting a seed of an idea and it is putting meat on the bones" (p13)."All of it is quite consistent, symbolically consistent and coherent and relates to the state of affairs of the state of the play [...]There is lots of emotion and content about relationships in some of the generations" (p8)."In terms of the interactive co-authorship process, I think it is great [...] " (p9)."What I like about the hierarchy is that you can do as much humaning as you want at any level" (p2)."In working with the machine I can see the content a little more clearly.As there is specificity, character arcs, then I can see how the story comes together [...] This [hierarchical generation] really felt so much cleaner than the process [GPT-2 or GPT-3 with flat prompting] I was using" (p15)."Let's try more!God, you could just waste your time doing this" (p3).Participants p1, p6 and p3 further noted how such hierarchical generation helped with dialogue: "there is good content from any generation" (p1) and (referring to one of the generations) "You got some big profound discussions in it.I am impressed with that one" (p3).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Probably not a substantial claim. 85
- It may state two separate claims as one. 34
- The quote only partly states this claim. 2
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“We do not evaluate Dramatron outputs for grammatical correctness as [80], as the few errors made by the LLM can be fixed by the human co-writer.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
What the sources say
- Disputedcites [80] 80. Vishakh Padmakumar and He He. 2022. Machine-in-the-Loop Rewriting for Creative Image Captioning. In Proceedings of the 20th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). sourceExperimental — accuracy not yet measured
Section 5.1, “User Evaluation of Suggestion Quality,” explicitly asks users how grammatically correct the model suggestions were; the Results report grammaticality ratings. The source therefore does evaluate grammatical correctness, contrary to the practice the citing sentence attributes to it. Its study concerns creative image captions, not Dramatron, and the cited passages do not give human correction of a few errors as a reason to omit evaluation. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The few grammatical errors made by the LLM can be fixed by the human co-writer.
- Disputed
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Hierarchical generation of a theatre play was first mentioned and used in [93,95] for the production of AI: Can a Robot Write a Play? by company THEaiTRE in 2021 in Prague 9 .”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.2 Symbolic and Hierarchical Story Generation
What the sources say
- Disputedcites [93] 93. Rudolf Rosa, Ondřej Dušek, Tom Kocmi, David Mareček, Tomáš Musil, Patrícia Schmidtová, Dominik Jurko, Ondřej Bojar, Daniel Hrbek, David Košt'ák, et al. 2020. THEaiTRE: Artificial intelligence to write a theatre play. arXiv preprint arXiv:2006.14668 (2020). sourceExperimental — accuracy not yet measured
The paper mentions hierarchical generation in “Unresolved issues and future plans” (p. 3), but says the authors intended to use it in a future version—not that they used it for the 2021 play. The Introduction (p. 1) instead says they generated individual scenes and a dramaturge joined them into a full play. The copy does not establish that this was the first mention of hierarchical theatre-play generation. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Not checked — no full text of the sourcecites [95] 95. Rudolf Rosa, Patrícia Schmidtová, Ondřej Dušek, Tomáš Musil, David Mareček, Saad Obaid, Marie Nováková, Klára Vosecká, and Josef Doležal. 2022. GPT-2- based Human-in-the-loop Theatre Play Script Generation. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022). 29-37. source
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Characters, plot, and location descriptions all meet the specification in the log line, in addition to causal dependencies, enabled by prompt chaining [121] and explained on the diagram of Figure 1.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.4 Hierarchical Language Generation
What the sources say
- Partly supportedcites [121] 121. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. arXiv:2109.10862 [cs.CL] sourceExperimental — accuracy not yet measured
The Framework (p. 2) describes a sequence of connected steps that generates each story sentence and then repeats using the expanded context. It does not describe this as prompt chaining or show that characters, plot, and location descriptions meet a log-line specification or preserve causal dependencies. Future Work (p. 8) also treats structure-level control as a potential direction rather than an established result. (LLM, conf=0.82)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Several participants requested conversational agents to help them rewrite text-which was prescient of recently published language agents trained from human feedback [44,79].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
What the sources say
- Supportedcites [44] 44. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judge- ments. arXiv preprint arXiv:2209.14375 (2022). source
The citation supports the sentence’s reference to a recently published language agent trained from human feedback, not the citing authors’ observation about participants requesting rewriting help. The source paper’s opening page describes Sparrow as a dialogue agent trained using reinforcement learning from human feedback (p. 1). (LLM, conf=0.93)
- Supportedcites [79] 79. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022). source
The citation supports the sentence’s reference to language agents trained with human feedback, not its report about the citing study’s participants. The Introduction describes InstructGPT as a language model fine-tuned with human feedback, and Appendix A.2.1 includes an illustrative prompt asking it to rewrite text. No page numbers are provided in our copy. (LLM, conf=0.90)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 2 RELATED WORK 2.1 Automated and Crowdsourced Evaluation of the Coherence of Generated Text Echoing Celikyilmaz et al. [16], one can split evaluation methods into automated or machine-learned metrics, and humancentric evaluation.As detailed in the Appendix A.4, automated and machine-learned metrics typically calculate 1) the similarity between generated and "ground truth" stories [38,39,88,105], the 2) consistency between generated stories and their writing prompts [92,102], or 3) the diversity of language in the generated output [43,45,46,88,127].These metrics were not designed for generated text of the length of a screenplay or theatre script.This motivates a focus on human-centric evaluation, which can be conducted with non-expert crowdworkers or with experts.We therefore review the limitations of crowdsourced evaluation of the coherence of generated text, and explain why non-expert, crowdsourced evaluation faces crucial quality and bias issues.Inheriting from research standards for large-scale natural language processing tasks, the majority of studies assessing the quality of generations from LLMs evaluate model performance by collecting data from crowdworkers [58,77,80,88,91,124].For instance Yao et al. [124] recruit crowdworkers to evaluate fidelity, coherence, interestingness, and popularity, and Rashkin et al. [88] to evaluate narrative flow and ordering of events.That said, it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].These issues have led some researchers to try evaluating their models with experts.Karpinska et al. [56] highlight the perils of using crowdsourced workers for evaluating open-ended generated text, because crowdworkers do not read text as carefully as expert teachers.Some studies consulted with expert linguists [31], university students in the sciences [42] or humanities [123], and amateur writers [2,21,62,100,125].In one recent study, Calderwood et al. [13] interviewed 4 novelists about their usage of GPT-2 via Talk To Transformer and Write With Transformer tools (see https://app.inferkit.com/demo),uncovering usages of the "model as antagonist" (i.e., random), for "description creation", as "constraint", or for "unexpected" ideas-some of the themes highlighted by our own study participants.Similarly, a recent user study explored evaluation of writing short stories with a collaborative text editing tool called Wordcraft [125].
What the sources say
- Not checked — no full text of the sourcecites [36] 36. Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. In Proceedings of the eleventh ACM international conference on web search and data mining. 162-170. source
- Supportedcites [30] 30. Tim Draws, David La Barbera, Michael Soprano, Kevin Roitero, Davide Ceolin, Alessandro Checco, and Stefano Mizzaro. 2022. The Effects of Crowd Worker Bi- ases in Fact-Checking Tasks. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2114-2124. source
The citation to this paper is attached to the fact-checking portion of the sentence. The paper’s Practical Implications (§6.2) state that workers’ individual characteristics and cognitive biases can negatively affect the accuracy of crowdsourced truthfulness judgments. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [53] 53. Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-12. source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6 PARTICIPANT SURVEYS 6.1 Qualitative and Quantitative Results on Participant Surveys Of the total 15 study participants, 13 provided responses on our postsession feedback form.The form gave participants the following instruction: "When answering these questions, please reflect on the interactive co-authorship session as well as considering the use of an interactive AI system like Dramatron in the future", and asked nine questions.Each question could be answered using a Likert-type scale ranging from Strongly Disagree 1 to 5 Strongly Agree.These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]: (1) I found the AI system helpful, (2) I felt like I was collaborating with the AI system, (3) I found it easy to write with the AI system, (4) I enjoyed writing with the AI system, (5) I was able to express my creative goals while writing with the AI system, (6) The script(s) written with the AI system feel unique, (7) I feel I have ownership over the created script(s), (8) I was surprised by the responses from the AI system, and (9) I'm proud of the final outputs.We also asked five free-form questions.Two questions aimed at assessing these questions to manually define a binary indicator variable (Has experience of AI writing tools) and a three-class category for their primary domain of expertise (Improvisation, Scripted Theatre and Film/TV ).Three more questions gave participants an opportunity to provide developmental feedback about the system: What is one thing that the AI system did well?, What is one improvement for the AI system? and Please provide any comments, reflections, or open questions that came up for you during the co-authorship session or when answering this survey.Aggregated responses are plotted on Figure 5 (Left), and (Right) factors responses to the question on collaboration by industry background and experience (see Figure 7 for more).The results are summarized below.
What the sources say
- Not matchedcites Yuan et al. [125]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Stevenson et al. [108] 108. Claire Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. 2022. Putting GPT-3's Creativity to the (Alternative Uses) Test. (2022). sourceExperimental — accuracy not yet measured
The Alternative Uses Task sections (p. 2) provide a question posed to GPT-3 and instructions given to human participants, so the paper contains wording that could be adapted into questions. Our copy does not show the citing document’s survey questions, however, and cannot establish that they were only slightly adapted from this wording. (LLM, conf=0.76)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar LLM risks have been previously mentioned in [119].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.3 Undermining creative economies.
What the sources say
- Supportedcites [119] 119. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022). source
The Broader impacts section (p. 14) explicitly discusses risks from the models and techniques studied, including enabling malicious actors to train harmful models and increasing job loss through automation. It says the paper's approach could amplify those risks, supporting the general claim that such risks had been mentioned before. (LLM, conf=0.88)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Currently, the rules are unclear for machine learning-based systems trained on copyright-protected material [9].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.2 Plagiarism.
What the sources say
- Could not verifycites [9] 9. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). https://arXiv.org/abs/2108.07258 sourceExperimental — accuracy not yet measured
The Data overview (p. 18) mentions legal regulation of training data, but does not say that copyright rules for training machine-learning systems are unclear. Searches for copyright, intellectual property, fair use, and copyrighted training data found no passage addressing that point in our copy; the copy is also marked as truncated. (LLM, conf=0.70)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous work has argued for engagement with subject matter experts, literary scholars, and industry professionals in the development of machine learning models [112].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.4
What the sources say
- Partly supportedcites [112] 112. Imke Van Heerden and Anil Bas. 2021. AI as Author-Bridging the Gap Between Machine Learning and Literary Theory. Journal of Artificial Intelligence Research 71 (2021), 175-189. sourceExperimental — accuracy not yet measured
The Introduction explicitly argues for involving literary scholars in developing machine-learning models, supporting engagement with those subject-matter experts. Section 5 discusses how scholars and professionals might respond to changes in the literary industry, but does not argue that industry professionals should participate in model development. Page numbers are unavailable in our copy. (LLM, conf=0.88)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Our work of decomposing a log line into a synopsis can be seen as a narrative equivalent of Chain of Thought prompting for reasoning tasks [118], and uses the idea of using LLM output as prompt for the next stage of the generation-also called prompt chaining [121] or language engineering [24].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.3 Controllable Story Generation
What the sources say
- Could not verifycites [118] 118. Stephen G Ware and Robert Michael Young. 2011. CPOCL: A Narrative Planner Supporting Conflict.. In AIIDE. sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
- Partly supportedcites [121] 121. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. arXiv:2109.10862 [cs.CL] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [24] 24. Antonia Creswell and Murray Shanahan. 2022. Faithful Reasoning Using Large Language Models. arXiv preprint arXiv:2208.14271 (2022). source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Prompt engineering has been used to write a short plot from two characters, a genre and a theme in [54].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.3 Controllable Story Generation
What the sources say
- Supportedcites [54] 54. Yiping Jin, Vishakha Kadam, and Dittaya Wanvarie. 2022. Plot Writing From Pre-Trained Language Models. arXiv preprint arXiv:2206.03021 (2022). https: //arXiv.org/abs/2206.03021 source
The Factorizing Elements of a Story section describes prompting a language model to generate plot elements, including characters, genre, and theme. The Conflict section specifies prompts for a male and a female character, and the examples show plots generated from a plan containing those elements. The paper also includes location in the plan, but the claim does not say its list is exhaustive. (LLM, conf=0.88)
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Lee et al. [61] distinguish between verbatim, paraphrase, and idea plagiarism and suggest to automatically evaluate the first one using BM25-based retrieval of queries in large set of reference documents, along with a text alignment tool.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.2 Plagiarism.
What the sources say
- Supportedcites [61] 61. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2022. Do Language Models Plagiarize? arXiv e-prints (2022), arXiv-2203. source
Section 3.1 distinguishes verbatim, paraphrase, and idea plagiarism. Section 3.2 describes an automated detection approach that retrieves candidate documents using Elasticsearch’s default Okapi-BM25 ranking, then uses text alignment to find plagiarized passages, including verbatim plagiarism. (LLM, conf=0.92)
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most were able to co-write a full script and the open discussion interview in the allotted 2-hours.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The show's run was titled Plays By Bots and ran 7 performances over two weeks”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants had prior exposure to AI writing tools.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5 PARTICIPANT INTERVIEWS
Strength:plain· not yet calibratedn = 15Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More than merely providing a creative spark for the main story, the model can be employed to populate the universe of the story”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.2 Generation of Alternative Choices and World Building (script writing).
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As p10 reiterates, iterative rewriting was a desired workflow”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For participant 12, it was difficult to come up with a log line, and the process seemed precious.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.6 Structural Problems of Dramatron (script writing)
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several participants wanted to be able to query and dialogue with the writing model”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some participants suggested this tool could be employed in a TV writers' room, to help with writing formulaic scripts.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.5 Potential of AI as Tool for TV Screenwriting (script writing).
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This problem raised the issue of coping strategies or cultural appropriation”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.3.2
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Widely relevant feedback from the interviews included: participation of experts is critical in tool development, different use-cases demand iterative improvement and development, and different LLMs satisfy different needs from participants.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant p1 tried and discussed three different modes of script generation: 1) interactive co-authorship, 2) modifying the output from one fully automated generation, and 3) curating and modifying outputs from 3-4 generations.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.2 Ease of use of Dramatron's UI and seed-based generation.
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learning could also happen by analysing how to improve Dramatron's outputs”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.3 Using the System for Learning and Analysis (storytelling)
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our method makes it possible for elements in the final scene to provide dramatic closure on elements introduced in the opening scene 6 , and for generated stories to follow narrative arcs (see Section 3.3).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.4 Hierarchical Language Generation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Widely relevant feedback from the interviews included: participation of experts is critical in tool development, different use-cases demand iterative improvement and development, and different LLMs satisfy different needs from participants.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“three participants (p8, p13, p15) suggested fitting a narrative arc within each scene.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
Strength:plain· not yet calibratedn = 3Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The scripts were brought to life by a cast of 4-6 experienced improvisers and actors.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“There were two reviews written about the production of Plays By Bots at the festival.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Beyond inspiration, several participants were interested by the co-writing potential of Dramatron, and thought it could provide them with material.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.4 Content Generation (script writing).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Glitches and language model limitations can be subverted for inspiration, in particular when the script is staged”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.1 Inspiration for the Writer (storytelling).
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This co-writing approach allows the human and Dramatron to both contribute to the authorship of a script.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.6 Interactive Writing with Dramatron
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As detailed in Section 5.7, the participants were engaged and provided constructive feedback about Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.8 Incremental Tool Improvement
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this way, the system positively benefited and evolved over the course of the participant study.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.8 Incremental Tool Improvement
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant p13 liked the user experience of interactive, step-by-step generation of title, characters and plot”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.2 Ease of use of Dramatron's UI and seed-based generation.
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several participants (p6, p9, p11, p13, p15) highlighted the potential for the script to be staged after editing”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.3
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“That said, we designed Dramatron for interactive co-writing, as an augmentative tool for human writers.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.6 Interactive Writing with Dramatron
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant p1 staged several scripts generated with Dramatron (see Section 5.9).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.3
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modeling characters and their relationships was a recurrent theme”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another participant had 22.5 hours of interaction in total, split over 5 sessions and spread over two months, where they co-wrote 5 scripts that were later staged.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These 4 were produced and and presented at a large theatre festival, as described in Section 5.9.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > Additional information
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reviewer also expressed surprise at the ability of the system to create a play that hangs together and creates a world.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, they argued against Dramatron's constrained top-down hierarchy:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Four key themes emerged through these discussions which echo the themes presented earlier in Section 5.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“During co-authorship we observed that participants tended to delete longer location descriptions and parts of dialogue (especially loops), and rewrite the character descriptions.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the system has a distinct glitch style, generated text can be repetitive and fun to work with.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One participant commented that the generated script was only a starting point provided by the AI and that the writer still needed to create their own script.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.5
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With a hint of incredulity, the reviewer complimented the abilities of Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that participants would skim the text and remark on specific inspiring lines, but were less focused on the overall coherence of the discourse.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that these two answers were highly correlated (𝑟 = 0.9)”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.4
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, the prevailing feedback from the majority of performers was that participating in the production was fun.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants 4 and 5, who self-identified as playwrights for the stage,”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They further noted that some lines from Dramatron are so funny they were reprised later in the show once the human actors were improvising.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the total 15 study participants, 13 provided responses on our postsession feedback form.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6 PARTICIPANT SURVEYS 6.1 Qualitative and Quantitative Results on Participant Surveys Of the total 15 study participants, 13 provided responses on our postsession feedback form.The form gave participants the following instruction: "When answering these questions, please reflect on the interactive co-authorship session as well as considering the use of an interactive AI system like Dramatron in the future", and asked nine questions.Each question could be answered using a Likert-type scale ranging from Strongly Disagree 1 to 5 Strongly Agree.These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]: (1) I found the AI system helpful, (2) I felt like I was collaborating with the AI system, (3) I found it easy to write with the AI system, (4) I enjoyed writing with the AI system, (5) I was able to express my creative goals while writing with the AI system, (6) The script(s) written with the AI system feel unique, (7) I feel I have ownership over the created script(s), (8) I was surprised by the responses from the AI system, and (9) I'm proud of the final outputs.We also asked five free-form questions.Two questions aimed at assessing these questions to manually define a binary indicator variable (Has experience of AI writing tools) and a three-class category for their primary domain of expertise (Improvisation, Scripted Theatre and Film/TV ).Three more questions gave participants an opportunity to provide developmental feedback about the system: What is one thing that the AI system did well?, What is one improvement for the AI system? and Please provide any comments, reflections, or open questions that came up for you during the co-authorship session or when answering this survey.Aggregated responses are plotted on Figure 5 (Left), and (Right) factors responses to the question on collaboration by industry background and experience (see Figure 7 for more).The results are summarized below.
Strength:plain· not yet calibratedn = 15Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We notice that four participants tended to add to generated outputs (three of them, p6, p13, and p15 expressed interest in staging the resulting scripts) while eight others tended to remove from and curate the outputs (including p1 who staged productions).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
Strength:plain· not yet calibratedn = 8Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants were more split about the ease of use of the tool (61% agreed) and whether they felt proud about the output of the system (46% agreed), with one participant strongly disagreeing.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.4
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Secondly, the generated text can sometimes be repetitive.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and that several questions had a relatively strong correlation (𝑟 ≥ 0.66): helpful, collaboration, ease, enjoyment, expression and ease.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.4
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, the system has a distinct glitch style that can sometimes be nonsensical, vague, or passive.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“some biases-in particular around gender-were noticed by the participants in the LLM outputs.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.1 Biases and stereotypes.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The third concern raised by some of the participants was about the potential impact of generative tools on creative economies:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.3 Undermining creative economies.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The statements that received the strongest positive response were I was surprised by the responses from the AI system (92% participants agreed, 46% strongly agreed) followed by I enjoyed writing with the AI system (77% agreed, 69% strongly agreed) and The script(s) written with the AI system feel unique (69% agreed, 46% strongly agreed)”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.1 AI-generated scripts can be surprising, enjoyable, and unique.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This expert also expressed enthusiasm to write alongside Dramatron again for future productions.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.6 Novelty Effects.
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“That said, participants did notice if the dialogue of a scene was unrelated to that scene's beat or log line, and most noticed when the generation had excessive repetition.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study participants raised concerns the source of the dataset:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.2 Plagiarism.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Positive responses to statements I felt like I was collaborating with the AI system, (77% agreed, 54% strongly agreed), I found the AI system helpful (84% agreed, 38% strongly agreed) and I was able to express my creative goals with the AI system (77% agreed, 23% strongly agreed)”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.2
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We present Dramatron: an interactive co-writing tool which allows writers to generate scripts from a provided log line.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 8 CONCLUSIONS
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reviewer also noted that the success of the performance was due to both the Dramatron system and the human actors”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The log line itself could be seen as a post-hoc summary, as”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As well, the team attributed agency to the system, and had expectations of the systems capabilities.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The second review was also positive.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Log lines typically contain the setting, protagonist, antagonist, a conflict or goal, and sometimes the inciting incident [8]”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.3 Inspiration from Narrative Elements and Story Arcs to Structure Script Generation To circumvent the limitations of LLMs and to enable the generation of theatre and film scripts, we take a divide-and-conquer approach: we structure script generation using interdependent narrative elements.We base our approach on Aristotle's Poetics [5], which identified 1) plot (mythos) as the most important part of a tragic story-the other elements being the 2) theme of the story (dianoia), 3) its characters (ethos), 4) dialogues (lexis), as well as melody and spectacle.Thus, we start from a user-defined theme and use Dramatron to help write characters, plot, and dialogue.Aristotle introduced the well-known plot structure (beginning, middle, and end).Many other dramatic structures followed [84].For example, variations on the Freytag's pyramid [41], popular in Western storytelling 5 (Exposition, Inciting Incident, a Rising Action composed of a series of Conflicts, Complications, and Dilemmas, Climax, Falling Action, Resolution, and Dénouement) or the Hero's Journey or Monomyth [15,113].More generally, the plot is the coherent sequence of consecutive actions that shape the story.In order to achieve narrative coherence despite the limited context 3 Anecdotally, and to verify how large the LLM needs to be to work with Dramatron, we evaluated models of different sizes by prompting them with the same log line ("A morality tale where a poor old fisher living on the cold Atlantic coast catches a magical marlin fish, who promises to grant all her wishes.Prompted by her greedy husband, the fisher asks for all fish to be turned to gold").We generated a list of characters for 50 different random seeds.Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.4 For example: https://theguardian.com/commentisfree/2020/sep/08/robot-wrote-this- article-gpt-3 5 Seminal work in narrative analysis [60,85,96] suggests general but mutable structures present in storytelling processes across many linguistic and cultural traditions (e.g.abstract, orientation, complicating action, and coda narrative clauses).The finding that narratives in many societies and in various languages make use of large, modular elements has inspired the use of models of narrative in automated storytelling, as can be seen in fields such as computational narratology [55,65].The general structures found in the Structuralist narratology of Propp [85] and the Personal Experience Narratives of Labov and Waletzky [60] are aligned with Freytag's pyramid, which we choose because it is more in line with the specific discourse genre of dramatic scripts, and arguably more familiar to the playwrights we engaged with over the course of our study.However, we note that our choice of narrative structure is not universal in scope, and is, in fact, "characteristically Western" [26,55].Alternative story shapes, or "story grammars" [97] are possible [6,17]. window of LLMs, we generate a plot synopsis in one go, as a sequence of beats.
What the sources say
- Not checked yetcites [8] 8. Lane Shefter Bishop. 2016. Sell Your Story in A Single Sentence: Advice from the Front Lines of Hollywood. The Countryman Press.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, such models lack long-range semantic coherence, limiting their usefulness for longform creative writing.”
- Also stated in: § 1 INTRODUCTION · plain
“This is a difficult task for LLMs because the narrative of a script or screenplay must exhibit long-term coherence and reincorporation, and LLMs are limited in their ability to model long-range dependencies (e.g., to reincorporate information from many pages ago).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we engaged 15-experts in two-hour long user study sessions to co-write a script alongside Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We engineered the hierarchical prompt structure and the few-shot examples through iterative prompting and testing.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.5 Generation of Narrative Elements using Prompt Engineering
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study design and data collection process was validated by an ethical review board external to DeepMind.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Oftentimes the seed of narrative inspiration for a screenplay or theatre script is a log line [101].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.3 Inspiration from Narrative Elements and Story Arcs to Structure Script Generation To circumvent the limitations of LLMs and to enable the generation of theatre and film scripts, we take a divide-and-conquer approach: we structure script generation using interdependent narrative elements.We base our approach on Aristotle's Poetics [5], which identified 1) plot (mythos) as the most important part of a tragic story-the other elements being the 2) theme of the story (dianoia), 3) its characters (ethos), 4) dialogues (lexis), as well as melody and spectacle.Thus, we start from a user-defined theme and use Dramatron to help write characters, plot, and dialogue.Aristotle introduced the well-known plot structure (beginning, middle, and end).Many other dramatic structures followed [84].For example, variations on the Freytag's pyramid [41], popular in Western storytelling 5 (Exposition, Inciting Incident, a Rising Action composed of a series of Conflicts, Complications, and Dilemmas, Climax, Falling Action, Resolution, and Dénouement) or the Hero's Journey or Monomyth [15,113].More generally, the plot is the coherent sequence of consecutive actions that shape the story.In order to achieve narrative coherence despite the limited context 3 Anecdotally, and to verify how large the LLM needs to be to work with Dramatron, we evaluated models of different sizes by prompting them with the same log line ("A morality tale where a poor old fisher living on the cold Atlantic coast catches a magical marlin fish, who promises to grant all her wishes.Prompted by her greedy husband, the fisher asks for all fish to be turned to gold").We generated a list of characters for 50 different random seeds.Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.4 For example: https://theguardian.com/commentisfree/2020/sep/08/robot-wrote-this- article-gpt-3 5 Seminal work in narrative analysis [60,85,96] suggests general but mutable structures present in storytelling processes across many linguistic and cultural traditions (e.g.abstract, orientation, complicating action, and coda narrative clauses).The finding that narratives in many societies and in various languages make use of large, modular elements has inspired the use of models of narrative in automated storytelling, as can be seen in fields such as computational narratology [55,65].The general structures found in the Structuralist narratology of Propp [85] and the Personal Experience Narratives of Labov and Waletzky [60] are aligned with Freytag's pyramid, which we choose because it is more in line with the specific discourse genre of dramatic scripts, and arguably more familiar to the playwrights we engaged with over the course of our study.However, we note that our choice of narrative structure is not universal in scope, and is, in fact, "characteristically Western" [26,55].Alternative story shapes, or "story grammars" [97] are possible [6,17]. window of LLMs, we generate a plot synopsis in one go, as a sequence of beats.
What the sources say
- Not checked yetcites [101] 101. Industrial Scripts. 2019. How to write outstanding TV & movie loglines: The ultimate guide. https://industrialscripts.com/loglines-guide/
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The incorporation of the middle layer, where the plot is summarised as a sequence of abstract scene descriptions, allows the entire plot to fit within the language models' context window.This overcomes prior limitations on long-term semantic coherence.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.4 Hierarchical Language Generation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To the best of our knowledge, this work represents the largest expert user study conducted on co-creative authorship to date.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“LLMs can give the impression of coherence within and between paragraphs [7], but have difficulty with long-term semantic coherence due to the restricted size of their context windows.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.2 Limited Contexts of LLMs and Need for Human-in-the-Loop Editing
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021), 610-623. source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study and data collection process was validated and approved by HuBREC (Human Behavioral Research Ethics Committee), which is a research ethics committee run within Deepmind which includes and is chaired by academics from outside the company.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Feedback was analysed first separately by two investigators to extract different themes, then their respective analyses were merged and jointly summarised into seven major themes.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5 PARTICIPANT INTERVIEWS
Strength:plain· not yet calibratedn = 15Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Over the course of the interviews, we incorporate the feedback we could by making small, incremental changes to the prompt prefix sets of Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.8 Incremental Tool Improvement
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We chose and reworded the prompts so that the outputs of the LLM could be parsed into various components (title, character description, etc.) with high probability of success.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.5 Generation of Narrative Elements using Prompt Engineering
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Along similar lines, Chung et al. [20] discuss how AI-based creativity support tools augment and automate portions of an artist's support network.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.4
What the sources say
- Not checked — no full text of the sourcecites [20] 20. John Joon Young Chung, Shiqing He, and Eytan Adar. 2022. Artist Support Networks: Implications for Future Creativity Support Tools. Proceedings of Designing Interactive Systems: DIS'22 (2022). source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This reinforces our observations during interviews (see the criticism of log line-based conditioning for generating theatre scripts in Section 5.6).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.2
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“their utility as writing tools increases as they generate more coherent outputs.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7 DISCUSSION AND FUTURE WORK 7.1 Three Avenues of Future Work towards
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This finding aligns with our observations in the interactive sessions where participants tended to delete degenerate repetitions”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We interpret our distance measures as indicators of engagement with Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
Strength:hedged· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Story arc generation was introduced in the Tale Brush graphical tool [21], using Kurt Vonnegut's theory about the fortune of the protagonist as the story progresses.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.3 Controllable Story Generation
What the sources say
- Not checked — no full text of the sourcecites [21] 21. John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. 1-19. source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While this sentiment partially modulates how generalizable the findings are over longer term engagements”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.6 Novelty Effects.
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: A single engagement of co-writing alongside Dramatron might exhibit novelty effects.
In the paper
“While common sense and logical consistency is an elusive goal for LLMs [7]”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7 DISCUSSION AND FUTURE WORK 7.1 Three Avenues of Future Work towards
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021), 610-623. source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“In contrast to our approach, they did not generate characters alongside synopsis and would start the”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.2 Symbolic and Hierarchical Story Generation
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron (choosing among generated text seeds is another aspect of interaction not accounted for with this metric).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > C DETAILS OF QUANTITATIVE OBSERVATIONS C.1 Levenshtein Distance Levenshtein distance was calculated at the character level for edited versus non-edited texts using the default edit distance function from the Natural Language Tool Kit 11 package's distance module (no transposition operations and substitution cost equal to 1).As an absolute measure, the distance metric indicates how many operations (insertion, deletion, substitution) are required to convert one string into another and is typically calculated with a dynamic programming approach to minimizing the cost of character-level edit operations for the conversion.However, as a measure dependent on the lengths of both input strings, comparing across edit distances becomes rather uninterpretable when the string sets differ in length, for example across sets of edited and unedited texts for characters, locations, plots, and dialogues.As we are interested in understanding the extent to which participants edited the output of Dramatron as a cohort, it is reasonable to normalise the distances with respect to their length, which we report in 6 (Right).To do this, we calculate a relative Levenshtein distance as the ratio of the raw Levenshtein distance between edited and non-edited (Dramatron) output text to the length of original Dramatron output.Conceptually, we can view the resulting measure as a proportional measure of interaction with our tool.Given that Levenshtein distance operations are character-level, and length is measured in characters, the proportion represents the a weighting of active versus passive interaction with Dramatron for the different levels of structure in the hierarchical story generation process.Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron (choosing among generated text seeds is another aspect of interaction not accounted for with this metric).12
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one. The quote only partly states this claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These reflections discussed by the creative team reflect the usefulness of co-written scripts.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Their work also only served the production of a specific theatrical play rather than being evaluated within a diverse community of writers.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.2 Symbolic and Hierarchical Story Generation
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We interpret our distance measures as indicators of engagement with Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The co-creativity of a system such as Dramatron extends beyond the playwright and Dramatron, and to the performer on the stage working with the generated and edited text.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One interpretation is that Dramatron is seen by the industry experts not as much as a full script writing tool but rather as a learning tool to practice writing and a source of inspiration”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.5
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hierarchical generation can be leveraged in various ways to improve the long-range coherence of long texts.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7 DISCUSSION AND FUTURE WORK 7.1 Three Avenues of Future Work towards
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tools may need to conform to the artists' expectations of collaboration, similarly to the types of interactions they have with human collaborators: for example, sub-contracting, co-creation, or inspiration.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.4
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The few grammatical errors made by the LLM can be fixed by the human co-writer.
“We do not evaluate Dramatron outputs for grammatical correctness as [80], as the few errors made by the LLM can be fixed by the human co-writer.”
The authors state this as an assumption.
A single engagement of co-writing alongside Dramatron might exhibit novelty effects.
“One potentially confounding factor is that a single engagement of co-writing alongside Dramatron might exhibit some sort of novelty effects.”
The paper relies on this without support.
Creative writing for theatre is fundamentally interactive between storytellers and the audience.
“Creative writing for theatre is fundamentally interactive: not just between collaborating storytellers, but between storytellers and the audience.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, pride of outcome was lower for wr-AI-ter (46%) compared to Dramatron.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The perception of uniqueness presents a signifcant divergence, with Dramatron's outputs being considered unique by 69% of participants, in contrast to wr-AI-ter's 14%.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Both tools were seen as helpful, with Dramatron receiving an 84% approval rating in this regard, compared to wr-AI-ter's substantial 71%.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our results showed a higher sense of ownership (57%) for the results generated by wr-AI-ter”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Ease of use was a split opinion for Dramatron (61% agreement), while wr-AI-ter experienced a lower agreement (48%).”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The diference in the element of surprise between wr-AI-ter and Dramatron (35% vs. 92%)”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Therefore, Q9 is no longer comparable with the answers of the participants in the Dramatron survey, as LLMs were even less well known at that time.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The enjoyment levels (71% for wr-AI-ter) remain high, albeit slightly lower than Dramatron's 77%”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical studies describing how users evaluate LLMs are still a minority and difficult to generalise due to a variation of participants, methods, systems, and disciplinary perspectives [13,27,28,38,47].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The models are often described as biased.They perpetuate stereotypes and jeopardise trust in the generated content [27,38].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A final barrier was identified in the politics embedded in LLMs.”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Not only did the participants report utility for editorial tasks [27,42,47], but they also acknowledge potential for idea generation [13,38,42].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Indeed, past research involving media artists with a long history of experimenting with generative AI highlighted a more positive evaluation and stronger acceptance [38].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Key risks include potential job displacement via automation [191], generating inappropriate or harmful content [199], and disrupting creative economies, which may face user objections [122].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [122]
The cited source says what the claim attributes to it.
In the citing paper
“Dramatron's capacity for expressing creative goals was afrmed by 77% of its users, a contrast to the 21% in our study.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“After realizing that more than half of Dramatron's respondents did not feel ownership of the scripts generated by the AI”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“the sense of collaboration with the AI system was more pronounced in Dramatron's evaluation (77% agreement) compared to our fndings (43% agreement)”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“To ensure optimal comparability with the Dramatron framework, our methodology adapted the same questions highlighted in [25,35], employing 5-point Likert-type scales.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“One interpretation was that Dramatron is not seen as a professional scriptwriting tool, but rather as a learning tool to practice writing or as a resource [25].”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Additionally, our results compare favorably to those from Dramatron but are slightly less positive when compared to Wordcraft.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“of which just a fraction focused on theatre [36,38,40].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“However, many technical limitations hamper the application of LLMs to creative writing.They include short memory [36,40], lack of semantic coherence [28,40], and common-sense and contextual awareness [36,38].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“Such research has demonstrated that AI has the potential to support a variety of creative writing activities and aspects of the creative writing process.”
From Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems (2026) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“a recent work closely related to ours is the Dramatron system [70], which explores the integration of LLMs into screenwriting. However, it focuses solely on LLMs, neglecting the impact of AI-generated images and videos.”
From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [70]
The cited source says what the claim attributes to it.
In the citing paper
“However, these studies focus solely on LLMs, neglecting advancements in AI-generated images and videos.”
From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [70]
In the citing paper
“Advanced Large Language Models (LLMs) such as ChatGPT [68], GPT-4 [69], PaLM 2 [32], LLaMA 2 [90], and Gemini [73] can enable writers to generate multiple variations of text via natural language prompting [8,15,26,66], potentially reducing the workload of generating text variations.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [66]
In the citing paper
“Advanced Large Language Models (LLMs) such as ChatGPT [68], GPT-4 [69], PaLM 2 [32], LLaMA 2 [90], and Gemini [73] can enable writers to generate multiple variations of text via natural language prompting [8,15,26,66], potentially reducing the workload of generating text variations.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [66]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Jaccard calculation reverses the stated meaning of similarity
MinorIntersection divided by union measures similarity, not distance. Subtracting that ratio from 1 would make identical sets score 0 and disjoint sets score 1, the reverse of the paper's stated interpretation; read literally as subtracting the distance by 1, the result could also be negative.
“To calculate Jaccard similarity, we first calculate the Jaccard distance, which is calculated by dividing the cardinality of the intersection of two sets and by the cardinality of their union.By subtracting Jaccard distance by 1, we obtain the Jaccard similarity.”
“Jaccard metrics are scaled between 0 and 1, with 0 representing zero set similarity and 1 representing total set similarity.”
Ethics committee described as both external and internal
MinorThe paper first calls the reviewing board external to DeepMind, but later says the approving committee is run within DeepMind. Having external academics on an internal committee does not make the committee external.
“Our study design and data collection process was validated by an ethical review board external to DeepMind.”
“Our study and data collection process was validated and approved by HuBREC (Human Behavioral Research Ethics Committee), which is a research ethics committee run within Deepmind which includes and is chaired by academics from outside the company.”
Relative edit distance cannot have negative scores
MinorThe defined metric divides a nonnegative edit distance by a text length, so it cannot be negative. The later interpretation of negative relative-distance scores conflicts with that definition.
“To do this, we calculate a relative Levenshtein distance as the ratio of the raw Levenshtein distance between edited and non-edited (Dramatron) output text to the length of original Dramatron output.”
“Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Rumelhart, D. E. (1975) · Representation and Understanding · doi:10.1016/b978-0-12-108550-6.50013-6 · Cited by 1,671
Propp, V. (1968) · University of Texas Press eBooks · doi:10.7560/783911 · Cited by 1,609
Cognitive structures in comprehension and memory of narrative discourse
Thorndyke, P. W. (1977) · Cognitive Psychology · doi:10.1016/0010-0285(77)90005-6 · Cited by 1,441
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Flagged first, then by citations
Research works printed but not linked: 16
- Lane Shefter Bishop. 2016. Sell Your Story in A Single Sentence: Advice from the Front Lines of Hollywood. The Countryman Press.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. https://arxiv.org/abs/2005.14165 · arXiv
- Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study.. In Proceedings of HAI-GEN+ user2agent@ IUI.
- Joseph Campbell. 1949. The hero with a thousand faces. Princeton, NJ: Princeton University Press.
- William Wallace Cook. 1928. Plotto: The Master Book of All Plots. Ellis, first edition.
- Barbara Hayes-Roth and Robert Van Gent. 1996. Improvisational puppets, actors, and avatars. In Proc Computer Game Developers’ Conf.
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.
- Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning. PMLR, 957–966.
- William Labov and Joshua Waletzky. 1967. Narrative analysis. In Essays on the Verbal and Visual Arts, ed. J. Helm. Seattle: U. of Washington Press, 12–44.
- 10.5555/2891460.2891543 · DOI
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc ACL Wkshp. Vol 8.
- James R Meehan. 1977. TALE-SPIN An Interactive Program that Writes Stories.. In IJCAI Vol. 77. 91–98.
- Pinelopi Papalampidi, Kris Cao, and Tomas Kocisky. 2022. Towards Coherent and Consistent Use of Entities in Narrative Generation. International Conference on Macine Learning(2022).
- 10.5555/1946417.1946422 · DOI
- Melissa Roemmele, Andrew S Gordon, and Reid Swanson. 2017. Evaluating story generation systems using automated linguistic analyses. In SIGKDD 2017 Workshop on Machine Learning for Creativity. 13–17.
- Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC. 2022. AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. In Joint Proceedings of the ACM IUI Workshops 2022, Vol. 10.
Cited by
Sorted by citations
Items 101–200 of 237
Evaluating Creativity Support Tools via Homogenization Analysis
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651088 · Cited by 11
“Notably, the TTCT does not include artifact quality as an evaluation criterion; where studies of CSTs have attempted to evaluate output quality, human raters have usually been employed to judge the results (e.g., [15, 18, 24, 36]).”
MethodThe Dearth of the Author in AI-Supported Writing
Kreminski, M. (2024) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.1145/3690712.3690725 · Cited by 13
Making the Write Connections: Linking Writing Support Tools with Writer Needs
Zhao, Z., Masson, D., Kim, Y., Penn, G., & Chevalier, F. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713161 · Cited by 21
“…has explored novel approaches focusing on one specific aspect of the creative writing process at a time, such as exploring character personalities using chatbots [85], authoring interactive narratives [8, 56], writing on phones [11], and, more recently, co-writing with large language models [73].”
InfluentialBiswas, S., Erlei, A., & Gadiraju, U. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713201 · Cited by 17
“LLMs themselves have demonstrated exceptional performance in open-ended writing, with great potential benefit for a wide variety of tasks [51, 71, 78, 81, 83, 132, 139].”
Chen, Y., Wang, Y., Chan, A. B., Li, J., & Lc, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92823-9_7 · Cited by 11
Wambsganß, T., Su, X., Swamy, V., Neshaei, S. P., Rietsche, R., & Käser, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.689 · Cited by 10
Wasi, A. T., Rafi, T. H., & Chae, D. (2024) · ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) · doi:10.1145/3678884.3681862 · Cited by 10
“LLMsshowpromiseinassistingcreativewritinginEnglish, such as screenplays [19], short stories [28], academic writing [4, 12], and legal documents [21].”
Character Animation Pipeline based on Latent Diffusion and Large Language Models
Clocchiatti, A., Fumerò, N., & Soccini, A. M. (2024) · IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR) · doi:10.1109/aixvr59861.2024.00067 · Cited by 9
“Several approaches were proposed to simplify or improve film production, including storyboard generation [1], screenplay writing [2] and AI motion capture [3] among others.”
BackgroundScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie Database
Rao, A., Chou, J.-P., & Agrawala, M. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676402 · Cited by 13
Challenges and Opportunities of LLM-Based Synthetic Personae and Data in HCI
Prpa, M., Troiano, G. M., Yao, B., Li, T. J.-J., Wang, D., & Gu, H. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3681826 · Cited by 16
“These include using LLMs for software engineering [9], writing stories and screen-plays [22, 57], public health interventions [13], summarizing medical records [39], and LLM-powered creativity support tools [4].”
Why So Serious? Exploring Timely Humorous Comments in AAC Through AI-Powered Interfaces
Weinberg, T. M., Kadoma, K., Penuela, R. E. G., Valencia, S., & Roumen, T. J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714102 · Cited by 11
“Similarly, Higginbotham and Wilkins [25] showed how mismatched timing in conversations disrupts the flow and creates barriers to effective communication.”
LIGS: Developing an LLM-infused Game System for Emergent Narrative
Jeong, J., & Lee, T. Y. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720212 · Cited by 14
You, W., Lu, Y., Ma, Z., Li, N., Zhou, M., Zhao, X., Chen, P., & Sun, L. (2025) · ACM Transactions on Graphics · doi:10.1145/3730919 · Cited by 13
CoCo Matrix: Taxonomy of Cognitive Contributions in Co-writing with Intelligent Agents
Wan, R., Gebreegziabher, S. A., Li, T. J.-J., & Badillo-Urquiola, K. (2024) · Creativity and Cognition · doi:10.1145/3635636.3664260 · Cited by 9
“Writers provide inputs in diverse forms and levels of abstraction, such as drafts [17, 41, 57], writing instructions [38], or a mix [12, 56], expanding the scope of agents for content generation.”
BackgroundCreative Robotics Theatre: Designing Creative Interactions with Tangible and Embodied Interfaces
Samani, H. A., Lalioti, V., Serbanescu, D. A., Chicau, J., Polydorou, D., Rodosthenous, G., Knowlson, A., Bakalos, Y., Neale, M., & Indurkhya, B. (2024) · Designing Interactive Systems Conference · doi:10.1145/3656156.3658387 · Cited by 9
“Existing methods include live action role-playing and scenario en-actment [23], bodystorming [13], choreographic techniques [6] amidst other examples drawn from performance and theatre practices [20, 21, 29, 30].”
MethodSARD: A Human-AI Collaborative Story Generation
Radwan, A. Y., Alasmari, K. M., Abdulbagi, O. A., & Alghamdi, E. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76806-4_7 · Cited by 17
Fu, J., Wang, X., Vi, K., Li, Z., Xu, C., & Sun, Y. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3721163 · Cited by 8
“For example, tools like Dramatron [8] can assist screenwriters in crafting theatre plays using LLMs.”
Mishra, A., Brudy, F., Zhou, Q., Fitzmaurice, G. W., & Anderson, F. (2025) · Creativity and Cognition · doi:10.1145/3698061.3726933 · Cited by 18
Song, J., Ashktorab, Z., Pan, Q., Dugan, C., Geyer, W., & Malone, T. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757486 · Cited by 11
“In creative fields such as music [35] and storytelling [36, 52], the collaboration between humans and artificial intelligence (AI) is fostering innovation.”
StoryDiffusion: How to Support UX Storyboarding With Generative-AI
Liang, Z. W., Zhang, X., Ma, K., Liu, Z., Ren, X., Goucher-Lambert, K., & Liu, C. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750793 · Cited by 24
“Existing systems for AI-assisted storytelling either focus on script writing or frame-by-frame text-to-image generation [24, 28], with less explored in how to support designers through the entire creative process involving both aspects, as well as how designers would apply their system to various…”
Designing a Language Model-Based Authoring Tool Prototype for Interactive Storytelling
Park, J., Shin, J., Kim, G., & Bae, B.-C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47658-7_22 · Cited by 10
Fine-Tuning Large Language Models in Education
Chen, Y., Chen, H., & Su, S. (2023) · International Conference on Information Technology in Medicine and Education · doi:10.1109/itme60234.2023.00148 · Cited by 13
Tsai, H.-R., Chiu, S.-K., & Wang, B. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714294 · Cited by 9
“Several research [11, 45] offer users the flexibility to interact with the script generation process by LLMs and enrich the co-creation experience with LLMs in writing.”
Zhou, P., Zhu, A., Hu, J., Pujara, J., Ren, X., Callison-Burch, C., Choi, Y., & Ammanabrolu, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.624 · Cited by 27
“Theater script co-writing has also been studied recently (Mirowski et al., 2022) for the simulation of a small-scale world.”
BackgroundUnderstanding the Potentials and Limitations of Prompt-based Music Generative AI
Choi, Y., Moon, J., Yoo, J., & Hong, J.-H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713762 · Cited by 12
“Recently, the emergence of LLMs has made deep learning more accessible to creators through natural language interfaces, facilitating the integration of prompt-based GenAI into creative processes [29, 48, 52, 54, 77, 81].”
Emails by LLMs: A Comparison of Language in AI-Generated and Human-Written Emails
Li, W., Lai, Y., Soni, S., & Saha, K. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3717867.3717872 · Cited by 10
Understanding Reader Perception Shifts upon Disclosure of AI Authorship
Nakano, H., Takezawa, J., Matulic, F., Yang, C.-L., & Yatani, K. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789076 · Cited by 17
“For instance, Schilke and Reimann [52] found that people with more positive attitudes toward AI show a smaller decline in trust upon disclosure.”
SimTube: Simulating Audience Feedback on Videos using Generative AI and User Personas
Hung, Y., Huang, Y.-C., Su, T.-Y., Lin, Y.-T. R., Cheng, L.-P., Wang, B. K., & Sun, S.-H. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712146 · Cited by 18
“LLMs, trained on vast datasets and knowledge [7, 57], have demonstrated the ability to perform tasks once thought to be exclusive to humans, e . g ., reasoning [46, 84, 91] and creative writing [41, 48, 54].”
Nabokov's Cards: An AI Assisted Prewriting System to Support Bottom-Up Creative Writing
Carrera, D., Zhao, Z., Ajin Thomas, A., & Wigdor, D. (2025) · Proceedings of the 2025 Conference on Creativity and Cognition · doi:10.1145/3698061.3726916 · Cited by 6
“Prior works on prewriting tools introduce ways to visually organize writing plans [9, 37] and aid ideation [14, 15, 19, 25, 40, 71].”
La Revolución en la Creación Visual
Arias, M. C., Díaz, A. P., & Lara-Martínez, M. (2024) · VISUAL REVIEW International Visual Culture Review / Revista Internacional de Cultura Visual · doi:10.62161/revvisual.v16.5304 · Cited by 12
Evaluating GPT for use in K-12 Block Based CS Instruction Using a Transpiler and Prompt Engineering
Gonzalez-Maldonado, D., Liu, J., & Franklin, D. M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701910 · Cited by 6
Amin, R. M., Kühle, O. H., Buschek, D., & Butz, A. M. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720243 · Cited by 6
Follow My Logic: Generative AI Workflows in Designing for Serious Table-Top Games
Ma, L., Pan, M., Siu, V., Chang, X., Holopainen, J. P., Li, J., & Lc, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92823-9_11 · Cited by 6
Infinity Book: Speculating Literary Expressions in the Age of Generative AI
Holopainen, J. P., Huang, Y., Shin, J., Nissinen, E., & Lucero, A. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735735 · Cited by 8
“Since then, numerous studies have evaluated GenAI’s ability to facilitate automatic literary creation, with varying results [21, 41, 52, 82, 92, 112].”
AI in the Writing Process: How Purposeful AI Support Fosters Student Writing
Siddiqui, M. N., Feliciano, V. A., Pea, R. D., & Subramonyam, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98459-4_14 · Cited by 11
Maier, S., Schneider, M., & Feuerriegel, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791185 · Cited by 26
“Third , while LLMs might support the quality of creative output, their dominant role in shaping ideas may undermine users’ perceived ownership [19, 39, 54, 62, 66, 87].”
Mo, W., Singh, A., & Holloway, C. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675609 · Cited by 8
“With just a few prompts, ChatGPT enables the rapid creation of scripts [53], student performance assessment, and teacher development [21, 74].”
How Do Human Creators Embrace Human-AI Co-Creation? A Perspective on Human Agency of Screenwriters
Tang, Y., Zhou, J., Li, H., Xie, X., Ma, X., & Qu, H. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790300 · Cited by 5
“The increasing integration of AI in screenwriting transforms not only the quality and style of final outputs [43, 46] but also the practices through which story ideas are conceived and developed [31, 32, 54, 63].”
Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and Occupations
Koonchanok, R., Pan, Y., & Jang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.12951 · Cited by 5
“It has captured people’s imagination about a revolutionized future of programming (Chen et al. 2021; Li et al. 2022), writing (Mirowski et al. 2022; Lee et al. 2022), and many more (Kocoń et al. 2023; Fatouros et al. 2023).”
Preserving Writer Values in AI Writing Assistance Tools
Guo, A., Wang, L., Heer, J., & Zhang, A. X. (2024) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.1145/3690712.3690727 · Cited by 6
“Recent large language models (LLMs) have demonstrated significant advances in the area of creative writing applications, ranging from poems [2] and video scripts [15] to screenplays [10].”
Chung, J. G., Hong, S., Choi, J., & Oh, C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713532 · Cited by 8
“In theater, notable studies have demonstrated the feasibility of human-AI co-creation by involving multiple play-wrights in the collaborative development of scripts for theatrical performances [101].”
FlowGPT: Exploring Domains, Output Modalities, and Goals of Community-Generated AI Chatbots
Li, X., Han, Y., Liu, D., An, P., & Niu, S. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3681875 · Cited by 5
“Based on our annotation of chatbots on FlowGPT using existing chatbot categorizations, we discuss the common use of the platform for sharing user-generated chatbots.”
Qadri, R., Mirowski, P., Gabriellan, A., Mehr, F., Gupta, H., Karimi, P., & Denton, R. (2024) · arXiv · doi:10.48550/arxiv.2412.14077 · Cited by 5
“…get closer to understanding AI as a socio-technical artifact, these studies have primarily centered on conversations with artists through interviews, with only a handful studies focusing on artists experimenting with tools directly or bringing artists into conversation with one [18, 19, 21–24].”
Self-citationCo-Creating Question-and-Answer Style Articles with Large Language Models for Research Promotion
Lim, H., Cho, J. Y., Kim, T., Park, J., Shin, H., Choi, S., Park, S., Lee, K., Kim, J., Lee, M., et al. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660705 · Cited by 7
“Similarly, other studies have found that human-in-the-loop editing is necessary for targeted writing in context, such as writing screenplay scripts that tell a coherent story with well-developed characters and scenes [43].”
BackgroundInfluentialKim, D. H., Jeong, D., Yadgarova, S., Shin, H., Son, J., Subramonyam, H., & Kim, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714044 · Cited by 4
Imteyaz, K., Flores-Saviaga, C., & Savage, S. (2024) · Avances en Interacción Humano-Computadora · doi:10.47756/aihc.y9i1.159 · Cited by 7
Endless forms most similar: the dearth of the author in AI-supported art
Kreminski, M. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02326-6 · Cited by 4
A Systematic Literature Review of Online Collaborative Story Writing
Montanelli, S., & Ruskov, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42286-7_5 · Cited by 8
Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems
Carrera, D., Thomas-Mitchell, J., & Wigdor, D. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791952 · Cited by 3
“Others used tool-like interfaces that were not found in the commercial systems, such as a tool for the generation of stories containing target words for language learning [54], and hierarchical generation [46].”
Theatre dialogues with machines
Lösel, G. (2024) · Interdisciplinary Science Reviews · doi:10.1177/03080188241257882 · Cited by 3
Tracing Creativity: A Design Space For Creative Activity Traces in HCI
Hammad, N., Lin, D. C.-E., Smith, A., Kreminski, M., Harpstead, E., & Hammer, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791263 · Cited by 10
Investigating How Computer Science Researchers Design Their Co-Writing Experiences With AI
Roffarello, A. M., Calò, T., Scibetta, L., & Russis, L. D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713205 · Cited by 5
“As previous research has clearly demonstrated, the overuse of AI-generated texts might be negatively perceived by writers, as it undermines their sense of agency and ownership [27, 44].”
Exploring the Creative Potential of AI in Filmmaking
Pan, Z., Benini, S., Grierson, M., Savardi, M., & Smith, T. J. (2025) · Creativity and Cognition · doi:10.1145/3698061.3728393 · Cited by 3
“Inscreenwriting, AI-assisted tools generate script drafts and dialogues and suggest alternative plotlines, enabling filmmakers to iterate rapidly and explore multiple narrative possibilities [34].”
DuoDrama: Supporting Screenplay Refinement Through LLM-Assisted Human Reflection
Tang, Y., Chen, X., Li, H., Xie, X., Ma, X., & Qu, H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790568 · Cited by 2
“AI has been integrated into screenwriting practice for audiovisual production (e.g., films and dramas) [28, 38, 60], assisting screenwrit-ers in generating materials during creative processes.”
Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
Chen, R., Jiang, S., Shen, J., Moon, A., & Wei, L. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00016 · Cited by 3
Creative Natural Language Generation
Chakrabarty, T., Padmakumar, V., He, H., & Peng, N. (2023) · doi:10.18653/v1/2023.emnlp-tutorial.6 · Cited by 5
“…be pulled from work by other researchers in NLP and machine learning communities include but not limited to (Ghazvininejad et al., 2016; Fan et al., 2018b, 2019; Van de Cruys, 2020; Riedl and Young, 2010; Lin and Riedl, 2021; Brahman and Chaturvedi, 2020; Mirowski et al., 2023; Clark et al., 2021).”
BackgroundInk and Individuality: Crafting a Personalised Narrative in the Age of LLMs
Wasi, A. T., Islam, R., & Islam, M. R. (2024) · Proceedings of the Third Workshop on Intelligent and Interactive Writing Assistants · doi:10.1145/3690712.3690724 · Cited by 2
“LLMs show potential for helping users with a range of creative writing activities, including screenplays [15] and short stories [23, 25].”
BackgroundBeyond Pre-Defined Scripts: Player Perceptions on Generative Non-Player Character Dialogues
Hochreiter, M., Kriglstein, S., & Wallner, G. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789221 · Cited by 7
Chakrabarty, T., & Dhillon, P. S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791276 · Cited by 6
“Beyond standard professional writing, LLMs have found applications in both scientific research [24, 35, 41, 42] and creative endeavors, including collaborative fiction writing and other artistic text generation [9, 13, 31, 50, 51, 70].”
Li, Y. L., Kuang, Q., Xiao, X., Labrune, J., & Ishii, H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791258 · Cited by 1
From Crafting Text to Crafting Thought: Grounding AI Writing Support to Writing Center Pedagogy
Liu, Y., Gallagher, J., Sterman, S., & August, T. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790292 · Cited by 4
“P5 highlighted the comparative value: “students can use this tool and then come into the writing center, and then compare the feedback... they can also analyze like, hey, how is this conversation with the [system] different from the conversation that I have with my tutor.”
Abhishek, A., Akhilesh, R., Swetha, B. N., et al. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-79041-6_9 · Cited by 1
LLMAGENTNET: A COLLABORATIVE NETWORK OF AUTONOMOUS AI AGENTS FOR COMPLEX TASK EXECUTION
Бідочко, А., & Виклюк, Я. І. (2025) · Науковий вісник НЛТУ України · doi:10.36930/40350412 · Cited by 3
Script2Screen: Supporting Dialogue-Centric Scriptwriting with Interactive Audiovisual Generation
Wang, Z., Ma, J., Grinspun, E., Grossman, T., & Wang, B. K. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789075 · Cited by 2
“Script2Screen is related to works in writing support tools, audio storytelling, and integrative tools for text and audio-visual media.”
Constella: Supporting Storywriters’ Interconnected Character Creation through LLM-Based Multi-Agents
Park, S., Park, S., & Lim, Y.-K. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3796234 · Cited by 6
Du, J., Liu, W., Zhou, D., Hong, S., & Liu, F. (2025) · Informatics · doi:10.3390/informatics12040139 · Cited by 1
“AI has significantly transformed cultural creation, enabling the generation of paintings [46], music [47], and poetry [48], as well as the development of screenplays and theatrical scripts [49,50].”
Ding, Z., Zhang, Q., Chi, M., & Wang, Z. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00011 · Cited by 12
“This anxiety has been particularly pronounced in creative industries, where screenwriters face competition from text-generation models [3] and illustrators contend with text-to-image models [4]–[6].”
FeedQUAC: Quick Unobtrusive AI-Generated Commentary
Long, T., Wannamaker, K., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2025) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3765766.3765861 · Cited by 11
“Despite these benefits, gathering constant feedback is often troublesome due to constraints like availability and individuality of the feedback providers [16, 31].”
ClearDose: AI Overfunctioning at the Boundaries of Care
Han, Y.-N., & Leigh, S.-W. (2026) · International Conference on Tangible, Embedded, and Embodied Interaction · doi:10.1145/3731459.3779135 · Cited by 1
“For instance, while generative AI could collaborate with users for co-creation (e.g., [11, 12, 14]) and provide emotional companionship (e.g., [15]), it may also deliver unsafe and biased content to users [22], erode users’ agency [9], and harm users’ well-being [13].”
Atalay, M. O., Alpdemir, M. N., Özkan, A. G., et al. (2026) · International Informatics and Software Engineering Conference (IISEC) · doi:10.1109/iisec69317.2026.11418405 · Cited by 1
Playing the Monograph: Retrieval-Augmented Generation for Interactive Ethnography
Hoffmann, M. P., John, J., Fillies, J., & Paschke, A. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798374 · Cited by 1
“Recent advances in generative AI have drawn attention to the expanding role of large language models (LLMs) in creative production, including creative writing, storytelling, and theatrical generation, as well as emerging applications in game design [1, 6, 15, 16, 20].”
Qi, P., & Zhu, W. (2026) · Sustainability · doi:10.3390/su18126117 · Cited by 1
Controllable Story Generation Based on Perplexity Minimization
Vychegzhanin, S., Kotelnikova, A., Sergeev, A., & Котельников, Е. В. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54534-4_11 · Cited by 1
How different mental models of AI-based writing assistants impact writers’ interactions with them
Rismani, S., Blodgett, S. L., Olteanu, A., Liao, Q. V., & Moon, A. (2024) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.1145/3690712.3690722 · Cited by 1
The application of artificial intelligence in film and television script creation
Xu, Z., & Xie, B. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00741-8 · Cited by 1
RDoFlow: Automatically assessing under-specified statistical analyses in HCI
Grunde-McLaughlin, M., Liu, W., Patil, R., Migineishvili, N., Reif, E., Krishna, R., Weld, D. S., & Heer, J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789160
“Self-verifcation techniques include running the same subtask via multiple models [23, 62, 104] and using a sequential LLM call to verify prior LLM outputs [69, 103, 105, 106].”
NarrativeLoom: Enhancing Creative Storytelling through Multi-Persona Collaborative Improvisation
Ma, Y., Peng, Y., Yang, F., Zha, S., Zhang, C., Jia, Z., Zheng, Z., & Zhu, Y. B. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790416 · Cited by 1
Beyond the Spotlight: Co-Designing AI for Theatre Audience Communication
Chitayat, A. P., Carter, S., Robertson, B. A., & Hook, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791710
Narrix: Remixing Narrative Strategies from Examples for Story Writing
Zhang, C., Guo, S., Davis, A., & Koh, E. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790813 · Cited by 4
“…through dust-and-magnet metaphors [15]; VISAR [102] and Polymind [87] provides node-based visual programming for interactive co-writing; Dramatron [62] ofers hierarchical story generation, enabling users to control screenplay generation across diferent narrative elements (e.g., characters, plots,…”
From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants
Rismani, S., Blodgett, S. L., Liao, Q. V., Olteanu, A., & Moon, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791670 · Cited by 1
“In this work, we focus on this question in the context of AI-based writing assistants, which are now widely used across domains ranging from creative writing to professional and scientific communication [1, 28, 30, 32, 34, 38, 52, 78].”
Vidmento: Creating Video Stories through Context-Aware Expansion with Generative Video
Yeh, C., Truong, A., Dontcheva, M., & Wang, B. K. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791337
“While LLMs can produce full, coherent stories (e.g., screenplays in Dramatron [65]), our formative study participants preferred drafting their own video scripts to preserve narrative intent, in line with prior work [116].”
Sci-Fi Spark: A Human-AI Co-Creation System for Science Fiction Ideation
Jiang, Z., Cheang, W., Xu, X., Zuo, H., Zheng, C., & Chen, L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791950 · Cited by 1
“In addition to novelty, LLM-generated text often lacks consistency and coherence [59, 81].”
Zhang, Y., Jia, K., Zhang, H. J., Zhu, K., Meng, C., Zhang, J., Li, Z., Chen, P., & Sun, L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791460 · Cited by 1
“Dramatron uses hierarchical prompt chains to structure stories—characters, plot beats, and dialogue—mitigating long-range semantic alignment issues and demonstrating the value of structured interaction in literary production [67].”
Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection
Ahuja, K., Sclar, M., & Tsvetkov, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.11900 · Cited by 1
Enhancing Reasoning Pathways for Power Defect Analysis Using CoT and Role-Play Prompt
Xu, J., Cao, X., Li, T., He, F., Yao, S., Jia, B., Zhang, H., & Cai, S. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0827-7_30 · Cited by 1
Tasks for LLMs and Their Evaluation
Ostapuk, N., & Audiffren, J. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_6 · Cited by 2
Fine-Tuning Open-Source LLMs for Plot Generation in Korean Popular Culture Content
Kim, J.-S., & Kim, M.-J. (2024) · Journal of Digital Contents Society · doi:10.9728/dcs.2024.25.11.3167
MM-IGLU-IT: Multi-modal Interactive Grounded Language Understanding in Italian
Borazio, F., Hromei, C. D., Passone, E., Croce, D., & Basili, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-80607-0_6
“These models can answer questions, generate narratives, and interpret natural language and images [14, 23, 35, 39].”
From Tokens to Tales: Semantic Similarity in Story Generation
Vychegzhanin, S., Kotelnikova, A., Sergeev, A., & Котельников, Е. В. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88036-0_9
Generation and Evaluation of Twisted-Plot Stories Using Large Language Models
Park, J., & Bae, B.-C. (2025) · Journal of Digital Contents Society · doi:10.9728/dcs.2025.26.3.763
Weber, C. J., Huang, J., & Rothe, S. (2025) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3706370.3727862
PTCL-SE: How AI Self-evaluation Influences Human-AI Collaborative Story Writing
Cao, X. (2024) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid63852.2024.00058
“Since melody and spectacle are unique to the music and performance of comedy, we have not incorporated these two aspects into the evaluation framework [18].AI will assess its generated content based on these dimensions, identifying both highlights and short-comings.”
Creative-Agent: A Creative Prototype Generation System Driven by Objectives and Key Results
Zheng, Y., Huang, H., Ma, C., et al. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033384
Vieses em Histórias Ficcionais no ChatGPT
Ribeiro, T. M. R., Siqueira, S. W. M., & de Bayser, M. G. (2025) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc_estendido.2025.8585
Co-Direct: A Knowledge-Augmented Multi-Agent Framework for Interactive Drama Script GenerationPreprint
yang, B., Peng, J., Tang, S., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5387181
Setting the Stage for Fictional and Artificial Characters
Angeli, A. D., & Grigis, P. (2025) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3750069.3750155
“Designing systems for creative writing remains a challenging and unresolved goal [21, 26, 29].”
InfluentialFang, X., Yuan, H. R., Li, H., Kou, J., Gu, C., Zhang, W., Duan, X., & Fang, Y. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189217 · Cited by 4
“A study with fifteen theatre and film professionals reported productivity gains and innovative plot twists under the assistance of DeepMind’s Dramatron [57].”
Fan, G. F., Liu, D., Pan, L., & Huang, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229343
Artificial Intelligence Applications in TV Industry Workflow
Guedes, Á. L. V. (2025) · doi:10.5753/webmedia_estendido.2025.16796
“More recently in 2024, Mirowski et al. [12] proposed "Dramatron" a system that uses Large Language Model (LLM) to generate scripts and screenplays hierarchically.”
Metadata
- Authors
- Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, Canada
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-34
- OpenAlex
- W4366547384
- Semantic Scholar
- f9b16559282e5bea0bb072f9e260a4f0af697f4a
- Cited by
- 240
- References
- 57
- Citations by year
- 2026: 58 · 2025: 77 · 2024: 82 · 2023: 22
- FWCI
- 42.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- public-domain
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Artistic and Creative Research
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
- Keywords
- language models, screenwriting, theatre scripts, prompt chaining, semantic coherence, creative writing, industry professionals, plagiarism, algorithmic bias, participatory design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
