The DOI matches its Crossref record; 109 of the 109 references it cites were found in a registry or an index; 20 of the 32 sources its claims cite were checked for support.
Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals
Open accessAbstract
Language models are increasingly attracting interest from writers. However, such models lack long-range semantic coherence, limiting their usefulness for longform creative writing. We address this limitation by applying language models hierarchically, in a system we call Dramatron. By building structural context via prompt chaining, Dramatron can generate coherent scripts and screenplays complete with title, characters, story beats, location descriptions, and dialogue. We illustrate Dramatron’s usefulness as an interactive co-creative system with a user study of 15 theatre and film industry professionals. Participants co-wrote theatre scripts and screenplays with Dramatron and engaged in open-ended interviews. We report reflections both from our interviewees and from independent reviewers who critiqued performances of several of the scripts to illustrate how both Dramatron and hierarchical text generation could be useful for human-machine co-creativity. Finally, we discuss the suitability of Dramatron for co-creativity, ethical considerations—including plagiarism and bias—and participatory models for the design and deployment of such tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
72 Own result · 22 Own interpretation · 13 No basis given · 8 Own method · 6 Partly supported · 5 Supported · 5 No full text of the sources · 4 Not checked yet · 2 Not supported · 1 Supported so far · 1 Couldn’t confirm · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Because these models do not achieve long-term semantic coherence, the current state-of-the-art method for longer text generation incorporates a human-in-the-loop who selects from various generations sampled from the model [10,67,75].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.2 Limited Contexts of LLMs and Need for Human-in-the-Loop Editing
What the sources say
- Partly supportedcites [10] 10. Boyd Branch, Piotr Mirowski, and Kory W Mathewson. 2021. Collaborative Storytelling with Human Actors and AI Narrators. Proceedings of the 12th International Conference on Computational Creativity (2021). https://arxiv.org/ abs/2109.14728 sourceExperimental — accuracy not yet measured
In “Interactive live AI narration on stage” (p. 2), the paper describes an operator choosing among three GPT-3-generated sets of sentences for a long-form storytelling performance. It supports the human-selection method as an example, but does not establish that this is the current state-of-the-art method for longer text generation generally. (LLM, conf=0.86)
- Partly supportedcites [67] 67. Kory Mathewson and Piotr Mirowski. 2018. Improbotics: Exploring the imitation game using machine intelligence in improvised theatre. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14. sourceExperimental — accuracy not yet measured
Section 2.2 describes a human controller choosing among model-generated dialogue suggestions, and Section 4.3 says the resulting lines can still be narratively inconsistent. The paper does not establish that this is the state-of-the-art method for longer text generation; Section 4.5 describes its dialogue system as only one component of a larger story-generation system. Page numbers were not available in our copy. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [75] 75. Piotr Mirowski and Kory Wallace Mathewson. 2019. Human improvised theatre augmented with artificial intelligence. In Proceedings of the 2019 on Creativity and Cognition. 527-530. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This limitation stems from the context window of LLMs, which today is limited to at most 2048 tokens (i.e. about 1500 words) in state-of-the-art models [78,86].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
What the sources say
- Partly supportedcites [78] 78. OpenAI. 2021. Pricing. https://openai.com/api/pricing/ sourceExperimental — accuracy not yet measured
Section C, “Additional model details” (page not specified), says the paper’s own language models and RL policies have a context length of 2k tokens. It does not establish that all state-of-the-art LLMs are limited to at most 2048 tokens, or equate that length with about 1500 words. (LLM, conf=0.88)
- Partly supportedcites [86] 86. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoff- mann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Su- sannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saf- fron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Lau- rent sourceExperimental — accuracy not yet measured
The Training section (p. 6) says the authors trained their models with a 2048-token context window. That does not establish a 2048-token maximum for all state-of-the-art LLMs; D.6 (p. 62) also reports extending Gopher’s context length at evaluation time beyond its training boundary. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We conducted our study using Chinchilla 70B because it outperformed other LLMs on a large collection of language tasks [50].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3 METHODS: CREATIVE TEXT GENERATION USING LARGE LANGUAGE MODELS AND DRAMATRON 3.1 Language Models
What the sources say
- Supportedcites [50] 50. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556 (2022). source
The paper’s opening summary (p. 1) identifies Chinchilla as a 70B-parameter model and states that it outperforms Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG across a large range of downstream tasks. Table 5 (p. 10) shows the breadth of the language-task evaluation. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“many participants noticed gender biases and ageism in the system outputs.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.3.2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants found Dramatron useful for getting inspiration”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.1 Inspiration for the Writer (storytelling).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants noticed how the system could enter generation loops”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.4.2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Hierarchical generation of stories can produce an entire script-sometimes tens of thousands of words-from a single user-provided summary of the central dramatic conflict, called the log line [107].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [107] 107. Josef Steiff. 2005. The complete idiot's guide to independent filmmaking. Penguin.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By prompting the system, writers could indirectly search the language model for literary styles and elements”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.3 Using the System for Learning and Analysis (storytelling)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.3 Inspiration from Narrative Elements and Story Arcs to Structure Script Generation To circumvent the limitations of LLMs and to enable the generation of theatre and film scripts, we take a divide-and-conquer approach: we structure script generation using interdependent narrative elements.We base our approach on Aristotle's Poetics [5], which identified 1) plot (mythos) as the most important part of a tragic story-the other elements being the 2) theme of the story (dianoia), 3) its characters (ethos), 4) dialogues (lexis), as well as melody and spectacle.Thus, we start from a user-defined theme and use Dramatron to help write characters, plot, and dialogue.Aristotle introduced the well-known plot structure (beginning, middle, and end).Many other dramatic structures followed [84].For example, variations on the Freytag's pyramid [41], popular in Western storytelling 5 (Exposition, Inciting Incident, a Rising Action composed of a series of Conflicts, Complications, and Dilemmas, Climax, Falling Action, Resolution, and Dénouement) or the Hero's Journey or Monomyth [15,113].More generally, the plot is the coherent sequence of consecutive actions that shape the story.In order to achieve narrative coherence despite the limited context 3 Anecdotally, and to verify how large the LLM needs to be to work with Dramatron, we evaluated models of different sizes by prompting them with the same log line ("A morality tale where a poor old fisher living on the cold Atlantic coast catches a magical marlin fish, who promises to grant all her wishes.Prompted by her greedy husband, the fisher asks for all fish to be turned to gold").We generated a list of characters for 50 different random seeds.Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.4 For example: https://theguardian.com/commentisfree/2020/sep/08/robot-wrote-this- article-gpt-3 5 Seminal work in narrative analysis [60,85,96] suggests general but mutable structures present in storytelling processes across many linguistic and cultural traditions (e.g.abstract, orientation, complicating action, and coda narrative clauses).The finding that narratives in many societies and in various languages make use of large, modular elements has inspired the use of models of narrative in automated storytelling, as can be seen in fields such as computational narratology [55,65].The general structures found in the Structuralist narratology of Propp [85] and the Personal Experience Narratives of Labov and Waletzky [60] are aligned with Freytag's pyramid, which we choose because it is more in line with the specific discourse genre of dramatic scripts, and arguably more familiar to the playwrights we engaged with over the course of our study.However, we note that our choice of narrative structure is not universal in scope, and is, in fact, "characteristically Western" [26,55].Alternative story shapes, or "story grammars" [97] are possible [6,17]. window of LLMs, we generate a plot synopsis in one go, as a sequence of beats.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants but p4 and p5 (who preferred a nonlinear writing workflow) were enthusiastic about the interactive hierarchical generation.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.1 Praise for the interactive hierarchical generation in Dramatron (storytelling). All participants but p4 and p5 (who preferred a nonlinear writing workflow) were enthusiastic about the interactive hierarchical generation."Once I see this, I know the shape of the series.I know the way that the story unfolds.I can see the narrative more clearly [...] I like this approach of making it a log line and then packing the detail inside it.You are planting a seed of an idea and it is putting meat on the bones" (p13)."All of it is quite consistent, symbolically consistent and coherent and relates to the state of affairs of the state of the play [...]There is lots of emotion and content about relationships in some of the generations" (p8)."In terms of the interactive co-authorship process, I think it is great [...] " (p9)."What I like about the hierarchy is that you can do as much humaning as you want at any level" (p2)."In working with the machine I can see the content a little more clearly.As there is specificity, character arcs, then I can see how the story comes together [...] This [hierarchical generation] really felt so much cleaner than the process [GPT-2 or GPT-3 with flat prompting] I was using" (p15)."Let's try more!God, you could just waste your time doing this" (p3).Participants p1, p6 and p3 further noted how such hierarchical generation helped with dialogue: "there is good content from any generation" (p1) and (referring to one of the generations) "You got some big profound discussions in it.I am impressed with that one" (p3).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 85
- It may state two separate claims as one. 34
- The quote only partly states this claim. 2
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“We do not evaluate Dramatron outputs for grammatical correctness as [80], as the few errors made by the LLM can be fixed by the human co-writer.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
What the sources say
- Disputedcites [80] 80. Vishakh Padmakumar and He He. 2022. Machine-in-the-Loop Rewriting for Creative Image Captioning. In Proceedings of the 20th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). sourceExperimental — accuracy not yet measured
Section 5.1, “User Evaluation of Suggestion Quality,” explicitly asks users how grammatically correct the model suggestions were; the Results report grammaticality ratings. The source therefore does evaluate grammatical correctness, contrary to the practice the citing sentence attributes to it. Its study concerns creative image captions, not Dramatron, and the cited passages do not give human correction of a few errors as a reason to omit evaluation. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The few grammatical errors made by the LLM can be fixed by the human co-writer.
- Disputed
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Hierarchical generation of a theatre play was first mentioned and used in [93,95] for the production of AI: Can a Robot Write a Play? by company THEaiTRE in 2021 in Prague 9 .”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.2 Symbolic and Hierarchical Story Generation
What the sources say
- Disputedcites [93] 93. Rudolf Rosa, Ondřej Dušek, Tom Kocmi, David Mareček, Tomáš Musil, Patrícia Schmidtová, Dominik Jurko, Ondřej Bojar, Daniel Hrbek, David Košt'ák, et al. 2020. THEaiTRE: Artificial intelligence to write a theatre play. arXiv preprint arXiv:2006.14668 (2020). sourceExperimental — accuracy not yet measured
The paper mentions hierarchical generation in “Unresolved issues and future plans” (p. 3), but says the authors intended to use it in a future version—not that they used it for the 2021 play. The Introduction (p. 1) instead says they generated individual scenes and a dramaturge joined them into a full play. The copy does not establish that this was the first mention of hierarchical theatre-play generation. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Not checked — no full text of the sourcecites [95] 95. Rudolf Rosa, Patrícia Schmidtová, Ondřej Dušek, Tomáš Musil, David Mareček, Saad Obaid, Marie Nováková, Klára Vosecká, and Josef Doležal. 2022. GPT-2- based Human-in-the-loop Theatre Play Script Generation. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022). 29-37. source
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Characters, plot, and location descriptions all meet the specification in the log line, in addition to causal dependencies, enabled by prompt chaining [121] and explained on the diagram of Figure 1.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.4 Hierarchical Language Generation
What the sources say
- Partly supportedcites [121] 121. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. arXiv:2109.10862 [cs.CL] sourceExperimental — accuracy not yet measured
The Framework (p. 2) describes a sequence of connected steps that generates each story sentence and then repeats using the expanded context. It does not describe this as prompt chaining or show that characters, plot, and location descriptions meet a log-line specification or preserve causal dependencies. Future Work (p. 8) also treats structure-level control as a potential direction rather than an established result. (LLM, conf=0.82)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Several participants requested conversational agents to help them rewrite text-which was prescient of recently published language agents trained from human feedback [44,79].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
What the sources say
- Supportedcites [44] 44. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judge- ments. arXiv preprint arXiv:2209.14375 (2022). source
The citation supports the sentence’s reference to a recently published language agent trained from human feedback, not the citing authors’ observation about participants requesting rewriting help. The source paper’s opening page describes Sparrow as a dialogue agent trained using reinforcement learning from human feedback (p. 1). (LLM, conf=0.93)
- Supportedcites [79] 79. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022). source
The citation supports the sentence’s reference to language agents trained with human feedback, not its report about the citing study’s participants. The Introduction describes InstructGPT as a language model fine-tuned with human feedback, and Appendix A.2.1 includes an illustrative prompt asking it to rewrite text. No page numbers are provided in our copy. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 2 RELATED WORK 2.1 Automated and Crowdsourced Evaluation of the Coherence of Generated Text Echoing Celikyilmaz et al. [16], one can split evaluation methods into automated or machine-learned metrics, and humancentric evaluation.As detailed in the Appendix A.4, automated and machine-learned metrics typically calculate 1) the similarity between generated and "ground truth" stories [38,39,88,105], the 2) consistency between generated stories and their writing prompts [92,102], or 3) the diversity of language in the generated output [43,45,46,88,127].These metrics were not designed for generated text of the length of a screenplay or theatre script.This motivates a focus on human-centric evaluation, which can be conducted with non-expert crowdworkers or with experts.We therefore review the limitations of crowdsourced evaluation of the coherence of generated text, and explain why non-expert, crowdsourced evaluation faces crucial quality and bias issues.Inheriting from research standards for large-scale natural language processing tasks, the majority of studies assessing the quality of generations from LLMs evaluate model performance by collecting data from crowdworkers [58,77,80,88,91,124].For instance Yao et al. [124] recruit crowdworkers to evaluate fidelity, coherence, interestingness, and popularity, and Rashkin et al. [88] to evaluate narrative flow and ordering of events.That said, it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].These issues have led some researchers to try evaluating their models with experts.Karpinska et al. [56] highlight the perils of using crowdsourced workers for evaluating open-ended generated text, because crowdworkers do not read text as carefully as expert teachers.Some studies consulted with expert linguists [31], university students in the sciences [42] or humanities [123], and amateur writers [2,21,62,100,125].In one recent study, Calderwood et al. [13] interviewed 4 novelists about their usage of GPT-2 via Talk To Transformer and Write With Transformer tools (see https://app.inferkit.com/demo),uncovering usages of the "model as antagonist" (i.e., random), for "description creation", as "constraint", or for "unexpected" ideas-some of the themes highlighted by our own study participants.Similarly, a recent user study explored evaluation of writing short stories with a collaborative text editing tool called Wordcraft [125].
What the sources say
- Not checked — no full text of the sourcecites [36] 36. Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. In Proceedings of the eleventh ACM international conference on web search and data mining. 162-170. source
- Supportedcites [30] 30. Tim Draws, David La Barbera, Michael Soprano, Kevin Roitero, Davide Ceolin, Alessandro Checco, and Stefano Mizzaro. 2022. The Effects of Crowd Worker Bi- ases in Fact-Checking Tasks. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2114-2124. source
The citation to this paper is attached to the fact-checking portion of the sentence. The paper’s Practical Implications (§6.2) state that workers’ individual characteristics and cognitive biases can negatively affect the accuracy of crowdsourced truthfulness judgments. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [53] 53. Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-12. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6 PARTICIPANT SURVEYS 6.1 Qualitative and Quantitative Results on Participant Surveys Of the total 15 study participants, 13 provided responses on our postsession feedback form.The form gave participants the following instruction: "When answering these questions, please reflect on the interactive co-authorship session as well as considering the use of an interactive AI system like Dramatron in the future", and asked nine questions.Each question could be answered using a Likert-type scale ranging from Strongly Disagree 1 to 5 Strongly Agree.These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]: (1) I found the AI system helpful, (2) I felt like I was collaborating with the AI system, (3) I found it easy to write with the AI system, (4) I enjoyed writing with the AI system, (5) I was able to express my creative goals while writing with the AI system, (6) The script(s) written with the AI system feel unique, (7) I feel I have ownership over the created script(s), (8) I was surprised by the responses from the AI system, and (9) I'm proud of the final outputs.We also asked five free-form questions.Two questions aimed at assessing these questions to manually define a binary indicator variable (Has experience of AI writing tools) and a three-class category for their primary domain of expertise (Improvisation, Scripted Theatre and Film/TV ).Three more questions gave participants an opportunity to provide developmental feedback about the system: What is one thing that the AI system did well?, What is one improvement for the AI system? and Please provide any comments, reflections, or open questions that came up for you during the co-authorship session or when answering this survey.Aggregated responses are plotted on Figure 5 (Left), and (Right) factors responses to the question on collaboration by industry background and experience (see Figure 7 for more).The results are summarized below.
What the sources say
- Not matchedcites Yuan et al. [125]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Stevenson et al. [108] 108. Claire Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. 2022. Putting GPT-3's Creativity to the (Alternative Uses) Test. (2022). sourceExperimental — accuracy not yet measured
The Alternative Uses Task sections (p. 2) provide a question posed to GPT-3 and instructions given to human participants, so the paper contains wording that could be adapted into questions. Our copy does not show the citing document’s survey questions, however, and cannot establish that they were only slightly adapted from this wording. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar LLM risks have been previously mentioned in [119].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.3 Undermining creative economies.
What the sources say
- Supportedcites [119] 119. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022). source
The Broader impacts section (p. 14) explicitly discusses risks from the models and techniques studied, including enabling malicious actors to train harmful models and increasing job loss through automation. It says the paper's approach could amplify those risks, supporting the general claim that such risks had been mentioned before. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Currently, the rules are unclear for machine learning-based systems trained on copyright-protected material [9].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.2 Plagiarism.
What the sources say
- Could not verifycites [9] 9. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). https://arXiv.org/abs/2108.07258 sourceExperimental — accuracy not yet measured
The Data overview (p. 18) mentions legal regulation of training data, but does not say that copyright rules for training machine-learning systems are unclear. Searches for copyright, intellectual property, fair use, and copyrighted training data found no passage addressing that point in our copy; the copy is also marked as truncated. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous work has argued for engagement with subject matter experts, literary scholars, and industry professionals in the development of machine learning models [112].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.4
What the sources say
- Partly supportedcites [112] 112. Imke Van Heerden and Anil Bas. 2021. AI as Author-Bridging the Gap Between Machine Learning and Literary Theory. Journal of Artificial Intelligence Research 71 (2021), 175-189. sourceExperimental — accuracy not yet measured
The Introduction explicitly argues for involving literary scholars in developing machine-learning models, supporting engagement with those subject-matter experts. Section 5 discusses how scholars and professionals might respond to changes in the literary industry, but does not argue that industry professionals should participate in model development. Page numbers are unavailable in our copy. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Our work of decomposing a log line into a synopsis can be seen as a narrative equivalent of Chain of Thought prompting for reasoning tasks [118], and uses the idea of using LLM output as prompt for the next stage of the generation-also called prompt chaining [121] or language engineering [24].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.3 Controllable Story Generation
What the sources say
- Could not verifycites [118] 118. Stephen G Ware and Robert Michael Young. 2011. CPOCL: A Narrative Planner Supporting Conflict.. In AIIDE. sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
- Partly supportedcites [121] 121. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. arXiv:2109.10862 [cs.CL] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [24] 24. Antonia Creswell and Murray Shanahan. 2022. Faithful Reasoning Using Large Language Models. arXiv preprint arXiv:2208.14271 (2022). source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Prompt engineering has been used to write a short plot from two characters, a genre and a theme in [54].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.3 Controllable Story Generation
What the sources say
- Supportedcites [54] 54. Yiping Jin, Vishakha Kadam, and Dittaya Wanvarie. 2022. Plot Writing From Pre-Trained Language Models. arXiv preprint arXiv:2206.03021 (2022). https: //arXiv.org/abs/2206.03021 source
The Factorizing Elements of a Story section describes prompting a language model to generate plot elements, including characters, genre, and theme. The Conflict section specifies prompts for a male and a female character, and the examples show plots generated from a plan containing those elements. The paper also includes location in the plan, but the claim does not say its list is exhaustive. (LLM, conf=0.88)
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Lee et al. [61] distinguish between verbatim, paraphrase, and idea plagiarism and suggest to automatically evaluate the first one using BM25-based retrieval of queries in large set of reference documents, along with a text alignment tool.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.2 Plagiarism.
What the sources say
- Supportedcites [61] 61. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2022. Do Language Models Plagiarize? arXiv e-prints (2022), arXiv-2203. source
Section 3.1 distinguishes verbatim, paraphrase, and idea plagiarism. Section 3.2 describes an automated detection approach that retrieves candidate documents using Elasticsearch’s default Okapi-BM25 ranking, then uses text alignment to find plagiarized passages, including verbatim plagiarism. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most were able to co-write a full script and the open discussion interview in the allotted 2-hours.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The show's run was titled Plays By Bots and ran 7 performances over two weeks”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants had prior exposure to AI writing tools.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5 PARTICIPANT INTERVIEWS
plainn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More than merely providing a creative spark for the main story, the model can be employed to populate the universe of the story”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.2 Generation of Alternative Choices and World Building (script writing).
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As p10 reiterates, iterative rewriting was a desired workflow”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For participant 12, it was difficult to come up with a log line, and the process seemed precious.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.6 Structural Problems of Dramatron (script writing)
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several participants wanted to be able to query and dialogue with the writing model”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some participants suggested this tool could be employed in a TV writers' room, to help with writing formulaic scripts.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.5 Potential of AI as Tool for TV Screenwriting (script writing).
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This problem raised the issue of coping strategies or cultural appropriation”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.3.2
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Widely relevant feedback from the interviews included: participation of experts is critical in tool development, different use-cases demand iterative improvement and development, and different LLMs satisfy different needs from participants.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant p1 tried and discussed three different modes of script generation: 1) interactive co-authorship, 2) modifying the output from one fully automated generation, and 3) curating and modifying outputs from 3-4 generations.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.2 Ease of use of Dramatron's UI and seed-based generation.
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learning could also happen by analysing how to improve Dramatron's outputs”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.3 Using the System for Learning and Analysis (storytelling)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our method makes it possible for elements in the final scene to provide dramatic closure on elements introduced in the opening scene 6 , and for generated stories to follow narrative arcs (see Section 3.3).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.4 Hierarchical Language Generation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Widely relevant feedback from the interviews included: participation of experts is critical in tool development, different use-cases demand iterative improvement and development, and different LLMs satisfy different needs from participants.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“three participants (p8, p13, p15) suggested fitting a narrative arc within each scene.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The scripts were brought to life by a cast of 4-6 experienced improvisers and actors.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“There were two reviews written about the production of Plays By Bots at the festival.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Beyond inspiration, several participants were interested by the co-writing potential of Dramatron, and thought it could provide them with material.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.4 Content Generation (script writing).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Glitches and language model limitations can be subverted for inspiration, in particular when the script is staged”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.2.1 Inspiration for the Writer (storytelling).
plainconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This co-writing approach allows the human and Dramatron to both contribute to the authorship of a script.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.6 Interactive Writing with Dramatron
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As detailed in Section 5.7, the participants were engaged and provided constructive feedback about Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.8 Incremental Tool Improvement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this way, the system positively benefited and evolved over the course of the participant study.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.8 Incremental Tool Improvement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant p13 liked the user experience of interactive, step-by-step generation of title, characters and plot”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.2 Ease of use of Dramatron's UI and seed-based generation.
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several participants (p6, p9, p11, p13, p15) highlighted the potential for the script to be staged after editing”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.3
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“That said, we designed Dramatron for interactive co-writing, as an augmentative tool for human writers.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.6 Interactive Writing with Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant p1 staged several scripts generated with Dramatron (see Section 5.9).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.1.3
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modeling characters and their relationships was a recurrent theme”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.7 Suggested Improvements to Dramatron (script writing)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another participant had 22.5 hours of interaction in total, split over 5 sessions and spread over two months, where they co-wrote 5 scripts that were later staged.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These 4 were produced and and presented at a large theatre festival, as described in Section 5.9.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reviewer also expressed surprise at the ability of the system to create a play that hangs together and creates a world.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, they argued against Dramatron's constrained top-down hierarchy:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Four key themes emerged through these discussions which echo the themes presented earlier in Section 5.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“During co-authorship we observed that participants tended to delete longer location descriptions and parts of dialogue (especially loops), and rewrite the character descriptions.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the system has a distinct glitch style, generated text can be repetitive and fun to work with.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One participant commented that the generated script was only a starting point provided by the AI and that the writer still needed to create their own script.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.5
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With a hint of incredulity, the reviewer complimented the abilities of Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that participants would skim the text and remark on specific inspiring lines, but were less focused on the overall coherence of the discourse.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that these two answers were highly correlated (𝑟 = 0.9)”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.4
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, the prevailing feedback from the majority of performers was that participating in the production was fun.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants 4 and 5, who self-identified as playwrights for the stage,”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They further noted that some lines from Dramatron are so funny they were reprised later in the show once the human actors were improvising.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the total 15 study participants, 13 provided responses on our postsession feedback form.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6 PARTICIPANT SURVEYS 6.1 Qualitative and Quantitative Results on Participant Surveys Of the total 15 study participants, 13 provided responses on our postsession feedback form.The form gave participants the following instruction: "When answering these questions, please reflect on the interactive co-authorship session as well as considering the use of an interactive AI system like Dramatron in the future", and asked nine questions.Each question could be answered using a Likert-type scale ranging from Strongly Disagree 1 to 5 Strongly Agree.These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]: (1) I found the AI system helpful, (2) I felt like I was collaborating with the AI system, (3) I found it easy to write with the AI system, (4) I enjoyed writing with the AI system, (5) I was able to express my creative goals while writing with the AI system, (6) The script(s) written with the AI system feel unique, (7) I feel I have ownership over the created script(s), (8) I was surprised by the responses from the AI system, and (9) I'm proud of the final outputs.We also asked five free-form questions.Two questions aimed at assessing these questions to manually define a binary indicator variable (Has experience of AI writing tools) and a three-class category for their primary domain of expertise (Improvisation, Scripted Theatre and Film/TV ).Three more questions gave participants an opportunity to provide developmental feedback about the system: What is one thing that the AI system did well?, What is one improvement for the AI system? and Please provide any comments, reflections, or open questions that came up for you during the co-authorship session or when answering this survey.Aggregated responses are plotted on Figure 5 (Left), and (Right) factors responses to the question on collaboration by industry background and experience (see Figure 7 for more).The results are summarized below.
plainn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We notice that four participants tended to add to generated outputs (three of them, p6, p13, and p15 expressed interest in staging the resulting scripts) while eight others tended to remove from and curate the outputs (including p1 who staged productions).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
plainn = 8- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants were more split about the ease of use of the tool (61% agreed) and whether they felt proud about the output of the system (46% agreed), with one participant strongly disagreeing.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.4
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Secondly, the generated text can sometimes be repetitive.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and that several questions had a relatively strong correlation (𝑟 ≥ 0.66): helpful, collaboration, ease, enjoyment, expression and ease.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.4
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, the system has a distinct glitch style that can sometimes be nonsensical, vague, or passive.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“some biases-in particular around gender-were noticed by the participants in the LLM outputs.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.1 Biases and stereotypes.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The third concern raised by some of the participants was about the potential impact of generative tools on creative economies:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.3 Undermining creative economies.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The statements that received the strongest positive response were I was surprised by the responses from the AI system (92% participants agreed, 46% strongly agreed) followed by I enjoyed writing with the AI system (77% agreed, 69% strongly agreed) and The script(s) written with the AI system feel unique (69% agreed, 46% strongly agreed)”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.1 AI-generated scripts can be surprising, enjoyable, and unique.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This expert also expressed enthusiasm to write alongside Dramatron again for future productions.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.6 Novelty Effects.
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“That said, participants did notice if the dialogue of a scene was unrelated to that scene's beat or log line, and most noticed when the generation had excessive repetition.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study participants raised concerns the source of the dataset:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.2 Plagiarism.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Positive responses to statements I felt like I was collaborating with the AI system, (77% agreed, 54% strongly agreed), I found the AI system helpful (84% agreed, 38% strongly agreed) and I was able to express my creative goals with the AI system (77% agreed, 23% strongly agreed)”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We present Dramatron: an interactive co-writing tool which allows writers to generate scripts from a provided log line.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reviewer also noted that the success of the performance was due to both the Dramatron system and the human actors”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The log line itself could be seen as a post-hoc summary, as”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As well, the team attributed agency to the system, and had expectations of the systems capabilities.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The second review was also positive.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.9 Staging and Evaluating Productions of Scripts Co-written by Dramatron
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Log lines typically contain the setting, protagonist, antagonist, a conflict or goal, and sometimes the inciting incident [8]”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.3 Inspiration from Narrative Elements and Story Arcs to Structure Script Generation To circumvent the limitations of LLMs and to enable the generation of theatre and film scripts, we take a divide-and-conquer approach: we structure script generation using interdependent narrative elements.We base our approach on Aristotle's Poetics [5], which identified 1) plot (mythos) as the most important part of a tragic story-the other elements being the 2) theme of the story (dianoia), 3) its characters (ethos), 4) dialogues (lexis), as well as melody and spectacle.Thus, we start from a user-defined theme and use Dramatron to help write characters, plot, and dialogue.Aristotle introduced the well-known plot structure (beginning, middle, and end).Many other dramatic structures followed [84].For example, variations on the Freytag's pyramid [41], popular in Western storytelling 5 (Exposition, Inciting Incident, a Rising Action composed of a series of Conflicts, Complications, and Dilemmas, Climax, Falling Action, Resolution, and Dénouement) or the Hero's Journey or Monomyth [15,113].More generally, the plot is the coherent sequence of consecutive actions that shape the story.In order to achieve narrative coherence despite the limited context 3 Anecdotally, and to verify how large the LLM needs to be to work with Dramatron, we evaluated models of different sizes by prompting them with the same log line ("A morality tale where a poor old fisher living on the cold Atlantic coast catches a magical marlin fish, who promises to grant all her wishes.Prompted by her greedy husband, the fisher asks for all fish to be turned to gold").We generated a list of characters for 50 different random seeds.Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.4 For example: https://theguardian.com/commentisfree/2020/sep/08/robot-wrote-this- article-gpt-3 5 Seminal work in narrative analysis [60,85,96] suggests general but mutable structures present in storytelling processes across many linguistic and cultural traditions (e.g.abstract, orientation, complicating action, and coda narrative clauses).The finding that narratives in many societies and in various languages make use of large, modular elements has inspired the use of models of narrative in automated storytelling, as can be seen in fields such as computational narratology [55,65].The general structures found in the Structuralist narratology of Propp [85] and the Personal Experience Narratives of Labov and Waletzky [60] are aligned with Freytag's pyramid, which we choose because it is more in line with the specific discourse genre of dramatic scripts, and arguably more familiar to the playwrights we engaged with over the course of our study.However, we note that our choice of narrative structure is not universal in scope, and is, in fact, "characteristically Western" [26,55].Alternative story shapes, or "story grammars" [97] are possible [6,17]. window of LLMs, we generate a plot synopsis in one go, as a sequence of beats.
What the sources say
- Not checked yetcites [8] 8. Lane Shefter Bishop. 2016. Sell Your Story in A Single Sentence: Advice from the Front Lines of Hollywood. The Countryman Press.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, such models lack long-range semantic coherence, limiting their usefulness for longform creative writing.”
- Also stated in: § 1 INTRODUCTION · plain
“This is a difficult task for LLMs because the narrative of a script or screenplay must exhibit long-term coherence and reincorporation, and LLMs are limited in their ability to model long-range dependencies (e.g., to reincorporate information from many pages ago).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we engaged 15-experts in two-hour long user study sessions to co-write a script alongside Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We engineered the hierarchical prompt structure and the few-shot examples through iterative prompting and testing.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.5 Generation of Narrative Elements using Prompt Engineering
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study design and data collection process was validated by an ethical review board external to DeepMind.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Oftentimes the seed of narrative inspiration for a screenplay or theatre script is a log line [101].”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.3 Inspiration from Narrative Elements and Story Arcs to Structure Script Generation To circumvent the limitations of LLMs and to enable the generation of theatre and film scripts, we take a divide-and-conquer approach: we structure script generation using interdependent narrative elements.We base our approach on Aristotle's Poetics [5], which identified 1) plot (mythos) as the most important part of a tragic story-the other elements being the 2) theme of the story (dianoia), 3) its characters (ethos), 4) dialogues (lexis), as well as melody and spectacle.Thus, we start from a user-defined theme and use Dramatron to help write characters, plot, and dialogue.Aristotle introduced the well-known plot structure (beginning, middle, and end).Many other dramatic structures followed [84].For example, variations on the Freytag's pyramid [41], popular in Western storytelling 5 (Exposition, Inciting Incident, a Rising Action composed of a series of Conflicts, Complications, and Dilemmas, Climax, Falling Action, Resolution, and Dénouement) or the Hero's Journey or Monomyth [15,113].More generally, the plot is the coherent sequence of consecutive actions that shape the story.In order to achieve narrative coherence despite the limited context 3 Anecdotally, and to verify how large the LLM needs to be to work with Dramatron, we evaluated models of different sizes by prompting them with the same log line ("A morality tale where a poor old fisher living on the cold Atlantic coast catches a magical marlin fish, who promises to grant all her wishes.Prompted by her greedy husband, the fisher asks for all fish to be turned to gold").We generated a list of characters for 50 different random seeds.Models with 1B parameters or more (e.g., Chinchilla 1B and GPT-3 babbage) succeeded in generating a valid list of characters at least 80% of times.4 For example: https://theguardian.com/commentisfree/2020/sep/08/robot-wrote-this- article-gpt-3 5 Seminal work in narrative analysis [60,85,96] suggests general but mutable structures present in storytelling processes across many linguistic and cultural traditions (e.g.abstract, orientation, complicating action, and coda narrative clauses).The finding that narratives in many societies and in various languages make use of large, modular elements has inspired the use of models of narrative in automated storytelling, as can be seen in fields such as computational narratology [55,65].The general structures found in the Structuralist narratology of Propp [85] and the Personal Experience Narratives of Labov and Waletzky [60] are aligned with Freytag's pyramid, which we choose because it is more in line with the specific discourse genre of dramatic scripts, and arguably more familiar to the playwrights we engaged with over the course of our study.However, we note that our choice of narrative structure is not universal in scope, and is, in fact, "characteristically Western" [26,55].Alternative story shapes, or "story grammars" [97] are possible [6,17]. window of LLMs, we generate a plot synopsis in one go, as a sequence of beats.
What the sources say
- Not checked yetcites [101] 101. Industrial Scripts. 2019. How to write outstanding TV & movie loglines: The ultimate guide. https://industrialscripts.com/loglines-guide/
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The incorporation of the middle layer, where the plot is summarised as a sequence of abstract scene descriptions, allows the entire plot to fit within the language models' context window.This overcomes prior limitations on long-term semantic coherence.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.4 Hierarchical Language Generation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To the best of our knowledge, this work represents the largest expert user study conducted on co-creative authorship to date.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“LLMs can give the impression of coherence within and between paragraphs [7], but have difficulty with long-term semantic coherence due to the restricted size of their context windows.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.2 Limited Contexts of LLMs and Need for Human-in-the-Loop Editing
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021), 610-623. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study and data collection process was validated and approved by HuBREC (Human Behavioral Research Ethics Committee), which is a research ethics committee run within Deepmind which includes and is chaired by academics from outside the company.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Feedback was analysed first separately by two investigators to extract different themes, then their respective analyses were merged and jointly summarised into seven major themes.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5 PARTICIPANT INTERVIEWS
plainn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Over the course of the interviews, we incorporate the feedback we could by making small, incremental changes to the prompt prefix sets of Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 5.8 Incremental Tool Improvement
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We chose and reworded the prompts so that the outputs of the LLM could be parsed into various components (title, character description, etc.) with high probability of success.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 3.5 Generation of Narrative Elements using Prompt Engineering
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Along similar lines, Chung et al. [20] discuss how AI-based creativity support tools augment and automate portions of an artist's support network.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.4
What the sources say
- Not checked — no full text of the sourcecites [20] 20. John Joon Young Chung, Shiqing He, and Eytan Adar. 2022. Artist Support Networks: Implications for Future Creativity Support Tools. Proceedings of Designing Interactive Systems: DIS'22 (2022). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This reinforces our observations during interviews (see the criticism of log line-based conditioning for generating theatre scripts in Section 5.6).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.2
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“their utility as writing tools increases as they generate more coherent outputs.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7 DISCUSSION AND FUTURE WORK 7.1 Three Avenues of Future Work towards
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This finding aligns with our observations in the interactive sessions where participants tended to delete degenerate repetitions”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We interpret our distance measures as indicators of engagement with Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Story arc generation was introduced in the Tale Brush graphical tool [21], using Kurt Vonnegut's theory about the fortune of the protagonist as the story progresses.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.3 Controllable Story Generation
What the sources say
- Not checked — no full text of the sourcecites [21] 21. John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. 1-19. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While this sentiment partially modulates how generalizable the findings are over longer term engagements”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.6 Novelty Effects.
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: A single engagement of co-writing alongside Dramatron might exhibit novelty effects.
In the paper
“While common sense and logical consistency is an elusive goal for LLMs [7]”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7 DISCUSSION AND FUTURE WORK 7.1 Three Avenues of Future Work towards
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021), 610-623. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“In contrast to our approach, they did not generate characters alongside synopsis and would start the”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.2 Symbolic and Hierarchical Story Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron (choosing among generated text seeds is another aspect of interaction not accounted for with this metric).”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > C DETAILS OF QUANTITATIVE OBSERVATIONS C.1 Levenshtein Distance Levenshtein distance was calculated at the character level for edited versus non-edited texts using the default edit distance function from the Natural Language Tool Kit 11 package's distance module (no transposition operations and substitution cost equal to 1).As an absolute measure, the distance metric indicates how many operations (insertion, deletion, substitution) are required to convert one string into another and is typically calculated with a dynamic programming approach to minimizing the cost of character-level edit operations for the conversion.However, as a measure dependent on the lengths of both input strings, comparing across edit distances becomes rather uninterpretable when the string sets differ in length, for example across sets of edited and unedited texts for characters, locations, plots, and dialogues.As we are interested in understanding the extent to which participants edited the output of Dramatron as a cohort, it is reasonable to normalise the distances with respect to their length, which we report in 6 (Right).To do this, we calculate a relative Levenshtein distance as the ratio of the raw Levenshtein distance between edited and non-edited (Dramatron) output text to the length of original Dramatron output.Conceptually, we can view the resulting measure as a proportional measure of interaction with our tool.Given that Levenshtein distance operations are character-level, and length is measured in characters, the proportion represents the a weighting of active versus passive interaction with Dramatron for the different levels of structure in the hierarchical story generation process.Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron (choosing among generated text seeds is another aspect of interaction not accounted for with this metric).12
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one. The quote only partly states this claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These reflections discussed by the creative team reflect the usefulness of co-written scripts.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Their work also only served the production of a specific theatrical play rather than being evaluated within a diverse community of writers.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > A.2 Symbolic and Hierarchical Story Generation
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We interpret our distance measures as indicators of engagement with Dramatron.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.2 Quantitative Observations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The co-creativity of a system such as Dramatron extends beyond the playwright and Dramatron, and to the performer on the stage working with the generated and edited text.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > B ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One interpretation is that Dramatron is seen by the industry experts not as much as a full script writing tool but rather as a learning tool to practice writing and a source of inspiration”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 6.1.5
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hierarchical generation can be leveraged in various ways to improve the long-range coherence of long texts.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7 DISCUSSION AND FUTURE WORK 7.1 Three Avenues of Future Work towards
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tools may need to conform to the artists' expectations of collaboration, similarly to the types of interactions they have with human collaborators: for example, sub-contracting, co-creation, or inspiration.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.3.4
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts:”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 7.2 Script and Screenplay Style Differences and a Critique of Dramatron's Hierarchy Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study.Dramatron was originally designed for a single style: topdown hierarchical generation where each subsequent step depends on those that came prior.But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting.This is summarized well in one participant's remarks: "I think the model where writers begin from inputting character and logline favours certain kinds of processesis there a version of the system that can originate from a different point?"Another participant echoed these sentiments, saying: "Creating theatre is not always a linear / story driven process -a lot of the time it is about depth (theme) as well as forward motion (story) -characters are not always the start of the story, but a means to deliver the deeper meaning of story"."Playwrights do not use the log line in the same way as in screen writing" (p9), as one participant succinctly stated.Participants 4 and 5, who self-identified as playwrights for the stage, "would never approach a piece of work with a story in [their] head.[They] might come with a more investigative approach with a theme".In fact, they argued against Dramatron's constrained top-down hierarchy: "Why generate characters first?At earlier parts of the creation process you might not know the characters.In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last".The log line itself could be seen as a post-hoc summary, as "often times playwrights find out what the play is about after [they] finish it.I will know the play once it is done" (p9).This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order. Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: "The difference between theatre and screen is that nobody's making theatre on demand, no outside pressure.Or at least not in the same way that there is for television.Theatre feels less like generating content" (p4, p5) whereas "film scripts, in the industry, want the traditional fourth wall" (p9).Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing?As one respondent noted, "[Dramatron] will be very useful for an average Hollywood movie and for television.It does not need to have a deep understanding of the human soul, unlike Shakespeare.[...] The thing with action movies is that is that actors are not expected to connect with the writer.A screenwriter on a TV set is just like [Dramatron] [...].It is a sublime skill to be a Hollywood writer because your creative input is small." (p9).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The few grammatical errors made by the LLM can be fixed by the human co-writer.
“We do not evaluate Dramatron outputs for grammatical correctness as [80], as the few errors made by the LLM can be fixed by the human co-writer.”
The authors state this as an assumption.
A single engagement of co-writing alongside Dramatron might exhibit novelty effects.
“One potentially confounding factor is that a single engagement of co-writing alongside Dramatron might exhibit some sort of novelty effects.”
The paper relies on this without support.
Creative writing for theatre is fundamentally interactive between storytellers and the audience.
“Creative writing for theatre is fundamentally interactive: not just between collaborating storytellers, but between storytellers and the audience.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, pride of outcome was lower for wr-AI-ter (46%) compared to Dramatron.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The perception of uniqueness presents a signifcant divergence, with Dramatron's outputs being considered unique by 69% of participants, in contrast to wr-AI-ter's 14%.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Both tools were seen as helpful, with Dramatron receiving an 84% approval rating in this regard, compared to wr-AI-ter's substantial 71%.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our results showed a higher sense of ownership (57%) for the results generated by wr-AI-ter”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Ease of use was a split opinion for Dramatron (61% agreement), while wr-AI-ter experienced a lower agreement (48%).”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The diference in the element of surprise between wr-AI-ter and Dramatron (35% vs. 92%)”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Therefore, Q9 is no longer comparable with the answers of the participants in the Dramatron survey, as LLMs were even less well known at that time.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The enjoyment levels (71% for wr-AI-ter) remain high, albeit slightly lower than Dramatron's 77%”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical studies describing how users evaluate LLMs are still a minority and difficult to generalise due to a variation of participants, methods, systems, and disciplinary perspectives [13,27,28,38,47].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The models are often described as biased.They perpetuate stereotypes and jeopardise trust in the generated content [27,38].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A final barrier was identified in the politics embedded in LLMs.”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Not only did the participants report utility for editorial tasks [27,42,47], but they also acknowledge potential for idea generation [13,38,42].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Indeed, past research involving media artists with a long history of experimenting with generative AI highlighted a more positive evaluation and stronger acceptance [38].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Key risks include potential job displacement via automation [191], generating inappropriate or harmful content [199], and disrupting creative economies, which may face user objections [122].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [122]
The cited source says what the claim attributes to it.
In the citing paper
“Dramatron's capacity for expressing creative goals was afrmed by 77% of its users, a contrast to the 21% in our study.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“After realizing that more than half of Dramatron's respondents did not feel ownership of the scripts generated by the AI”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“the sense of collaboration with the AI system was more pronounced in Dramatron's evaluation (77% agreement) compared to our fndings (43% agreement)”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“To ensure optimal comparability with the Dramatron framework, our methodology adapted the same questions highlighted in [25,35], employing 5-point Likert-type scales.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“One interpretation was that Dramatron is not seen as a professional scriptwriting tool, but rather as a learning tool to practice writing or as a resource [25].”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Additionally, our results compare favorably to those from Dramatron but are slightly less positive when compared to Wordcraft.”
From wr-AI-ter: Enhancing Ownership Perception in AI-Driven Script Writing (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“of which just a fraction focused on theatre [36,38,40].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“However, many technical limitations hamper the application of LLMs to creative writing.They include short memory [36,40], lack of semantic coherence [28,40], and common-sense and contextual awareness [36,38].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“Such research has demonstrated that AI has the potential to support a variety of creative writing activities and aspects of the creative writing process.”
From Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems (2026) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“a recent work closely related to ours is the Dramatron system [70], which explores the integration of LLMs into screenwriting. However, it focuses solely on LLMs, neglecting the impact of AI-generated images and videos.”
From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [70]
The cited source says what the claim attributes to it.
In the citing paper
“However, these studies focus solely on LLMs, neglecting advancements in AI-generated images and videos.”
From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [70]
In the citing paper
“Advanced Large Language Models (LLMs) such as ChatGPT [68], GPT-4 [69], PaLM 2 [32], LLaMA 2 [90], and Gemini [73] can enable writers to generate multiple variations of text via natural language prompting [8,15,26,66], potentially reducing the workload of generating text variations.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [66]
In the citing paper
“Advanced Large Language Models (LLMs) such as ChatGPT [68], GPT-4 [69], PaLM 2 [32], LLaMA 2 [90], and Gemini [73] can enable writers to generate multiple variations of text via natural language prompting [8,15,26,66], potentially reducing the workload of generating text variations.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [66]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > C.3 Jaccard Similarity (Relatedness)
“To calculate Jaccard similarity, we first calculate the Jaccard distance, which is calculated by dividing the cardinality of the intersection of two sets and by the cardinality of their union.By subtracting Jaccard distance by 1, we obtain the Jaccard similarity.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > C.3 Jaccard Similarity (Relatedness)
“Jaccard metrics are scaled between 0 and 1, with 0 representing zero set similarity and 1 representing total set similarity.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 1 INTRODUCTION
“Our study design and data collection process was validated by an ethical review board external to DeepMind.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > 4 EVALUATION METHODS FOR SCRIPTS GENERATED BY DRAMATRON
“Our study and data collection process was validated and approved by HuBREC (Human Behavioral Research Ethics Committee), which is a research ethics committee run within Deepmind which includes and is chaired by academics from outside the company.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > C DETAILS OF QUANTITATIVE OBSERVATIONS C.1 Levenshtein Distance
“To do this, we calculate a relative Levenshtein distance as the ratio of the raw Levenshtein distance between edited and non-edited (Dramatron) output text to the length of original Dramatron output.”
§ Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals > C DETAILS OF QUANTITATIVE OBSERVATIONS C.1 Levenshtein Distance
“Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Rumelhart, D. E. (1975) · Representation and Understanding · doi:10.1016/b978-0-12-108550-6.50013-6 · Cited by 1,671
Propp, V. (1968) · University of Texas Press eBooks · doi:10.7560/783911 · Cited by 1,609
Cognitive structures in comprehension and memory of narrative discourse
Thorndyke, P. W. (1977) · Cognitive Psychology · doi:10.1016/0010-0285(77)90005-6 · Cited by 1,441
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Research works printed but not linked: 16
- Lane Shefter Bishop. 2016. Sell Your Story in A Single Sentence: Advice from the Front Lines of Hollywood. The Countryman Press.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. https://arxiv.org/abs/2005.14165 · arXiv
- Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study.. In Proceedings of HAI-GEN+ user2agent@ IUI.
- Joseph Campbell. 1949. The hero with a thousand faces. Princeton, NJ: Princeton University Press.
- William Wallace Cook. 1928. Plotto: The Master Book of All Plots. Ellis, first edition.
- Barbara Hayes-Roth and Robert Van Gent. 1996. Improvisational puppets, actors, and avatars. In Proc Computer Game Developers’ Conf.
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.
- Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning. PMLR, 957–966.
- William Labov and Joshua Waletzky. 1967. Narrative analysis. In Essays on the Verbal and Visual Arts, ed. J. Helm. Seattle: U. of Washington Press, 12–44.
- 10.5555/2891460.2891543 · DOI
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc ACL Wkshp. Vol 8.
- James R Meehan. 1977. TALE-SPIN An Interactive Program that Writes Stories.. In IJCAI Vol. 77. 91–98.
- Pinelopi Papalampidi, Kris Cao, and Tomas Kocisky. 2022. Towards Coherent and Consistent Use of Entities in Narrative Generation. International Conference on Macine Learning(2022).
- 10.5555/1946417.1946422 · DOI
- Melissa Roemmele, Andrew S Gordon, and Reid Swanson. 2017. Evaluating story generation systems using automated linguistic analyses. In SIGKDD 2017 Workshop on Machine Learning for Creativity. 13–17.
- Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC. 2022. AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. In Joint Proceedings of the ACM IUI Workshops 2022, Vol. 10.
Cited by
Sorted by citations
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
Lau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
“, creative writing [77]), and broader societal implications of LLMs [17].”
BackgroundHomogenization Effects of Large Language Models on Human Creative Ideation
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656204 · Cited by 188
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136
“The most common mechanism is to use text formatting such”
InfluentialArt or Artifice? Large Language Models and the False Promise of Creativity
Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., & Wu, C.-S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642731 · Cited by 135
“Despite focusing on short stories, TTCW also critiques story closure in TTCW Fluency4, contrasting with plays that seek non-cathartic endings for specific impacts, as seen in the Brechtian tradition.”
BackgroundMethodInfluential
Metadata
- Authors
- Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, Canada
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-34
- OpenAlex
- W4366547384
- Semantic Scholar
- f9b16559282e5bea0bb072f9e260a4f0af697f4a
- Cited by
- 240
- References
- 57
- Citations by year
- 2026: 58 · 2025: 77 · 2024: 82 · 2023: 22
- FWCI
- 42.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- public-domain
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Artistic and Creative Research
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
- Keywords
- language models, screenwriting, theatre scripts, prompt chaining, semantic coherence, creative writing, industry professionals, plagiarism, algorithmic bias, participatory design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
