The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Wordcraft: Story Writing With Large Language Models
Open accessAbstract
The latest generation of large neural language models such as GPT-3 have achieved new levels of performance on benchmarks for language understanding and generation. These models have even demonstrated an ability to perform arbitrary tasks without explicit training. In this work, we sought to learn how people might use such models in the process of creative writing. We built Wordcraft, a text editor in which users collaborate with a generative language model to write a story. We evaluated Wordcraft with a user study in which participants wrote short stories with and without the tool. Our results show that large language models enable novel co-writing experiences. For example, the language model is able to engage in open-ended conversation about the story, respond to writers’ custom requests expressed in natural language (such as ”rewrite this text to be more Dickensian”), and generate suggestions that serve to unblock writers in the creative process. Based on these results, we discuss design implications for future human-AI co-writing systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical studies describing how users evaluate LLMs are still a minority and difficult to generalise due to a variation of participants, methods, systems, and disciplinary perspectives [13,27,28,38,47].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [47]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite these limitations, playwrights identified utility at different stages of creative writing, from brainstorming through the summarisation of relevant sources to editing [27,42,47].”
From Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes (2024) · cites this paper as [47]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These interfaces typically provide end-users with only one text area for inputs, which is frequently shared with the output, and one control panel to confgure global settings for the parameters [1,69,85,109].”
From Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models (2023) · cites this paper as [109]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Students can continually alter prompts to refine the model output helping them to 'build-up' a solution [196].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [196]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“End-user studies suggest that prompting is challenging, with nonexpert users making various errors and adopting ineffective strategies-a reflection of the demand on users' metacognitive monitoring and control [33,41,43,85,103,174,205].”
From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [205]
Experimental — accuracy not yet measured
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“[38] proposed the idea of designing a prompt that contains several examples of the target task, written in natural language.”
BackgroundInfluential2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,800
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“Despite the simplicity of this API, large language models (LLMs) have demonstrated proficiency at a wide variety of tasks—from storytelling [1, 28] to code synthesis [4, 9] and email auto-completion [40].”
BackgroundLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 2×
“In the exit interview, one user commented about the chatbot: ” I loved that it felt like a more targeted version of Google, and I could do background research while writing ” (U21).”
MethodCTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · cited 1×
“Researchers have also built story generation models that incorporate event sequences [33], desired topic [29], and story title [2] as control signals.”
BackgroundExtracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · cited 1×
“LLMs have also been found to output memorized passages from their training data [26].”
BackgroundGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 404
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · cited 1×
“Neural language models, such as GPT-2 [34] or GPT-Neo [5], are neural networks that are trained only to predict the next word in a sequence given the previous words (aka a prompt).”
Background
Research works printed but not linked: 15
- Nader Akoury , Shufan Wang , Josh Whiting , Stephen Hood , Nanyun Peng , and Mohit Iyyer . 2020 . STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. Empirical Methods in Natural Language Processing (2020). Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. Empirical Methods in Natural Language Processing (2020).
- Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Illia Polosukhin Ashish Vaswani , Noam Shazeer. 2017 . Attention Is All You Need. Neural Information Processing Systems( 2017). Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Illia Polosukhin Ashish Vaswani, Noam Shazeer. 2017. Attention Is All You Need. Neural Information Processing Systems(2017).
- Daniel Buschek , Martin Zurn , and Malin Eiband . 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behavior of Native and Non-Native English Writers. ACM ( 2021 ). Daniel Buschek, Martin Zurn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behavior of Native and Non-Native English Writers. ACM (2021).
- Chenhao Tan Yangfeng Ji Elizabeth Clark , Anne Spencer Ross and Noah A. Smith . 2018 . Creative Writing with a Machine in the Loop : Case Studies on Slogans and Stories. Intelligent User Interfaces( 2018). Chenhao Tan Yangfeng Ji Elizabeth Clark, Anne Spencer Ross and Noah A. Smith. 2018. Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories. Intelligent User Interfaces(2018).
- Katy Gero and Lydia Chilton . 2019. How a Stylistic , Machine-Generated Thesaurus Impacts a Writer’s Process. C&C ( 2019 ). Katy Gero and Lydia Chilton. 2019. How a Stylistic, Machine-Generated Thesaurus Impacts a Writer’s Process. C&C (2019).
- Cheng-Zhi Huang . 2020 . AI Song Contest: Human-AI Co-Creation in Songwriting. International Society for Music Information Retrieval Conference (2020). Cheng-Zhi Huang. 2020. AI Song Contest: Human-AI Co-Creation in Songwriting. International Society for Music Information Retrieval Conference (2020).
- Christian Remy Michael Mose Biskjaer Jonas Frich , Lindsay MacDonald Vermeulen and Peter Dalsgaard . 2019 . Mapping the Landscape of Creativity Support Tools in HCI.Computer Human Interaction( 2019). Christian Remy Michael Mose Biskjaer Jonas Frich, Lindsay MacDonald Vermeulen and Peter Dalsgaard. 2019. Mapping the Landscape of Creativity Support Tools in HCI.Computer Human Interaction(2019).
- Varun Kumar Satyapriya Krishna Yada Pruksachatkun Kai- Wei Chang Jwala Dhamala , Tony Sun and Rahul Gupta . 2021 . Bold : Dataset and metrics for measuring biases in open-ended language generation. arXiv preprint arXiv:2101.11718(2021). Varun Kumar Satyapriya Krishna Yada Pruksachatkun Kai-Wei Chang Jwala Dhamala, Tony Sun and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. arXiv preprint arXiv:2101.11718(2021). · arXiv
- Jay Priyadarshi Marjan Ghazvininejad , Xing Shi and Kevin Knight . 2017 . Hafez: an Interactive Poetry Generation System. Association for Computational Linguistics( 2017). Jay Priyadarshi Marjan Ghazvininejad, Xing Shi and Kevin Knight. 2017. Hafez: an Interactive Poetry Generation System. Association for Computational Linguistics(2017).
- Jonathan Chen Shao-Yu Chen Shukan Shah-Vishwa Shah Joshua Reno Gillian Smith Matthew Guzdial , Nicholas Liaoand Mark O. Riedl .2019. Friend , Collaborator , Student, Manager : How Design of an AI-Driven Game Level Editor Affects Creators.Computer Human Interaction( 2019 ). Jonathan Chen Shao-Yu Chen Shukan Shah-Vishwa Shah Joshua Reno Gillian Smith Matthew Guzdial, Nicholas Liaoand Mark O. Riedl.2019. Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators.Computer Human Interaction(2019).
- Wesley Cheung Zhaochen Luo William Ma Lara J Martin Prithviraj Ammanabrolu , Ethan Tien and Mark O Riedl . 2020 . Story realization: Expanding plot events into sentences . Proceedings of the AAAI Conference on Artificial Intelligence (2020). Wesley Cheung Zhaochen Luo William Ma Lara J Martin Prithviraj Ammanabrolu, Ethan Tien and Mark O Riedl. 2020. Story realization: Expanding plot events into sentences. Proceedings of the AAAI Conference on Artificial Intelligence (2020).
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , Ilya Sutskever , 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Nathan Dass Sadao Kurohashi Dan Jurafsky Diyi Yang Reid Pryzant , Richard Diehl Martinez . 2020. Automatically Neutralizing Subjective Bias in Text. AAAI ( 2020 ). Nathan Dass Sadao Kurohashi Dan Jurafsky Diyi Yang Reid Pryzant, Richard Diehl Martinez. 2020. Automatically Neutralizing Subjective Bias in Text. AAAI (2020).
- Shi Feng Dan Klein Sameer Singh Tony Z. Zhao , Eric Wallace. 2021 . Calibrate Before Use: Improving Few-Shot Performance of Language Models . Proceedings of Machine Learning Research(2021) . Shi Feng Dan Klein Sameer Singh Tony Z. Zhao, Eric Wallace. 2021. Calibrate Before Use: Improving Few-Shot Performance of Language Models. Proceedings of Machine Learning Research(2021).
- Yonghui Wu. 2018. Smart compose: Using neural networks to help write emails. Google AI Blog ( 2018 ). Yonghui Wu. 2018. Smart compose: Using neural networks to help write emails. Google AI Blog (2018).
Cited by
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“Furthermore, they can showcase autonomy through exhibiting creativity like coming up with novel ideas, stories, or solutions that have not been explicitly programmed into them [97,98].”
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
“These chatbots can generate human-like text that is unparalleled in its sophistication, ushering in novel applications across a multitude of sectors [23], [7], [48], [55].”
BackgroundThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Metadata
- Authors
- Yuan, A., Coenen, A., Reif, E., & Ippolito, D.
- Affiliations
- Google (United States) (US)
- Venue
- 27th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2022-03-22
- Pages
- 841-852
- OpenAlex
- W4221055872
- Semantic Scholar
- fb5c11bbf63884f75d2da615fbf37a3bcfa2bd20
- Cited by
- 374
- References
- 12
- Citations by year
- 2026: 65 · 2025: 123 · 2024: 121 · 2023: 59 · 2022: 6
- FWCI
- 42.71
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, creative writing, story writing, text editor, GPT-3, text generation, user study, AI writing tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Wordcraft: Story Writing With Large Language Models. https://app.cochat.ai/papers/10.1145/3490099.3511105
- The paper
- Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022). Wordcraft: Story Writing With Large Language Models. 27th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3490099.3511105
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.