Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.

Wordcraft: Story Writing With Large Language Models

Open access
Yuan, Ann, Coenen, Andy, Reif, Emily, Ippolito, Daphne · 2022 · 27th International Conference on Intelligent User Interfaces

Abstract

The latest generation of large neural language models such as GPT-3 have achieved new levels of performance on benchmarks for language understanding and generation. These models have even demonstrated an ability to perform arbitrary tasks without explicit training. In this work, we sought to learn how people might use such models in the process of creative writing. We built Wordcraft, a text editor in which users collaborate with a generative language model to write a story. We evaluated Wordcraft with a user study in which participants wrote short stories with and without the tool. Our results show that large language models enable novel co-writing experiences. For example, the language model is able to engage in open-ended conversation about the story, respond to writers’ custom requests expressed in natural language (such as ”rewrite this text to be more Dickensian”), and generate suggestions that serve to unblock writers in the creative process. Based on these results, we discuss design implications for future human-AI co-writing systems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3490099.3511105

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

7 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 15

  1. Nader Akoury , Shufan Wang , Josh Whiting , Stephen Hood , Nanyun Peng , and Mohit Iyyer . 2020 . STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. Empirical Methods in Natural Language Processing (2020). Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. Empirical Methods in Natural Language Processing (2020).
  2. Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Illia Polosukhin Ashish Vaswani , Noam Shazeer. 2017 . Attention Is All You Need. Neural Information Processing Systems( 2017). Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Illia Polosukhin Ashish Vaswani, Noam Shazeer. 2017. Attention Is All You Need. Neural Information Processing Systems(2017).
  3. Daniel Buschek , Martin Zurn , and Malin Eiband . 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behavior of Native and Non-Native English Writers. ACM ( 2021 ). Daniel Buschek, Martin Zurn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behavior of Native and Non-Native English Writers. ACM (2021).
  4. Chenhao Tan Yangfeng Ji Elizabeth Clark , Anne Spencer Ross and Noah A. Smith . 2018 . Creative Writing with a Machine in the Loop : Case Studies on Slogans and Stories. Intelligent User Interfaces( 2018). Chenhao Tan Yangfeng Ji Elizabeth Clark, Anne Spencer Ross and Noah A. Smith. 2018. Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories. Intelligent User Interfaces(2018).
  5. Katy Gero and Lydia Chilton . 2019. How a Stylistic , Machine-Generated Thesaurus Impacts a Writer’s Process. C&C ( 2019 ). Katy Gero and Lydia Chilton. 2019. How a Stylistic, Machine-Generated Thesaurus Impacts a Writer’s Process. C&C (2019).
  6. Cheng-Zhi Huang . 2020 . AI Song Contest: Human-AI Co-Creation in Songwriting. International Society for Music Information Retrieval Conference (2020). Cheng-Zhi Huang. 2020. AI Song Contest: Human-AI Co-Creation in Songwriting. International Society for Music Information Retrieval Conference (2020).
  7. Christian Remy Michael Mose Biskjaer Jonas Frich , Lindsay MacDonald Vermeulen and Peter Dalsgaard . 2019 . Mapping the Landscape of Creativity Support Tools in HCI.Computer Human Interaction( 2019). Christian Remy Michael Mose Biskjaer Jonas Frich, Lindsay MacDonald Vermeulen and Peter Dalsgaard. 2019. Mapping the Landscape of Creativity Support Tools in HCI.Computer Human Interaction(2019).
  8. Varun Kumar Satyapriya Krishna Yada Pruksachatkun Kai- Wei Chang Jwala Dhamala , Tony Sun and Rahul Gupta . 2021 . Bold : Dataset and metrics for measuring biases in open-ended language generation. arXiv preprint arXiv:2101.11718(2021). Varun Kumar Satyapriya Krishna Yada Pruksachatkun Kai-Wei Chang Jwala Dhamala, Tony Sun and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. arXiv preprint arXiv:2101.11718(2021). · arXiv
  9. Jay Priyadarshi Marjan Ghazvininejad , Xing Shi and Kevin Knight . 2017 . Hafez: an Interactive Poetry Generation System. Association for Computational Linguistics( 2017). Jay Priyadarshi Marjan Ghazvininejad, Xing Shi and Kevin Knight. 2017. Hafez: an Interactive Poetry Generation System. Association for Computational Linguistics(2017).
  10. Jonathan Chen Shao-Yu Chen Shukan Shah-Vishwa Shah Joshua Reno Gillian Smith Matthew Guzdial , Nicholas Liaoand Mark O. Riedl .2019. Friend , Collaborator , Student, Manager : How Design of an AI-Driven Game Level Editor Affects Creators.Computer Human Interaction( 2019 ). Jonathan Chen Shao-Yu Chen Shukan Shah-Vishwa Shah Joshua Reno Gillian Smith Matthew Guzdial, Nicholas Liaoand Mark O. Riedl.2019. Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators.Computer Human Interaction(2019).
  11. Wesley Cheung Zhaochen Luo William Ma Lara J Martin Prithviraj Ammanabrolu , Ethan Tien and Mark O Riedl . 2020 . Story realization: Expanding plot events into sentences . Proceedings of the AAAI Conference on Artificial Intelligence (2020). Wesley Cheung Zhaochen Luo William Ma Lara J Martin Prithviraj Ammanabrolu, Ethan Tien and Mark O Riedl. 2020. Story realization: Expanding plot events into sentences. Proceedings of the AAAI Conference on Artificial Intelligence (2020).
  12. Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , Ilya Sutskever , 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  13. Nathan Dass Sadao Kurohashi Dan Jurafsky Diyi Yang Reid Pryzant , Richard Diehl Martinez . 2020. Automatically Neutralizing Subjective Bias in Text. AAAI ( 2020 ). Nathan Dass Sadao Kurohashi Dan Jurafsky Diyi Yang Reid Pryzant, Richard Diehl Martinez. 2020. Automatically Neutralizing Subjective Bias in Text. AAAI (2020).
  14. Shi Feng Dan Klein Sameer Singh Tony Z. Zhao , Eric Wallace. 2021 . Calibrate Before Use: Improving Few-Shot Performance of Language Models . Proceedings of Machine Learning Research(2021) . Shi Feng Dan Klein Sameer Singh Tony Z. Zhao, Eric Wallace. 2021. Calibrate Before Use: Improving Few-Shot Performance of Language Models. Proceedings of Machine Learning Research(2021).
  15. Yonghui Wu. 2018. Smart compose: Using neural networks to help write emails. Google AI Blog ( 2018 ). Yonghui Wu. 2018. Smart compose: Using neural networks to help write emails. Google AI Blog (2018).

Cited by

See all 389

Metadata

Authors
Yuan, A., Coenen, A., Reif, E., & Ippolito, D.
Affiliations
Google (United States) (US)
Venue
27th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2022-03-22
Pages
841-852
OpenAlex
W4221055872
Cited by
374
References
12
Citations by year
2026: 65 · 2025: 123 · 2024: 121 · 2023: 59 · 2022: 6
FWCI
42.71
Type
conference paper
Language
en
Open access
Locations
ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, creative writing, story writing, text editor, GPT-3, text generation, user study, AI writing tools

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Wordcraft: Story Writing With Large Language Models. https://app.cochat.ai/papers/10.1145/3490099.3511105
The paper
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022). Wordcraft: Story Writing With Large Language Models. 27th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3490099.3511105

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.