Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
The DOI matches its Crossref record; 29 of the 30 references it cites were found in a registry or an index.
Abstract
As the quality of natural language generated by artificial intelligence systems improves, writing interfaces can support interventions beyond grammar-checking and spell-checking, such as suggesting content to spark new ideas. To explore the possibility of machine-in-the-loop creative writing, we performed two case studies using two system prototypes, one for short story writing and one for slogan writing. Participants in our studies were asked to write with a machine in the loop or alone (control condition). They assessed their writing and experience through surveys and an open-ended interview. We collected additional assessments of the writing from Amazon Mechanical Turk crowdworkers. Our findings indicate that participants found the process fun and helpful and could envision use cases for future systems. At the same time, machine suggestions do not necessarily lead to better written artifacts. We therefore suggest novel natural language models and design choices that may better support creative writing.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“researchers have begun to frame AI as a”
From AI-Integrated Scaffolding to Enhance Agency and Creativity in K-12 English Language Learners: A Systematic Review (2025) · cites this paper as [8]
In the citing paper
“Specifically, AI-supported creative writing processes are currently categorized into story development and character creation [16, 26, 27, 80, 106].”
From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [27]
In the citing paper
“Specifically, AI-supported creative writing processes are currently categorized into story development and character creation [16, 26, 27, 80, 106].”
From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [26]
In the citing paper
“Prior studies have shown that AI reduces creative workloads by providing valuable suggestions [10, 33] and refining content for professional use [26, 43, 83], thereby helping writers overcome creative blocks.”
From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [26]
In the citing paper
“Within human-computer interaction research, language models are increasingly treated as participants in writing, with machine-in-the-loop studies documenting cycles of proposal and evaluation and negotiation of control across composing phases (Bhat et al., 2021, 2023; Buschek et al., 2018; Clark et al., 2018).”
From Identity, Agency, and Human–AI Mediation in L2 Academic Writing: An Interactional Study (2026) · cites this paper as Clark et al., 2018
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“Second, recent studies show that machines outperform humans in some tasks [15, 24, 31], including identifying which message will be retweeted more on Twitter.”
BackgroundI.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“Researchers have made signifcant progress in advancing artifcial intelligence since Turing famously asked “Can machines think?” [32].”
BackgroundShow and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“Neural language models have been used for various natural language generation tasks, including image captioning [33],”
MethodComputing Machinery and Intelligence (1950)
Turing, A. (2004) · The Essential Turing · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,118
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“A sentence-level language model [23] defnes the probability distribution over the next word, from within a predefned vocabulary, conditioned on the left context (i.”
BackgroundBoden, M. A. (2004) · doi:10.4324/9780203508527 · Cited by 2,775
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 2×
“This concept resonates with “mixed-initiative user interfaces” [17].”
BackgroundA cognitive process theory of writing
Flower, L., & Hayes, J. R. (1981) · College Composition & Communication · doi:10.58680/ccc198115885 · Cited by 1,025 · cited 1×
“For example, Flower and Hayes [10] describe the hierarchical nature of the creative writing process; future system designs could refect knowledge about the cognitive processes of writing to better support the writing process.”
BackgroundA Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.-Y., Gao, J., & Dolan, B. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1020 · Cited by 764 · cited 1×
“conversational modeling [28], and poetry generation [13].”
Self-citationSoylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686 · cited 2×
“However, Soylent assists in shortening text and editing grammar rather than providing content and new ideas.”
BackgroundFails, J. A., & Olsen, D. R. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604056 · Cited by 633 · cited 1×
“1In contrast, human-in-the-loop machine learning actively includes humans in the process of training machine learning models by asking humans to provide feedback such as labeling diffcult examples or suggesting new features [4, 6, 9, 20].”
BackgroundUnbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531 · cited 1×
“1In contrast, human-in-the-loop machine learning actively includes humans in the process of training machine learning models by asking humans to provide feedback such as labeling diffcult examples or suggesting new features [4, 6, 9, 20].”
BackgroundAllen, J. E., Guinn, C. I., & Horvtz, E. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.796083 · Cited by 442
Visual Recognition with Humans in the Loop
Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P., & Belongie, S. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_32 · Cited by 402 · cited 1×
“1In contrast, human-in-the-loop machine learning actively includes humans in the process of training machine learning models by asking humans to provide feedback such as labeling diffcult examples or suggesting new features [4, 6, 9, 20].”
BackgroundMixed-initiative co-creativity
Yannakakis, G. N., Liapis, A., & Alexopoulos, C. (2014) · OAR@UM (University of Malta) · Cited by 299
Sternberg, R. J. (2005) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.04.003 · Cited by 261
Tan, C., Lee, L., & Pang, B. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1017 · Cited by 188 · cited 2×
“[31] that it is hard for humans to evaluate the quality of writing.”
BackgroundSelf-citationGhazvininejad, M., Shi, X., Choi, Y., & Knight, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1126 · Cited by 142 · cited 1×
“conversational modeling [28], and poetry generation [13].”
Hafez: an Interactive Poetry Generation System
Ghazvininejad, M., Shi, X., Priyadarshi, J., & Knight, K. K. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4008 · Cited by 129 · cited 1×
“Past collaborative creativity research has also looked at other related writing tasks include headlines for newspaper articles [12] and lines for poetry [14], and other”
BackgroundInteractive improvisation with a robotic marimba player
Hoffman, G., & Weinberg, G. (2011) · Autonomous Robots · doi:10.1007/s10514-011-9237-0 · Cited by 121 · cited 1×
“artistic domains like music [16] and dance [18].”
BackgroundEnsemble: exploring complementary strengths of leaders and crowds in creative collaboration
Kim, J. O., Cheng, J., & Bernstein, M. S. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531638 · Cited by 109 · cited 1×
“For example, “Ensemble” was a system that had multiple participants work together to write a single story [21].”
BackgroundCreative Help: A Story Writing Assistant
Roemmele, M., & Gordon, A. S. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-27036-4_8 · Cited by 83 · cited 4×
“In “Creative Help” [26], writers are offered suggestions as they write stories, but only when they explicitly request them (i.e., a pull method of interaction).”
BackgroundInfluentialSay Anything: A Massively Collaborative Open Domain Story Writing Companion
Swanson, R., & Gordon, A. S. (2006) · Lecture notes in computer science · doi:10.1007/978-3-540-89454-4_5 · Cited by 78 · cited 2×
“Swanson and Gordon’s “Say Anything” [30] provides suggestions for writing short stories by prompting writers with full sentences retrieved from a database of stories scraped from the web after every turn of writing.”
BackgroundDocument Context Language ModelsPreprint
Ji, Y., Cohn, T., Kong, L., Dyer, C., & Eisenstein, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03962 · Cited by 66 · cited 1×
“account of left context from both the current sentence and the previous sentence [19].”
BackgroundSelf-citationBRAINSUP: Brainstorming Support for Creative Sentence Generation
Özbal, G., Pighin, D., & Strapparava, C. (2013) · Cited by 39
Garfield, M. J. (2008) · Springer eBooks · doi:10.1007/978-3-540-48716-6_34 · Cited by 28 · cited 1×
“First, writers often experience “cognitive inertia,” a phenomenon known in the writing domain as “writer’s block” [11].”
BackgroundJacob, M., & Magerko, B. S. (2015) · Creativity and Cognition · doi:10.1145/2757226.2757400 · Cited by 28 · cited 1×
“Past collaborative creativity research has also looked at other related writing tasks include headlines for newspaper articles [12] and lines for poetry [14], and other artistic domains like music [16] and dance [18].”
BackgroundGatti, L., Özbal, G., Guerini, M., Stock, O., & Strapparava, C. (2016) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2876456.2879469 · Cited by 16 · cited 1×
“Past collaborative creativity research has also looked at other related writing tasks include headlines for newspaper articles [12] and lines for poetry [14], and other”
Background
Research works printed but not linked: 1
- Myle Ott , Yejin Choi , Claire Cardie , and Jeffrey T . Hancock . 2011 . Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of ACL. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of ACL.
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“ful in many scenarios: communicating model predictions [17, 30, 33, 56], teaching humans tasks like translation [48] or content moderation [28], augmenting human analysis procedure [28] or creativity [14], legal imperatives [51, 64], and others. An assumption underlying all these case is that a human will be in the lo…”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
“ful in many scenarios: communicating model predictions [17, 30, 33, 56], teaching humans tasks like translation [48] or content moderation [28], augmenting human analysis procedure [28] or creativity [14], legal imperatives [51, 64], and others. An assumption underlying all these case is that a human will be in the lo…”
BackgroundTeachers’ AI digital competencies and twenty-first century skills in the post-pandemic world
(2023) · doi:10.1007/s11423-023-10203-6 · Cited by 734
“Educators should incorporate learning activities and assessments for students to create content through creative writing (Clark et al., 2018), music composition (Lopez-Rincon et al., 2018) and stylising painting (Hertz-mann, 2018).”
BackgroundWu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500
Metadata
- Authors
- Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A.
- Affiliations
- University of Washington (US)
- University of Colorado Boulder (US)
- Venue
- 23rd International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2018-03-05
- Pages
- 329-340
- OpenAlex
- W2790165607
- Cited by
- 257
- References
- 30
- Citations by year
- 2026: 34 · 2025: 39 · 2024: 47 · 2023: 30 · 2022: 25 · 2021: 35 · 2020: 23 · 2019: 19 · 2018: 5
- FWCI
- 8.37
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- National Science Foundation Graduate Research Fellowship (DGE-1256082)
- Defense Advanced Research Projects Agency (W911NF-15-1-0543)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- creative writing, natural language generation, short story writing, writing quality, user experience, AI-assisted writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.