Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Open accessThe DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Abstract
While developing a story, novices and published writers alike have had to look outside themselves for inspiration. Language models have recently been able to generate text fluently, producing new stochastic narratives upon request. However, effectively integrating such capabilities with human cognitive faculties and creative processes remains challenging. We propose to investigate this integration with a multimodal writing support interface that offers writing suggestions textually, visually, and aurally. We conduct an extensive study that combines elicitation of prior expectations before writing, observation and semi-structured interviews during writing, and outcome evaluations after writing. Our results illustrate the individual and situational variation in machine-in-the-loop writing approaches, suggestion acceptance, and ways the system is helpful. Centrally, we report how participants perform integrative leaps , by which they do cognitive work to integrate suggestions of varying semantic relevance into their developing stories. We interpret these findings, offering modeling and design recommendations for future creative writing support technologies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Translating support provides preliminary drafts or helps the user to restructure their ideas in a different form (e.g., point form to prose), but encourages the user to write and integrate ideas on their own [21,122,124].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [122]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reviewing support is delivered in qualitative form, such as suggestions or summaries [7,114,122].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [122]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lenses can also be assembled together to view the same generation outputs through multiple representations [64,82].”
From Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models (2023) · cites this paper as [82]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
The Interpretation of Cultures
Fenn, R., & Geertz, C. (1974) · Journal for the Scientific Study of Religion · doi:10.2307/1384392 · Cited by 19,154 · cited 2×
“writing and to observe immediate reaction and, to the extent possible, raw and unmediated answers, allowed us to produce "thick description" [43], as noted earlier.”
BackgroundMethodDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 1×
“"In which editor was the text that you wrote more creative?") and four items on cognitive load adapted from the NASA TLX survey [47] (e.”
MethodBaddeley, A. D., & Hitch, G. (1974) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60452-1 · Cited by 9,860
Marian, V., Blumenfeld, H. K., & Kaushanskaya, M. (2007) · Journal of Speech, Language, and Hearing Research · doi:10.1044/1092-4388(2007/067 · Cited by 1,991
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · cited 1×
“In parallel, models with implicitly richer knowledge bases [74] may also extend more diverse suggestions to users, inding interesting relations with aspects of their writing, and assisting them in performing contextually appropriate and creatively fulilling integrations.”
BackgroundSome observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
Creative Cognition: Theory, Research, and Applications
Finke, R. A., Ward, T. B., & Smith, S. M. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/7722.001.0001 · Cited by 1,433 · cited 1×
“More recent psychological approaches to studying creative writing have emphasized the role of retrieval, conceptual combination, and analogical mapping as some of the fundamental cognitive processes that explain creative cognition [35, 48, 95].”
BackgroundPaivio, A. (1990) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195066661.001.0001 · Cited by 1,149
Research works printed but not linked: 21
- Kenneth Arnold Kai-Wei Chang and Adam Kalai. 2017. Counterfactual Language Model Adaptation for Suggesting Phrases. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) . 49–54.
- Eden Bensaid. 2021. Multimodal generative models for storytelling. Master’s Thesis. Massachusetts Institute of Technology.
- Eden Bensaid Mauro Martino Benjamin Hoover Jacob Andreas and Hendrik Strobelt. 2021. FairyTailor: A multimodal generative framework for storytelling. arXiv:2108.04324. Retrieved from https://arxiv.org/abs/2108.04324. · arXiv
- Lukas Bieringer Kathrin Grosse Michael Backes and Katharina Krombholz. 2021. Mental Models of Adversarial Machine Learning. In ICCV 2021 Workshop on Adversarial Robustness in the Real World .
- Margaret A. Boden. 2004. The creative mind: Myths and mechanisms. Routledge.
- Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger T. J. Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeff Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020) 1877–1901.
- Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B. Chilton. 2020. How novelists use generative language models: An exploratory user study. In Proceedings of the HAI-GEN+ user2agent@ IUI.
- Orson Scott Card. 1990. How to Write Science Fiction and Fantasy. Writer’s digest books Cincinnati, OH.
- Michelene T. H. Chi. 2000. Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. Advances in Instructional Psychology 5 (2000), 161–238.
- Stephen R. Donaldson. 2008. The Gap Into Conflict: The Real Story. CNIB.
- Angela Fan Mike Lewis and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 889–898.
- 10.2307/356600 · doi:10.2307/356600 · Paper page
- 10.1207/s15516709cog1404_3 · doi:10.1207/s15516709cog1404_3 · Paper page
- Roger Levy and Galen Andrew. 2006. Tregex and Tsurgeon: Tools for querying and manipulating tree data structures.. In Proceedings of the LREC. Citeseer, 2231–2234.
- Ellen W. Nold. 1981. Revising. Writing: The Nature, Development, and Teaching of Written Communication 2 (1981), 67–79.
- Ben Pietrzak Ben Swanson Kory Mathewson Monica Dinculescu and Sherol Chen. 2021. Story centaur: Large language model few shot learning as a creative writing tool. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations.
- Paul A. Prior. 2005. A Sociocultural Theory of Writing. In The Handbook of Writing Research . Guilford Press 54–66.
- Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning . PMLR 8748–8763.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- David Ramsay, Ishwarya Ananthabhotla, and Joseph Paradiso. 2019. The intrinsic memorability of everyday sounds. In Proceedings of the Audio Engineering Society Conference: 2019 AES International Conference on Immersive and Interactive Audio. Audio Engineering Society.
- Thomas B. Ward and E. Thomas Lawson. 2009. Creative Cognition in Science Fiction and Fantasy Writing . Cambridge University Press.
Cited by
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · National Bureau of Economic Research · doi:10.3386/w31161 · Cited by 995
“writing suggestion tool, Singh et al. (2022) finds that four of the 23 study participants refused to engage with AI suggestions.”
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265
“Studies have investigated interface design factors and interactions with writing assistants that directly or indirectly support inspiration [17, 70, 95, 105], language prof-ciency [27], story writing [95, 105], text revision [36, 107] or creative writing [33, 46].”
BackgroundDraxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3637875 · Cited by 186
“Related, Singh et al. [96] describes how writers actively invest into taking ‘leaps’ to fit their story around a suggested phrase.”
BackgroundHomogenization Effects of Large Language Models on Human Creative Ideation
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656204 · Cited by 188
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G., & Matejka, J. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 163
“3DALL-E’s interdisciplinary design knowledge is both a strength of AI pretraining as well as the ability of designers to make integrative leaps to meet the AI halfway [81].”
Background
Metadata
- Authors
- Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L.
- Affiliations
- MIT Media Lab (US)
- Harvard University (US)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-09-23
- Volume
- 30
- Issue
- 5
- Pages
- 1-57
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4213451626
- Cited by
- 119
- References
- 77
- Citations by year
- 2026: 15 · 2025: 32 · 2024: 38 · 2023: 23
- FWCI
- 12.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion, cc-by)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- NSF (2107391)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Artistic and Creative Research
- Keywords
- creative writing, language models, text generation, human-AI collaboration, story development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.