Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Open accessThe DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Abstract
While developing a story, novices and published writers alike have had to look outside themselves for inspiration. Language models have recently been able to generate text fluently, producing new stochastic narratives upon request. However, effectively integrating such capabilities with human cognitive faculties and creative processes remains challenging. We propose to investigate this integration with a multimodal writing support interface that offers writing suggestions textually, visually, and aurally. We conduct an extensive study that combines elicitation of prior expectations before writing, observation and semi-structured interviews during writing, and outcome evaluations after writing. Our results illustrate the individual and situational variation in machine-in-the-loop writing approaches, suggestion acceptance, and ways the system is helpful. Centrally, we report how participants perform integrative leaps , by which they do cognitive work to integrate suggestions of varying semantic relevance into their developing stories. We interpret these findings, offering modeling and design recommendations for future creative writing support technologies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Translating support provides preliminary drafts or helps the user to restructure their ideas in a different form (e.g., point form to prose), but encourages the user to write and integrate ideas on their own [21,122,124].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [122]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reviewing support is delivered in qualitative form, such as suggestions or summaries [7,114,122].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [122]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lenses can also be assembled together to view the same generation outputs through multiple representations [64,82].”
From Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models (2023) · cites this paper as [82]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
The Interpretation of Cultures
Fenn, R., & Geertz, C. (1974) · Journal for the Scientific Study of Religion · doi:10.2307/1384392 · Cited by 19,154 · cited 2×
“writing and to observe immediate reaction and, to the extent possible, raw and unmediated answers, allowed us to produce "thick description" [43], as noted earlier.”
BackgroundMethodDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 1×
“"In which editor was the text that you wrote more creative?") and four items on cognitive load adapted from the NASA TLX survey [47] (e.”
MethodBaddeley, A. D., & Hitch, G. (1974) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60452-1 · Cited by 9,860
Marian, V., Blumenfeld, H. K., & Kaushanskaya, M. (2007) · Journal of Speech, Language, and Hearing Research · doi:10.1044/1092-4388(2007/067 · Cited by 1,991
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · cited 1×
“In parallel, models with implicitly richer knowledge bases [74] may also extend more diverse suggestions to users, inding interesting relations with aspects of their writing, and assisting them in performing contextually appropriate and creatively fulilling integrations.”
BackgroundSome observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
Creative Cognition: Theory, Research, and Applications
Finke, R. A., Ward, T. B., & Smith, S. M. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/7722.001.0001 · Cited by 1,433 · cited 1×
“More recent psychological approaches to studying creative writing have emphasized the role of retrieval, conceptual combination, and analogical mapping as some of the fundamental cognitive processes that explain creative cognition [35, 48, 95].”
BackgroundPaivio, A. (1990) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195066661.001.0001 · Cited by 1,149
Research works printed but not linked: 21
- Kenneth Arnold Kai-Wei Chang and Adam Kalai. 2017. Counterfactual Language Model Adaptation for Suggesting Phrases. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) . 49–54.
- Eden Bensaid. 2021. Multimodal generative models for storytelling. Master’s Thesis. Massachusetts Institute of Technology.
- Eden Bensaid Mauro Martino Benjamin Hoover Jacob Andreas and Hendrik Strobelt. 2021. FairyTailor: A multimodal generative framework for storytelling. arXiv:2108.04324. Retrieved from https://arxiv.org/abs/2108.04324. · arXiv
- Lukas Bieringer Kathrin Grosse Michael Backes and Katharina Krombholz. 2021. Mental Models of Adversarial Machine Learning. In ICCV 2021 Workshop on Adversarial Robustness in the Real World .
- Margaret A. Boden. 2004. The creative mind: Myths and mechanisms. Routledge.
- Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger T. J. Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeff Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020) 1877–1901.
- Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B. Chilton. 2020. How novelists use generative language models: An exploratory user study. In Proceedings of the HAI-GEN+ user2agent@ IUI.
- Orson Scott Card. 1990. How to Write Science Fiction and Fantasy. Writer’s digest books Cincinnati, OH.
- Michelene T. H. Chi. 2000. Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. Advances in Instructional Psychology 5 (2000), 161–238.
- Stephen R. Donaldson. 2008. The Gap Into Conflict: The Real Story. CNIB.
- Angela Fan Mike Lewis and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 889–898.
- 10.2307/356600 · doi:10.2307/356600 · Paper page
- 10.1207/s15516709cog1404_3 · doi:10.1207/s15516709cog1404_3 · Paper page
- Roger Levy and Galen Andrew. 2006. Tregex and Tsurgeon: Tools for querying and manipulating tree data structures.. In Proceedings of the LREC. Citeseer, 2231–2234.
- Ellen W. Nold. 1981. Revising. Writing: The Nature, Development, and Teaching of Written Communication 2 (1981), 67–79.
- Ben Pietrzak Ben Swanson Kory Mathewson Monica Dinculescu and Sherol Chen. 2021. Story centaur: Large language model few shot learning as a creative writing tool. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations.
- Paul A. Prior. 2005. A Sociocultural Theory of Writing. In The Handbook of Writing Research . Guilford Press 54–66.
- Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning . PMLR 8748–8763.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- David Ramsay, Ishwarya Ananthabhotla, and Joseph Paradiso. 2019. The intrinsic memorability of everyday sounds. In Proceedings of the Audio Engineering Society Conference: 2019 AES International Conference on Immersive and Interactive Audio. Audio Engineering Society.
- Thomas B. Ward and E. Thomas Lawson. 2009. Creative Cognition in Science Fiction and Fantasy Writing . Cambridge University Press.
Cited by
Items 101–124 of 124
Understanding How Paper Writers Use AI-Generated Captions in Figure Caption Writing
Ng, H. Y., Hsu, T.-Y., Min, J., Kim, S., Rossi, R. A., Yu, T., Jung, H., & Huang, T.-H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-8912-5_8 · Cited by 4
Writing with Technologies: Reimagining the Author
Spencer, A. (2025) · Springer Proceedings in Humanities and Social Sciences · doi:10.1007/978-3-031-88334-7_6 · Cited by 1
Ownership, Accuracy, and Aesthetics: University Writers’ Perceptions of GenAI Poetry
Nicholes, J. (2025) · Written Communication · doi:10.1177/07410883251349195 · Cited by 2
“Findings are partially in line with previous scholarship on feelings of ownership decreasing through the use of GenAI (Guo et al., 2024; Singh et al., 2023; Tsao & Nogues, 2024), as well as perhaps a general bias against AI concerning artistic output (Chamberlain et al., 2018).”
Beyond the human pen: The role of artificial intelligence in literary creation
Ali, E. H. F., Kottaparamban, M., Ahmed, F. E. Y., Usmani, S., Hamd, M., Ibrahim, M. K., & Hamed, S. (2025) · Research Journal in Advanced Humanities · doi:10.58256/c0ymkm61 · Cited by 12
“Predictive text systems not only prompt the writing process, but they also offer prompt and diversification of writing expression (Singh, 2023).”
Augmenting Human Cognition Through Generative AI
Rogers, Y., Reicherts, L., Zhang, Z. T., & Hassib, M. (2026) · Springer eBooks · doi:10.1007/978-3-031-61050-9_10-1 · Cited by 1
Wang, Y., Sun, G., Fu, Z., & Liang, R. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50356-6 · Cited by 4
“Additionally, visualization techniques can assist in text proofreading and refinement by providing graphical feedback that helps users identify and correct errors, ultimately improving text quality and accuracy [21].”
Na, W. A. N. G. (2026) · Journal of Mechanics in Medicine and Biology · doi:10.1142/s0219519426500326 · Cited by 1
Rafner, J. F. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197698181.013.0043 · Cited by 4
Cloudy, J., Banks, J., & Bowman, N. D. (2022) · Technology Mind and Behavior · doi:10.1037/tmb0000083.supp
Використання ChatGPT як асистента у творчому письмі
Фіялка, С. Б., & Гончарук, Т. Г. (2024) · Технологія і техніка друкарства · doi:10.20535/2077-7264.1(83).2024.297845
Suggestions and Prompts of Foreign Language Activities in Expressive Writing
Tyrou, I. (2024) · International Journal of Education (IJE) · doi:10.5121/ije.2024.12202
“The improvement in learning motivation and creativity, as well as foreign language fluency and originality is highlighted by the international literature [10], [11], [12], [8], [13], [14], [15].”
BackgroundA Novel Multilingual Human-AI Collaborative Writing Design for Indian Languages
Pendharkar, G., Shibani, A., & Rajalakshmi, R. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-05855-3_12
Compile-Time QoS Scheme for Deep Learning Inferences
Hong, S., Kim, H., & Han, H. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759846
“Recent advancements in deep learning (DL) research have expanded its application to fields requiring expertise and creativity, such as finance [5], medicine [35], law [4], art [39], literary creation [42], and entertainment [21].”
StarBurst: Aiding Design Ideation Through AI-Generated Remote Associations
Zhang, P., Fang, R., Liu, F., Ye, Y., Cui, S., & Yin, M. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2582774 · Cited by 1
“…solutions for open-ended problems (Kittur et al., 2019; Taramigkou et al., 2017), creating stories based on broad prompts (Qin et al., 2024; Singh et al., 2023), or conducting design tasks centered on open-ended topics (Choi et al., 2024; Chuan-En Lin et al., 2025; Kang et al., 2021), the…”
TegakARi: Augmenting Creative Drawing With Audio and Visual Cues
Iyer, D., Uhde, A., Karlgren, K., & Kuzuoka, H. (2024) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3726986.3727899
“They present such cues to the user to increase the chances of sparking new ideas through new associations [69, 78].”
Privacy-Aware Real-Time Stream Processing for Health Data Using Federated Learning over Kafka
Ahmed, A., & Sinha, N. (2025) · IEEE International Conference on Intelligent Reality (ICIR) · doi:10.1109/icir68135.2025.11361565
“Beyond removing names, systems must use encryption or random noise to conceal personal data while remaining functional [9].”
Zhang, Y., Jia, K., Zhang, H. J., Zhu, K., Meng, C., Zhang, J., Li, Z., Chen, P., & Sun, L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791460 · Cited by 1
“Singh et al. systematically characterize the cognitive writing work by which users interacting with multimodal AI make “integrative leaps,” showing that effective interaction design and material presentation are critical for triggering new connections [96].”
A Multimodal Interactive System for Autobiographical Memory: AI-Assisted Reflection and Expression
Zhong, C., Yu, J., Wu, W., Wu, S., Shen, H., & Zhang, K. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8256-3_2 · Cited by 1
EmoArc: Interactive Emotion Graph for Human-AI Collaborative Writing
Amin, R. M., Özdemir, S. B., Mayer, S., & Butz, A. M. (2026) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3811427.3811429
“Systems such as StanceVis Prime [16], and Emotion-Vis [39] provide richer temporal and semantic views of affective information, while multimodal tools like Speak From Heart [18] integrate emotional signals across text, audio, or video.”
Author and Scribe: Authoring and Authorizing With and Without Generative AI
Bylieva, D. S., Nordmann, A., & Vida, K. (2026) · Philosophy of engineering and technology · doi:10.1007/978-981-95-8023-1_9 · Cited by 3
Mapping English storytelling in GenAI-enhanced digital multimodal composing: an intervention study
Ranjbaran, F., Zare, J., & Altamimi, A. A. M. (2026) · Innovation in Language Learning and Teaching · doi:10.1080/17501229.2026.2734550
ECHO: Explainable Co-editing with Human-in-the-loop Operations for Presentation Slide Refinement
Fu, Y., Kang, Y., Zhou, Y., & Zhao, Y. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831120
Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems
Carrera, D., Thomas-Mitchell, J., & Wigdor, D. (2025) · arXiv · doi:10.48550/arxiv.2509.15440
Metadata
- Authors
- Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L.
- Affiliations
- MIT Media Lab (US)
- Harvard University (US)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-09-23
- Volume
- 30
- Issue
- 5
- Pages
- 1-57
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4213451626
- Cited by
- 119
- References
- 77
- Citations by year
- 2026: 15 · 2025: 32 · 2024: 38 · 2023: 23
- FWCI
- 12.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion, cc-by)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- NSF (2107391)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Artistic and Creative Research
- Keywords
- creative writing, language models, text generation, human-AI collaboration, story development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.