CoChat
Sign Up
Cited by 9Natural Language Processing TechniquesDesign Education and PracticeMulti-Agent Systems and Negotiation

CoCo Matrix: Taxonomy of Cognitive Contributions in Co-writing with Intelligent Agents

Open access
Wan, Ruyuan, Gebreegziabher, Simret Araya, Li, Toby Jia-Jun, Badillo-Urquiola, Karla·2024·Creativity and Cognition
DOI 10.1145/3635636.3664260
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, there has been a growing interest in employing intelligent agents in writing. Previous work emphasizes the evaluation of the quality of end product—whether it was coherent and polished, overlooking the journey that led to the product, which is an invaluable dimension of the creative process. To understand how to recognize human efforts in co-writing with intelligent writing systems, we adapt Flower and Hayes’ cognitive process theory of writing and propose CoCo Matrix, a two-dimensional taxonomy of entropy and information gain, to depict the new human-agent co-writing model. We define four quadrants and situate thirty-four published systems within the taxonomy. Our research found that low entropy and high information gain systems are under-explored, yet offer promising future directions in writing tasks that benefit from the agent’s divergent planning and the human’s focused translation. CoCo Matrix, not only categorizes different writing systems but also deepens our understanding of the cognitive processes in human-agent co-writing. By analyzing minimal changes in the writing process, CoCo Matrix serves as a proxy for the writer’s mental model, allowing writers to reflect on their contributions. This reflection is facilitated through the measured metrics of information gain and entropy, which provide insights irrespective of the writing system used.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3635636.3664260

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Initiative-whether a contribution was initiated by the human or AI partner-is a key process factor that impacts feelings of ownership.

    “One key process factor that impacts feelings of ownership is initiative: whether a contribution was initiated by the human or AI partner [71], and whether that contribution was made via “direct generation” or “indirect suggestions” [84].”

    From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [84]

    Section 4.3 distinguishes agent-initiated planning from writer-led contributions, supporting the distinction between who initiates a contribution. But neither that section nor the discussion of writer autonomy in Section 2.2 says that initiative affects feelings of ownership; that part of the attributed claim is unsupported. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Prior work has identified factors that affect people's sense of ownership over work produced with AI assistance, including types of contributions, amount of material contributed, process factors such as critical integration and control, and the role of AI in the writing process.

    “Some of these factors regard characteristics of the contributions themselves, such as the types of contributions made [33, 71, 84, 92] and the amount of material contributed [44].”

    From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [84]

    The Introduction and Section 4 classify types of human and AI contributions to co-writing, supporting the citation's reference to types of contributions. The passages do not establish that those types affect people's sense of ownership; the citation is not attached to the separate claim about the amount of material contributed. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. Abdulla Alsharhan. [n. d.]. Natural Language Generation and Creative Writing A Systematic Review. ([n. d.]).
  2. Robert W Andrews, J Mason Lilly, Divya Srivastava, and Karen M Feigh. 2023. The role of shared mental models in human-AI teams: a theoretical review. Theoretical Issues in Ergonomics Science 24, 2 (2023), 129-175.
  3. B Azhagusundari, Antony Selvadoss Thanamani, et al. 2013. Feature selection based on information gain. International Journal of Innovative Technology and Exploring Engineering (IJITEE) 2, 2 (2013), 18-21.
  4. Jianzhu Bao, Yasheng Wang, Yitong Li, Fei Mi, and Ruifeng Xu. 2022. AEG: Argu- mentative Essay Generation via A Dual-Decoder Model with Content Planning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 5134-5148. https://doi.org/10.18653/v1/2022.emnlp-main.343 · DOI
  5. Advait Bhat, Saaket Agashe, Parth Oberoi, Niharika Mohile, Ravi Jangir, and Anirudha Joshi. 2023. Interacting with next-phrase suggestions: How suggestion systems aid and influence the cognitive processes of writing. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 436-452.
  6. Alan S Brown. 1973. An empirical verification of Mednick's associative theory of creativity. Bulletin of the Psychonomic Society 2, 6 (1973), 429-430.
  7. Oğuz'Oz' Buruk. 2023. Academic writing with GPT-3.5 (ChatGPT): reflections on practices, efficacy and transparency. In Proceedings of the 26th International Academic Mindtrek Conference. 144-153.
  8. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The impact of multiple parallel phrase suggestions on email input and composition behaviour of native and non-native english writers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-13.
  9. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The impact of multiple parallel phrase suggestions on email input and composition behaviour of native and non-native english writers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-13.
  10. Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces. 329-340.
  11. Elizabeth Clark and Noah A Smith. 2021. Choose your own adventure: Paired suggestions in collaborative writing for evaluating story generation models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3566-3575.
  12. Hai Dang, Karim Benharrak, Florian Lehmann, and Daniel Buschek. 2022. Beyond text generation: Supporting writers with continuous automatic text summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1-13.
  13. Hai Dang, Karim Benharrak, Florian Lehmann, and Daniel Buschek. 2022. Beyond text generation: Supporting writers with continuous automatic text summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1-13.
  14. Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. 2019. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences 116, 39 (2019), 19251-19257.
  15. Linda Flower and John R Hayes. 1981. A cognitive process theory of writing. College composition and communication 32, 4 (1981), 365-387.
  16. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research 61 (2018), 65-170.
  17. Katy Gero, Alex Calderwood, Charlotte Li, and Lydia Chilton. 2022. A Design Space for Writing Support Tools Using a Cognitive Process Model of Writing. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), Ting-Hao 'Kenneth' Huang, Vipul Raheja, Dongyeop Kang, John Joon Young Chung, Daniel Gissin, Mina Lee, and Katy Ilonka Gero (Eds.). Association for Computational Linguistics, Dublin, Ireland, 11-24. https://doi. org/10.18653/v1/2022.in2writing-1.2 · DOI · Paper page
  18. Katy Ilonka Gero and Lydia B Chilton. 2019. How a Stylistic, Machine-Generated Thesaurus Impacts a Writer's Process. In Proceedings of the 2019 Conference on Creativity and Cognition. 597-603.
  19. Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Proceedings of the 2022 ACM Designing Interactive Systems Conference. 1002-1019.
  20. Maliheh Ghajargar, Jeffrey Bardzell, and Love Lagerkvist. 2022. A redhead walks into a bar: Experiences of writing fiction with artificial intelligence. In Proceedings of the 25th international academic MindTrek conference. 230-241.
  21. Mika Hämäläinen and Khalid Alnajjar. 2021. Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), Antoine Bosselut, Esin Durmus, Varun Prashant Gangal, Sebastian Gehrmann, Yacine Jernite, Laura Perez-Beltrachini, Samira Shaikh, and Wei Xu (Eds.). Association for Computational Linguistics, Online, 84-95. https: //doi.org/10.18653/v1/2021.gem-1.9 · DOI
  22. Dennis Hocevar. 1980. Intelligence, divergent thinking, and creativity. Intelligence 4, 1 (1980), 25-40.
  23. Chieh-Yang Huang, Shih-Hong Huang, and Ting-Hao Kenneth Huang. 2020. Heteroglossia: In-situ story ideation with the crowd. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-12.
  24. Karen Sparck Jones and Julia R Galliers. 1995. Evaluating natural language processing systems: An analysis and review. (1995).
  25. Jakob Karolus, Sebastian S Feger, Albrecht Schmidt, and Paweł W Woźniak. 2023. Your Text Is Hard to Read: Facilitating Readability Awareness to Support Writing Proficiency in Text Production. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 147-160.
  26. Hyung-Kwon Ko, Subin An, Gwanmo Park, Seung Kwon Kim, Daesik Kim, Bo- hyoung Kim, Jaemin Jo, and Jinwook Seo. 2022. We-toon: A Communication Support System between Writers and Artists in Collaborative Webtoon Sketch Revision. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1-14.
  27. Mina Lee, Katy Ilonka Gero, John Joon Young Chung, Simon Buckingham Shum, Vipul Raheja, Hua Shen, Subhashini Venugopalan, Thiemo Wambsganss, David Zhou, Emad A Alghamdi, et al. 2024. A Design Space for Intelligent and Interactive Writing Assistants. arXiv preprint arXiv:2403.14117 (2024). · arXiv
  28. Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human- ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1-19.
  29. Zhexiong Liu, Diane Litman, Elaine Wang, Lindsay Matsumura, and Richard Correnti. 2023. Predicting the Quality of Revisions in Argumentative Writing. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). Association for Computational Linguistics, Toronto, Canada, 275-287. https://doi.org/10.18653/v1/2023.bea-1.24 · DOI
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311-318.
  31. Ritika Poddar, Rashmi Sinha, Mor Naaman, and Maurice Jakesch. 2023. AI Writing Assistants Influence Topic Choice in Self-Presentation. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1-6.
  32. Melissa Roemmele and Andrew S Gordon. 2018. Automated assistance for creative writing with an rnn language model. In Proceedings of the 23rd international conference on intelligent user interfaces companion. 1-2.
  33. Claude Elwood Shannon. 2001. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review 5, 1 (2001), 3-55.
  34. Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. interactions 4, 6 (1997), 42-61.
  35. Sarah Sterman, Evey Huang, Vivian Liu, and Eric Paulos. 2020. Interacting with literary style through computational tools. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-12.
  36. Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems.
  37. Xaver Tomihiro Toyozaki and Keita Watanabe. 2018. Ambientletter: Letter presen- tation method for discreet notification of unknown spelling when handwriting. In Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 36-38.
  38. Thiemo Wambsganss, Andreas Janson, Tanja Käser, and Jan Marco Leimeister. 2022. Improving students argumentation learning with adaptive self-evaluation nudging. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1-31.
  39. Ruyuan Wan, Naome Etori, Karla Badillo-urquiola, and Dongyeop Kang. 2022. User or Labor: An Interaction Framework for Human-Machine Relationships in NLP. In Proceedings of the Fourth Workshop on Data Science with Human- in-the-Loop (Language Advances), Eduard Dragut, Yunyao Li, Lucian Popa, Slobodan Vucetic, and Shashank Srivastava (Eds.). Association for Computa- tional Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 112-121. https: //aclanthology.org/2022.dash-1.14
  40. Kento Watanabe, Yuichiroh Matsubayashi, Kentaro Inui, Tomoyasu Nakano, Satoru Fukayama, and Masataka Goto. 2017. Lyrisys: An interactive support system for writing lyrics based on topic transition. In Proceedings of the 22nd international conference on intelligent user interfaces. 559-563.
  41. Richard W Woodman and Lyle F Schoenfeldt. 1990. An interactionist model of creative behavior. The Journal of Creative Behavior (1990).
  42. Shaomei Wu, Lindsay Reynolds, Xian Li, and Francisco Guzmán. 2019. Design and evaluation of a social media writing support tool for people with dyslexia. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-14.
  43. Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. arXiv preprint arXiv:2304.07810 (2023). · arXiv
  44. Yubo Zhao and Xiying Bao. 2023. Narratron: Collaborative Writing and Shadow- playing of Children Stories with Large Language Models. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1-6.

Cited by

Sorted by citations

Authors
Wan, R., Gebreegziabher, S. A., Li, T. J.-J., & Badillo-Urquiola, K.
Affiliations
University of Notre Dame (US)
ORCID
Venue
Creativity and Cognition
Publisher
ACM
Published
2024-06-23
Pages
504-511
DOI
10.1145/3635636.3664260
OpenAlex
W4398230047
Cited by
9
References
11
Citations by year
2026: 2 · 2025: 7
FWCI
2.18
Type
conference paper
Language
en
Open access
Licence
Locations
  • Creativity and Cognition (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
entropy (information theory), information gain, creative process

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review