The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 26 of the 42 sources its claims cite were checked for support.
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
Open accessAbstract
Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs’ generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3’s capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3’s language, ideation, and collaboration capabilities, and reveal its contribution as a writing “collaborator” under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs’ promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
48 Own result · 24 Own interpretation · 12 Partly supported · 9 No basis given · 4 Sources not found · 3 No full text of the sources · 2 Supported · 1 Supported so far · 1 Not checked yet · 1 Own method · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Any LM's performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 1 INTRODUCTION
What the sources say
- Partly supportedcites [38] 38. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv:2107.13586 [cs.CL] sourceExperimental — accuracy not yet measured
Citation [38] is attached to the preceding-text part of the sentence, not decoding parameters, which are attributed to [27]. The survey’s “Prompt Engineering” section (p. 11) says prompts are designed to improve downstream performance, supporting the point that preceding text can matter. It does not establish that performance fluctuates significantly for any language model. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [27] 27. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1689-1701. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This matches with Dou et al. [20]'s fnding that sentences written by writers tend to have more grammar and usage errors compared to those written by GPT-3.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
What the sources say
- Partly supportedcites [20] 20. Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021. Scarecrow: A Framework for Scrutinizing Machine Text. arXiv preprint arXiv:2107.01294 (2021). sourceExperimental — accuracy not yet measured
Section E.6 (page not provided) reports more Grammar and Usage error annotations for human-authored text than for the best GPT-3 configuration tested. But the claim presents those annotations as a general finding about actual writing errors. The same section cautions that many human-text annotations were artifacts of the text format and estimates that human-authored paragraphs had far fewer legitimate issues overall. (LLM, conf=0.85)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, most previous work considered restricted interaction settings (e.g.strict turn-taking [15]) and adapted LMs and interfaces (e.g.fnd-tuned GPT-2 [58]) on specifc tasks (e.g.email writing [8]), thereby making it hard to generalize to other tasks and confgurations of the same model.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.3 Limitations of Traditional Methods.
What the sources say
- Supportedcites [15] 15. Elizabeth Clark and Noah A. Smith. 2021. Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 3566-3575. source
Section 2.1 (Writing Setup; page unavailable) describes a turn-taking process in which writers alternate between writing alone and receiving model suggestions. This supports the restricted-interaction example attached to this citation; the other examples in the sentence have separate citations. (LLM, conf=0.90)
- Supportedcites [58] 58. Lu Wang, Munif Ishad Mujib, Jake Williams, George Demiris, and Jina Huh-Yoo. 2021. An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers. arXiv:2107.13115 [cs.HC] source
The citation [58] is attached to the example of a fine-tuned GPT-2, rather than to the sentence’s broader assessment of previous log-analysis work. Section 3.1 (Dataset and Fine-tuning) says the authors applied GPT-2 and fine-tuned it for therapist–patient dialogue, so the paper is the example cited. (LLM, conf=0.90)
- Supportedcites [8] 8. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. Association for Computing Machinery, New York, NY, USA. source
The citation identifies this paper as an example of work on the specific task of email writing, not as evidence for the claim about most previous work. The Language Model section (p. 5) says its model was fine-tuned for an email use case, and the Limitations section (p. 16) describes the study’s email-composition tasks. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we show that the sentences generated by GPT-3 had less spelling and grammar errors than the sentences written by writers in CoAuthor.”
- Also stated in: § 5.1.1 Language Capabilities: Ability to Generate Fluent Text. · plain
“Figure 4 (a) shows that sentences written by writers had more spelling and grammar errors compared to GPT-3-generated sentences in fnal texts.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1 GPT-3's Generative Capabilities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § 5.1.1 Language Capabilities: Ability to Generate Fluent Text. · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the number of errors per word (averaged across all sentences in creative and argumentative writing) was 0.037 ± 0.001 for writers, 0.033 ± 0.001 for GPT-3, and 0.032 ± 0.001 for both (the number next to the average indicates the standard error of measurement).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoAuthor contains interactions between writers and LMs across multiple writing tasks, writing prompts, and writers.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The grammaticality of GPT-3-generated text varies across prompts (Figure 5).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Do prompts infuence GPT-3's grammaticality?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When we computed the distinct-2 score on all suggestions generated by GPT-3, the score was even lower (0.868 ± 0.001), further confrming the previous fndings.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We recruited 63 crowd workers (writers) from Amazon Mechanical Turk to account for individuals with potentially diferent backgrounds and writing styles.”
- Also stated in: § 4.3.1 Writers and prompts. · plain
“Among these writers, 63 writers participated in the main round, where 62 writers were native English speakers and one was not.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § 4.3.1 Writers and prompts. · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the distinct-2 score (averaged across all sentences in creative and argumentative writing) was 0.884 ± 0.001 for writers, 0.876 ± 0.001 for GPT-3, and 0.923 ± 0.001 for both.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 23
- The quote only partly states this claim. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Datasets are central to evaluating LMs' generative capabilities in NLP [12,25,28,32,49,55].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.4 Datasets in NLP
What the sources say
- Not matchedcites [12] 12. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of Text Generation: A Survey.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [25] 25. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg source
The Introduction (page not available in our copy) describes GEM datasets and challenging test sets as a way to assess model performance. The paper’s discussion of NLG benchmarks also treats dataset selection as integral to evaluation, supporting the broad claim. (LLM, conf=0.90)
- Not matchedcites [28] 28. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [32] 32. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Association for Computational Linguistics, Online, 4110-4124. sourceExperimental — accuracy not yet measured
The Background (p. 2) says task-level datasets have traditionally been used to measure progress in NLP, supporting the general point about datasets and evaluation. But Caveats and Objections (p. 8) says Dynabench currently focuses on classification and span extraction, while evaluating generative tasks requires further research. The paper therefore does not establish the claim specifically about evaluating language models’ generative capabilities. (LLM, conf=0.88)
- Not matchedcites [49] 49. Sebastian Ruder. 2021. Challenges and Opportunities in NLP Benchmarking. http://ruder.io/nlp-benchmarking
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [55] 55. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation. Association for Computational Linguistics, Tokyo, Japan, 355-368. sourceExperimental — accuracy not yet measured
Section 3.4 (Design; page unavailable) says the number of generated items matters for the statistical power of human evaluations. But our copy does not say that datasets are central to evaluating language models’ generative capabilities; searches for datasets, benchmarks, and test sets found no passage making that point. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Harnessing LMs' generative capabilities to power interaction designs begins with a holistic understanding of these capabilities [5,68]; this includes understanding what LMs can and cannot do under diverse interaction contexts.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 1 INTRODUCTION
What the sources say
- Partly supportedcites [5] 5. Sara Bly and Elizabeth F Churchill. 1999. Design through matchmaking: technol- ogy in search of users. interactions 6, 2 (1999), 23-31. sourceExperimental — accuracy not yet measured
The Summary (page unavailable) describes assessing a technology’s capabilities against work domains, and “Mapping Technology to Work Activities” (page unavailable) explicitly considers activities the technology supports easily and those it does not. The paper applies this approach to virtual environments, not language models or their generative capabilities, so the LM-specific application is an extension beyond the source. (LLM, conf=0.84)
- Partly supportedcites [68] 68. Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re- Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difcult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Comput- ing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376301 sourceExperimental — accuracy not yet measured
In “What Are AI’s Capabilities and Limits?” (page unavailable), the paper says designers need to understand a technology’s capabilities and limitations to see its design possibilities. “Capability Uncertainty” adds that users and use contexts affect AI capabilities. The paper makes these points about AI systems generally, including systems with open-ended outputs, but does not specifically establish them for language models’ generative capabilities. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Datasets can account for the subjective interpretation of capabilities and allow various interpretations.For example, Cuadra et al. [17] provided a video dataset capturing user interactions with a voice assistant without defning which interactions are good.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.3 Datasets in HCI
What the sources say
- Partly supportedcites [17] 17. Andrea Cuadra, Hansol Lee, Jason Cho, and Wendy Ju. 2021. Look at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors. arXiv preprint arXiv:2104.07153 (2021). sourceExperimental — accuracy not yet measured
Section 3.5 (p. 6) confirms that Cuadra et al. provided video clips of people interacting with a voice assistant, and Section 6.1 (p. 8) acknowledges subjectivity in classifying those interactions. But Section 3.4 (p. 6) says the authors labeled each clip using defined categories for successful responses, unsuccessful responses, omissions, and idle periods. The claim that they did so without defining which interactions were good is therefore not supported. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25,32,47,56].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 1 INTRODUCTION
What the sources say
- Supportedcites [25] 25. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg source
Section 2, “Benchmarks in NLG” (page not provided), says NLP benchmarks are common and enable researchers to compare models fairly; it calls them critical for measuring modeling progress. The paper’s discussion of selecting datasets for its own benchmark supports the citing sentence’s broad point about datasets being useful for model evaluation. (LLM, conf=0.88)
- Supportedcites [32] 32. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Association for Computational Linguistics, Online, 4110-4124. source
The Background (p. 2) says NLP progress has traditionally been measured using task-level datasets that became accepted benchmarks. The Introduction (p. 1) says benchmarks enable model comparisons and have supported advances in NLP, including the era of pre-trained language models. (LLM, conf=0.90)
- Supportedcites [47] 47. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 784-789. source
Section 3.2 describes earlier question-answering datasets used to test systems, and Section 5.2 evaluates reading-comprehension models on SQuAD and SQUADRUN. Together, these passages support the broad point that datasets have been useful for evaluating NLP models over time. (LLM, conf=0.86)
- Supportedcites [56] 56. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Brussels, Belgium, 353-355. source
The paper’s GLUE overview (page not provided) describes a benchmark built from existing datasets and an online platform for evaluating and comparing language-understanding models. Its Diagnostic Dataset section also describes using dataset examples to probe trained models. These passages support the broad claim that datasets are useful for evaluating models in NLP research. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Datasets can cover diverse contexts.For example, Theodorou et al. [54] published a dataset of photos taken by blind and low vision users, for assessing how well diferent object recognition models can serve this user population.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.3 Datasets in HCI
What the sources say
- Partly supportedcites [54] 54. Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morri- son, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021. Disability- First Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors. Association for Computing Machinery, New York, NY, USA. sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says the ORBIT project collected a dataset from blind and low vision people for developing teachable object recognisers. However, the Introduction and Conclusion specify that contributors supplied videos rather than photos; the cited passages also do not establish that the dataset was published to compare different recognition models. (LLM, conf=0.86)
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although this approach may provide less rich insights than interviews, it can cover relatively diverse contexts across tasks and writers, while allowing for a fne-grained analysis of interactions.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.3 Limitations of Traditional Methods.
What the sources say
- Supportedcites [8] 8. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. Association for Computing Machinery, New York, NY, USA. source
Study Design and Apparatus (p. 6) describes logging detailed interactions across three email scenarios. Participants and Procedure (p. 8) reports four tasks per person, and Writing Research (p. 4) identifies both native and non-native English writers and the detailed examination of interaction logs. Together, these support the claim about varied tasks and writers and fine-grained analysis. (LLM, conf=0.91)
- Partly supportedcites [48] 48. Melissa Roemmele and Andrew S. Gordon. 2018. Automated Assistance for Creative Writing with an RNN Language Model. In Proceedings of the 23rd In- ternational Conference on Intelligent User Interfaces Companion (Tokyo, Japan) (IUI '18 Companion). Association for Computing Machinery, New York, NY, USA, Article 21, 2 pages. sourceExperimental — accuracy not yet measured
The Experiment and Results section (page unavailable) reports 139 writers, 940 suggestion–modification pairs, and an edit-distance analysis, supporting fine-grained analysis across writers. Participants wrote stories on any topic, but all performed the same story-writing task; the paper does not establish coverage across different tasks or compare interaction logs with interviews. (LLM, conf=0.86)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Pre-trained on massive amounts of text on the Internet, LMs are known to produce linguistically fawed, factually incorrect, or even ethically problematic text at times [1,24].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.1 Language Models' Generative Capabilities.
What the sources say
- Partly supportedcites [1] 1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. arXiv:2101.05783 [cs.CL] sourceExperimental — accuracy not yet measured
The paper says language models are trained on large corpora including internet text (p. 2), and its Results (p. 5) document biased, violent GPT-3 completions. A figure caption (p. 7) also describes some generated descriptions as less coherent. Our copy does not establish the claim that the models produce factually incorrect text. (LLM, conf=0.88)
- Partly supportedcites [24] 24. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3356-3369. sourceExperimental — accuracy not yet measured
The Abstract and §3.1 report that pretrained language models sometimes generate toxic text, supporting the ethically problematic part of the claim. §8 (Related Work) also notes incoherence in model generations. The paper describes factually unreliable content in web-text corpora, but the passages do not establish that the models generate factually incorrect text. (LLM, conf=0.88)
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“How users adapt to systems over time is of interest for not only HCI researchers but also NLP researchers [59,60].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.2 Assess the Plausibility of Hypotheses.
What the sources say
- Partly supportedcites [59] 59. Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a Programming Language via Interactive Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 929-938. sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) studies how users’ language choices change as they interact with a natural-language system, supporting the NLP-research part of the claim. Our copy does not establish that this question is also of interest to HCI researchers. (LLM, conf=0.76)
- Supportedcites [60] 60. Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning Language Games through Interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 2368-2378. source
The Abstract identifies human adaptation to the computer as a central feature of the paper’s language-learning setting. The section “Most successful players (1st–20th)” reports how players changed their language over time, supporting this citation as an example of NLP research on user adaptation. Page numbers are not provided in our copy. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On an interaction level, good writing collaboration can exhibit diferent interaction patterns, such as diferent levels of mutuality and equality[2].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Partly supportedcites [2] 2. Zsuzsanna I Abrams. 2019. Collaborative Writing and Text Quality in Google Docs. Language Learning & Technology 23, 2 (2019), 22-42. sourceExperimental — accuracy not yet measured
The Review of the Literature describes interaction patterns using differing levels of equality and mutuality, and Research Question 1 reports several such patterns among the writing groups (page numbers unavailable). The paper does not establish that all those patterns constitute *good* collaboration; it associates greater collaboration with some better writing outcomes. (LLM, conf=0.82)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Partly supportedcites [6] 6. Joel Boch. 2007. Abdullah's blogging: A generation 1.5 student enters the blogo- sphere. Language Learning & Technology 11, 2 (2007), 128-141. sourceExperimental — accuracy not yet measured
The Introduction and Background Research (page unavailable) says that blogging can give students a greater sense of possible audiences and help them learn to respond to those audiences. The paper also describes interaction between readers and writers, but it does not establish that a human writing collaborator improves a text’s fluency. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [53] 53. Neomy Storch. 2005. Collaborative writing: Product, process, and students' refections. Journal of second language writing 14, 3 (2005), 153-173. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Not checked — no full text of the sourcecites [65] 65. Luxin Yang. 2014. Examining the mediational means in collaborative writing: Case studies of undergraduate ESL students in business courses. Journal of Second Language Writing 23 (2014), 74-89. source
- Supportedcites [70] 70. Soobin Yim and Mark Warschauer. 2017. Web-based collaborative writing in L2 contexts: Methodological insights from text mining. Language Learning & Technology 21, 1 (2017), 146-165. source
The Introduction links collaborative writing to socialization opportunities with specific discourse communities. This supports the attributed point about a human writing collaborator helping writers engage with those communities; the paper cites Yang (2014) for that point. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“HCI researchers can use CoAuthor to formulate hypotheses via replay enactment [29].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.1 Formulate Hypotheses.
What the sources say
- Partly supportedcites [29] 29. Kenneth Holstein, Erik Harpstead, Rebecca Gulotta, and Jodi Forlizzi. 2020. Replay Enactments: Exploring Possible Futures through Historical Data. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS '20). Association for Computing Machinery, New York, NY, USA, 1607-1618. https://doi.org/10.1145/3357236.3395427 sourceExperimental — accuracy not yet measured
The Introduction describes replay enactments as a way for designers to explore possible system behaviors, and the Conclusions say they help designers play out the implications of design decisions. Our copy does not mention CoAuthor or say that researchers use it to formulate hypotheses through this method (searched: CoAuthor, formulate hypotheses, hypothes). (LLM, conf=0.82)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoAuthor contains 20 writing prompts, brief passages of text that provides a potential topic idea or starting point.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor covers creative writing and argumentative writing, which have distinct goals and require diferent sets of writing skills.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we demonstrate that the amount of collaboration between writers and GPT-3 varied signifcantly across writers, but less depended on writing prompts and the randomness of suggestions.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1 GPT-3's Generative Capabilities
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, each writing session results in 418 words of text, contains 11.8 queries to the system, has acceptance rate of 72.3%, and 72.6% of text is written by writers.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.4 Dataset Overview
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“13% and 7% of accepted suggestions contained new named entities”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.2 Ideation Capabilities: Ability to Generate New Ideas.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We provide a tool for replaying all writing sessions in CoAuthor, giving designers a felt understanding of the interactions.”
- Also stated in: § Abstract · plain
“The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > I
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § Abstract · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We designed CoAuthor to be reusable and easily expandable in the future.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor supports multiple analytical perspectives and goals by providing various measurements in three categories that can be used to defne good collaboration: writing outcome, writer perception, and writer behavior.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor consists of a set of writing sessions.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains 830 stories written by 58 writers for creative writing and 615 essays written by 49 writers for argumentative writing.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.4 Dataset Overview
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor preserves details of rich interactions as events and event blocks.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrated the feasibility of this approach and discussed insights designers can draw from the dataset.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 7 CONCLUSION
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In argumentative writing, the sessions with GPT-3 with high randomness resulted in equality of 0.29 ± 0.01 and mutuality of 0.47 ± 0.01, whereas GPT-3 with low randomness resulted in 0.22 ± 0.01 and 0.38 ± 0.01 on average (Figure 9 (b)).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, some writers had more equal and mutual collaboration with GPT-3 with high randomness (dark blue), whereas others had such collaboration with GPT-3 with low randomness (light blue).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, each writing session is 418 words long, contains 11.8 queries to the system, has acceptance rate of 72.3% (how often writers accepted suggestions from GPT-3), and results in 72.6% of fnal texts written by writers (as opposed to GPT-3).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for instance, Writer #1 scored around 0.45 and 0.75 for equality and mutuality, whereas Writer #2 scored below 0.1 and 0.2”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Example of a story in which a writer accepted a suggestion from GPT-3 with a new named entity”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Equality and mutuality (y-axis) and vary across prompts (x-axis) in creative (a) and argumentative writing (b), but to the less extent than across writers.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When aggregated by prompts, 5 out of 10 prompts (50%) and 8 out of 10 prompts (80%) had higher equality and mutuality scores with GPT-3 with low randomness.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, in creative writing, writing sessions with GPT-3 with high randomness resulted in 0.30 ± 0.01 for equality and 0.48 ± 0.01 for mutuality, whereas the sessions with GPT-3 with low randomness resulted in 0.29 ± 0.01 and 0.49 ± 0.01, respectively (Figure 9 (a)).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in creative writing, some writers used the system to advance plots (62.9%), whereas some used it to add details to stories (51.4%) according to the survey's multiple choice question.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, it identifes a need for holistic understanding of LMs' generative capabilities for interaction design.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > I
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LMs' generative capabilities are highly context-dependent; therefore, it is difcult to cover all contexts.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.2 Challenges in Understanding Generative Capabilities.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LMs' generative capabilities can be subjectively interpreted; therefore, they are susceptible to varying evaluations even within a given interaction context.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.2 Challenges in Understanding Generative Capabilities.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we present evidence that GPT-3 is capable of providing new ideas to writers, infuencing their subsequent writing.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1 GPT-3's Generative Capabilities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
In the paper
“Appropriate interaction design for a new technology requires a deep understanding of its capabilities and limitations.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.1.1 Types of Understanding.
What the sources say
- Not matchedcites [9] 9. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
In the paper
“When we computed the distinct-2 score on all suggestions generated by GPT-3, the score was even lower (0.868 ± 0.001), further confrming the previous fndings.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
What the sources say
- Not checked — no full text of the sourcecites [27] 27. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1689-1701. source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
In the paper
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Not matchedcites [19] 19. Richard Donato. 1994. Collective scafolding in a second language. Vygotskian approaches to second language research (1994), 33-56.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, datasets in NLP can provide resources to help HCI researchers reason about LMs' generative capabilities in a holistic manner.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“HCI researchers can provide analytical tools for NLP researchers to better investigate LMs's capabilities interactive settings.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argued that curating and analyzing large interaction datasets is one viable approach, as it can cover diverse interaction contexts and support various interpretations of good collaboration.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 7 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that datasets have the potential to serve as boundary objects between the HCI and NLP communities.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, large interaction datasets in HCI can embed human-centered values and practices, and further incorporate them into technical advances, when used to train or evaluate LMs.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By having datasets as boundary objects, the HCI and NLP communities can easily communicate results and have shared understanding of LMs' generative capabilities.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Concretely, researchers can replay writing sessions in CoAuthor, which materializes the dynamics of interactions between writers and GPT-3 and makes complex system behavior more tangible.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.1 Formulate Hypotheses.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The authors of this paper manually rated 201 submissions with the goal of qualifying writers who demonstrated that they could meet two requirements.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > D QUALIFICATION ROUND
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoAuthor can serve as a starting point to assess the plausibility of hypotheses.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.2 Assess the Plausibility of Hypotheses.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
“Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing.”
“We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities”
The paper relies on this without support.
The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
“We recruited 63 crowd workers (writers) from Amazon Mechanical Turk to account for individuals with potentially diferent backgrounds and writing styles.”
“Among these writers, 63 writers participated in the main round, where 62 writers were native English speakers and one was not.”
The paper relies on this without support.
Most current language models are trained and evaluated on datasets that ignore interactive settings.
“Most current LMs are trained and evaluated on datasets that do not consider interactive settings.”
The paper relies on this without support.
High-randomness GPT-3 suggestions tend to be more diverse.
“as they tend to be more diverse.”
The paper relies on this without support.
The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
“On an interaction level, good writing collaboration can exhibit diferent interaction patterns, such as diferent levels of mutuality and equality[2].”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Systems commonly provided suggestions for revisions which could be integrated directly into the writing area [14,18,75], aiding efficiency and idea exploration in the text.”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [75]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies of interactive writing systems note that the point of contact between the writer and the model (when suggestions appear, what the writer must do to take them, etc.) shapes how much of the writing remains the writer's own (Lee et al., 2022).”
From Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions? (2026) · cites this paper as Lee et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast, prior research has found that people’s perceptions of AI authorship in co-creation are nuanced. These perceptions depend on various dimensions, including the type of contribution made by the AI, how much each party contributed, and who led the co-creative effort [33, 44, 71, 92].”
From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [44]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing (Shuster et al., 2022;Thoppilan et al., 2022;Lee et al., 2022, inter alia).”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Lee et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“More recent work on AI co-writing for creative and argumentative texts has explicitly investigated issues of agency and ownership, finding that the less a user writes, the less ownership they feel over the final text [36].”
From Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems (2026) · cites this paper as [36]
The cited source says what the claim attributes to it.
In the citing paper
“GenAI has signi cantly impacted various aspects of daily life, including enhancing productivity [68], writing [57, 58], interpersonal communication [35], and relationships [34, 76].”
From Creativity in the Age of AI: Evaluating the Impact of Generative AI on Design Outputs and Designers' Creative Thinking (2024) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“In particular, a writer's sense of ownership may diminish as the proportion of system-generated content increases (Lee et al., 2022).”
From Ink and algorithm: Exploring temporal dynamics in generative AI ‐assisted writing (2026) · cites this paper as Lee et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“More commonly, systems display suggestions in a separated interface [29,43,75,89].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [75]
The cited source says what the claim attributes to it.
In the citing paper
“Some of these factors regard characteristics of the contributions themselves, such as the types of contributions made [33, 71, 84, 92] and the amount of material contributed [44].”
From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [44]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Furthermore, students may be less inclined to disclose GenAI use, given widespread awareness of the difficulty humans face in accurately detecting AI-generated text 33,34,35 and the unreliability of current detection tools 36,37 .”
From Style, sentiment, and quality of undergraduate writing in the AI era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredIn the citing paper
“These interfaces are highly intuitive, and mimic conversational interactions between humans, but lack scaffolding for crafting prompts, which can be difficult for novice AI users [49,98].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [49]
In the citing paper
“While these approaches provide an objective representation of writer-suggestion interactions at a behavioural level, they lack insight into the inner cognitive decision-making processes that writers might engage in.”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [45]
In the citing paper
“Lee et al. [54] find that LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.”
From Homogenization Effects of Large Language Models on Human Creative Ideation (2024) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
“| | | Combined | 20 | | 63 | | | 1445 | 11.2 | 366 | 11.8 | 72.3 | 72.6 | |”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.4 Dataset Overview
“On average, each writing session results in 418 words of text”
Citations
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 2×
“The goal of Natural Language Generation (NLG) is to produce fluent text in many domains, such as machine translation [21], summarization [34], dialogue [30], style transfer [18], and programming code [13].”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“Any LM’s performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1×
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25, 32, 47, 56].”
Background5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Self-citationGaver, W. W. (1991) · Proceedings of the SIGCHI conference on Human factors in computing systems Reaching through technology - CHI '91 · doi:10.1145/108844.108856 · Cited by 1,536
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Patterns of Interaction in ESL Pair Work
Storch, N. (2002) · Language Learning · doi:10.1111/1467-9922.00179 · Cited by 1,080 · cited 2×
“• Writer interaction: We measured how much interaction writers had with GPT-3 in the writing process using the notions of equality and mutuality from Storch [52].”
Method
Research works printed but not linked: 8
- Joel Boch . 2007 . Abdullah’s blogging: A generation 1.5 student enters the blogosphere . Language Learning & Technology 11 , 2 (2007), 128 – 141 . Joel Boch. 2007. Abdullah’s blogging: A generation 1.5 student enters the blogosphere. Language Learning & Technology 11, 2 (2007), 128–141.
- Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates , Inc ., 1877 –1901. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901.
- Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann.
- Alex Calderwood , Vivian Qiu , Katy Ilonka Gero , and Lydia B Chilton . 2020. How Novelists Use Generative Language Models: An Exploratory User Study . In Proceedings of the Workshops on Human-AI Co-Creation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces . Association for Computing Machinery . Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Proceedings of the Workshops on Human-AI Co-Creation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces. Association for Computing Machinery.
- Richard Donato . 1994. Collective scaffolding in a second language. Vygotskian approaches to second language research ( 1994 ), 33–56. Richard Donato. 1994. Collective scaffolding in a second language. Vygotskian approaches to second language research (1994), 33–56.
- Rebecca Fiebrink and Perry R Cook . 2010 . The Wekinator: a system for real-time, interactive machine learning in music . In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010) (Utrecht). Rebecca Fiebrink and Perry R Cook. 2010. The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)(Utrecht).
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog 1, 8 ( 2019 ). Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog 1, 8 (2019).
- Donald A Schön. 1984. The reflective practitioner: How professionals think in action. Vol. 5126. Basic books. Donald A Schön. 1984. The reflective practitioner: How professionals think in action. Vol. 5126. Basic books.
Cited by
Sorted by citations
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Self-citationPerkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839
“…AI based technologies, there is ongoing research exploring how the intricacies of co-creation may be further be codified and explored in the fields of creative writing (M. Lee et al., 2022; Y. Lee et al., 2022), software development (Biderman & Raff, 2022) and artistic outputs (Oppenlaender, 2022).”
Alqahtani, T., Badreldin, H. A., Alrashed, M., Alshaya, A. I., Alghamdi, S. S., bin Saleh, K., Alowais, S. A., Alshaya, O. A., Rahman, I., Al Yami, M. S., et al. (2023) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2023.05.016 · Cited by 565
Marzuki, Widiati, U., Rusdin, D., Darwin, & Indrawati, I. (2023) · Cogent Education · doi:10.1080/2331186x.2023.2236469 · Cited by 524
“Works by Nazari et al. (2021), Dale and Viethen (2021), M. Lee et al. (2022), and Zhao (2022) served as valuable references.”
BackgroundSharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500
Impact of Artificial Intelligence on Dental Education: A Review and Guide for Curriculum Update
Thurzo, A., Strunga, M., Urban, R., Surovková, J., & Afrashtehfar, K. I. (2023) · Education Sciences · doi:10.3390/educsci13020150 · Cited by 345
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240
“Some studies consulted with expert linguists [31], university students in the sciences [42] or humanities [123], and amateur writers [2, 21, 62, 100, 125].”
BackgroundKar, A. K., Varsha, P. S., & Rajan, S. (2023) · Global Journal of Flexible Systems Management · doi:10.1007/s40171-023-00356-x · Cited by 229
“…technologies bring flexibility in firms to tackle the co-creation complexities which can be codified and located in the domains of creative writing (M Lee et al., 2022a, 2022b; Y Lee et al., 2022a, 2022b), information technology (Bider-man & Raff, 2022) and artistic outputs (Oppenlaender, 2022).”
BackgroundFrom explainable to interactive AI: A literature review on current trends in human-AI interaction
Raees, M., Meijerink, I., Lykourentzou, I., Khan, V.-J., & Papangelis, K. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103301 · Cited by 180
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
Draxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3637875 · Cited by 186
“The levels of control are derived from [29] and [70].”
BackgroundMethodInfluentialHomogenization Effects of Large Language Models on Human Creative Ideation
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656204 · Cited by 188
Jo, E., Epstein, D. A., Jung, H., & Kim, Y.-H. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581503 · Cited by 174
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 206
“Coauthor [50] provides a dataset of rich interactions between GPT-3 and writers.”
BackgroundEnabling Conversational Interaction with Mobile UI using Large Language Models
Wang, B., Li, G., & Li, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580895 · Cited by 156
“[25] present CoAuthor, a dataset designed to reveal GPT-3’s capabilities in assisting creative and argumentative writing.”
BackgroundMcGuire, J., De Cremer, D., & Van de Cruys, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-69423-2 · Cited by 150
Oniani, D., Hilsman, J., Peng, Y., Poropatich, R. K., Pamplin, J. C., Legault, G. L., & Wang, Y. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00965-x · Cited by 143
Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671573 · Cited by 142
“…advances in large language models (LLMs) [5, 11, 44, 55, 60, 67, 68] have enabled a wide range of real-world applications across various domains, such as chatbots [43], writing assistants [24, 31], search engines (e.g., New Bing), tool/API retriever [46, 49] and multimodal systems [19, 23, 63].”
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136
“Outputs that are coherent in terms of grammar, content, and tone”
Self-citationArt or Artifice? Large Language Models and the False Promise of Creativity
Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., & Wu, C.-S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642731 · Cited by 135
“In prior work, Baer [5] argued how divergent thinking remains the most frequently used indicator of creativity in both creativity research and educational practice, and divergent thinking theory has a strong hold on everyday conceptions of what it means to be creative.”
BackgroundSocial Dynamics of AI Support in Creative Writing
Gero, K. I., Long, T., & Chilton, L. B. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580782 · Cited by 129
“[22] looks at interactions with a GPT-3 autocomplete system across 63 writers.”
BackgroundInfluentialGenerative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking
Sharma, N., Liao, Q. V., & Xiao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642459 · Cited by 132
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 178
“, 2021) and show potential for creative writing tasks (Lee et al., 2022).”
BackgroundZhang, Z., Gao, J., Dhaliwal, R. S., & Li, T. J.-J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606800 · Cited by 108
“Similarly, CoAuthor [33] provides writers with context-based argumentative suggestions, inspiring them with the next piece of content to write.”
BackgroundMethodInfluentialOn the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113
“Indeed, we believe that LLMs can also foster human-AI co-creativity (Lee et al. 2022), since they can be used to write portions of stories to serve specific purposes, e.g., they can typify all the dialogues from a character, or they can provide more detailed descriptions of scenes (Calderwood et…”
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Cheng, M., Durmus, E., & Jurafsky, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.84 · Cited by 235
Han, A., Zhou, X., Cai, Z., Han, S., Ko, R., Corrigan, S., & Peppler, K. A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642438 · Cited by 114
Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models
Dhillon, P. S., Molaei, S., Li, J., Golub, M., Zheng, S., & Robert, L. P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642134 · Cited by 111
“Writing Prompts: We utilized 10 argumentative writing prompts originally selected by the CoAuthor study [41] from a set provided by The New York Times [48].”
BackgroundMethodResultInfluentialAI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
Agarwal, D., Naaman, M., & Vashistha, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11360 · Cited by 99
AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
Agarwal, D., Naaman, M., & Vashistha, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713564 · Cited by 214
“Beyond creative writing, HCI researchers have also proposed tools for argumentative writing, including jour-nalism [65], scientific writing [30], email writing [46], and crafting self-introductions for networking [41].”
Investigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87
“Given that trust is deeply embedded in contextual factors, the factors identified in prior empirical studies might change for AI-powered code-generation tools as system properties and the social and organizational contexts of usage shift.”
BackgroundBeyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries
Dang, H., Benharrak, K., Lehmann, F., & Buschek, D. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545672 · Cited by 103
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637364 · Cited by 84
Park, H., & Ahn, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642785 · Cited by 84
“Accordingly, recent HCI research has expanded LLMs’ application to tasks such as co-writing with LLMs [15, 32, 49] and providing health interventions for socially isolated or marginalized individuals [37, 90].”
BackgroundA systematic review of artificial intelligence technologies used for story writing
Fang, X., Ng, D. T. K., Leung, J. K. L., & Chu, S. K. W. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11741-5 · Cited by 82
“Similarly, Lee et al. (2022) mentioned that although AI was incorporated into story generators, they still have multiple aspects that need to be improved, such as language technicalities and story suggestions.”
BackgroundMethodInfluentialUnderstanding EFL students’ chatbot-assisted argumentative writing: An activity theory perspective
Guo, K., Li, Y., Li, Y., & Chu, S. K. W. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12230-5 · Cited by 118
“First, due to its human-like design features and interactivity, Argumate was perceived by the student users as a writing partner (Lee et al., 2022; Yuan et al., 2022) that provided support rather than as a mere tool that they used.”
Dang, H., Goller, S., Lehmann, F., & Buschek, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580969 · Cited by 83
“Single text suggestions can be shown inline [4, 8, 17, 48], whereas multiple suggestions are shown as popup lists of about three to six entries [6, 26].”
BackgroundResultInfluentialLiao, Q. V., Subramonyam, H., Wang, J., & Vaughan, J. W. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 120
“Similarly, to enable exploring GPT-3’s promises for interaction design, Lee et al. [65] created a dataset of instances from writers working with GPT-3.”
BackgroundQin, H. X., Jin, S., Gao, Z., Fan, M., & Hui, P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642105 · Cited by 82
The Value, Benefits, and Concerns of Generative AI-Powered Assistance in Writing
Li, Z., Liang, C., Peng, J., & Yin, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642625 · Cited by 81
“This assistance could also tailor to diverse writing tasks including stories [12, 13, 29, 41, 43, 45, 46], slogans [13], argumentative essays [29], emails [9, 19], and script writing [33].”
BackgroundCummings, R. E., Monroe, S. M., & Watkins, M. (2024) · Computers and Composition · doi:10.1016/j.compcom.2024.102827 · Cited by 83
Instruction Tuning for Large Language Models: A Survey
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wang, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3777411 · Cited by 1,271
“As depicted in Figure 1, it involves further training LLMs using (instruction, output) pairs, where instruction denotes the human instruction for the model, and output denotes the desired output that follows the instruction.”
Kim, T., Shin, D., Kim, Y.-H., & Hong, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642693 · Cited by 75
“While LLMs have been shown to enhance general writing abilities—by promoting varied vocabulary use and clarifying ambiguous ideas [10, 17, 29]—their application in the realm of personal journaling remains under-explored.”
InfluentialGenerative Image AI Using Design Sketches as input: Opportunities and Challenges
Zhang, C., Wang, W., Pangaro, P., Martelaro, N., & Byrne, D. (2023) · Creativity and Cognition · doi:10.1145/3591196.3596820 · Cited by 95
“Emerging research has demonstrated the potential of generative AI models in providing creative writing suggestions [5] as well as in generating advertisement posters [3], illustrative images for news [7], and graphical user interfaces[16].”
BackgroundBuilding machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 125
Self-citationHerding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3
Zamfirescu-Pereira, J. D., Wei, H., Xiao, A., Gu, K., Jung, G., Lee, M. G., Hartmann, B., & Yang, Q. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596138 · Cited by 88
“Our dialogue annotation and error analyses included: • Basic text-generation quality metrics, such as factualness [15]; • The extent to which the best-performing prompt can enable a GPT bot to exhibit the four interaction qualities listed in §3.1.2; • The severity of the interaction breakdowns,…”
MethodInfluentialSelf-citationThe Idea Machine: LLM-based Expansion, Rewriting, Combination, and Suggestion of Ideas
Fede, G. D., Rocchesso, D., Dow, S. P., & Andolina, S. (2022) · Creativity and Cognition · doi:10.1145/3527927.3535197 · Cited by 68
“With its design grounded in creativity and Human-Centered AI literature [15], the Idea Machine will enable the collection of valuable interaction data in LLM-based augmented ideation, similar to what has been done in other domains [11].”
ResultWriter-Defined AI Personas for On-Demand Feedback Generation
Benharrak, K., Zindulka, T., Lehmann, F., Heuer, H., & Buschek, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642406 · Cited by 63
“More recently, systems such asWordcraft [59] or CoAuthor [39] provide co-writing experiences with AI.”
BackgroundKim, J., Suh, S., Chilton, L. B., & Xia, H. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595996 · Cited by 61
“To evaluate mapping, fve criteria were used: fuency [11], scientifc precision, relatedness [27], willingness to adopt [45], and inspirational efect [24, 57].”
BackgroundMethodChakrabarty, T., Padmakumar, V., Brahman, F., & Muresan, S. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656201 · Cited by 61
Liu, Y., Chen, S., Cheng, H., Yu, M., Ran, X., Mo, A., Tang, Y., & Huang, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642698 · Cited by 61
“Previous HCI studies suggest that while users might sometimes view unexpected outputs favorably [15], they can also find them unhelpful at times [35].”
Evolving Roles and Workflows of Creative Practitioners in the Age of Generative AI
Palani, S., & Ramos, G. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656190 · Cited by 84
Kobiella, C., Flores López, Y. S., Waltenberger, F., Draxler, F., & Schmidt, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641964 · Cited by 57
“GenAI tools have been used in the hope of bolstering the productivity of knowledge workers [9, 11, 26] and providing valuable support in problem-solving, idea generation, and conceptual development [20, 23, 53], key facets of knowledge-intensive careers.”
MethodLiu, J. Q. J., Hui, K. T. K., Al Zoubi, F., Zhou, Z. Z. X., Samartzis, D., Yu, C. C. H., Chang, J. R., & Wong, A. Y. L. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00155-6 · Cited by 56
A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
Gao, J., Gebreegziabher, S. A., Choo, K. T. W., Li, T. J.-J., Perrault, S. T., & Malone, T. W. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650786 · Cited by 57
“One example is the OpenAI Playground; similarly, literature such as CoAuthor leverages LLMs to ofer writing suggestions based on the user’s input text [40].”
A Redhead Walks into a Bar: Experiences of Writing Fiction with Artificial Intelligence
Ghajargar, M., Bardzell, J., & Lagerkvist, L. (2022) · Proceedings of the 25th International Academic Mindtrek Conference · doi:10.1145/3569219.3569418 · Cited by 54
“CoAuthor is a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities, which assists humans in writing creative and argumentative pieces [47].”
BackgroundReza, M., Laundry, N. M., Musabirov, I., Dushniku, P., Yu, Z. Y. “., Mittal, K., Grossman, T., Liut, M., Kuzminykh, A., & Williams, J. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641899 · Cited by 52
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
InfluentialPang, R. Y., Schroeder, H., Smith, K. S., Barocas, S., Xiao, Z., Tseng, E., & Bragg, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713726 · Cited by 112
Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
Kumar, H., Vincentius, J., Jordan, E., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714198 · Cited by 129
“Lee et al. [37] introduced a dataset for analyzing GPT-3’s use in creative and argumentative writing, suggesting that the HCI community could foster more detailed examinations of LLMs’ generative capabilities through such datasets.”
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Wang, R., Ribeiro, A., Robinson, C., Loeb, S., & Demszky, D. (2024) · doi:10.21203/rs.3.rs-5363154/v1 · Cited by 53
Evaluating language models for mathematics through interactions
Collins, K. M., Jiang, A. Q., Frieder, S., Wong, L., Zilka, M., Bhatt, U., Lukasiewicz, T., Wu, Y., Tenenbaum, J. B., Hart, W., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2318124121 · Cited by 108
Hoque, M. N., Mashiat, T., Ghai, B., Shelton, C. D., Chevalier, F., Kraus, K., & Elmqvist, N. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641895 · Cited by 48
“Visualization has long been used to provide explainable machine learning [73] and NLP models [11, 45], and these ideas have also recently begun to be applied to LLMs. Recently, Jiang et al. [39] proposed Graphologue, that converts responses from LLMs to interactive graphs for fast and non-linear…”
BackgroundImran, M., Almusharraf, N., Abdellatif, M. S., & Abbasova, M. Y. (2024) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v18i18.49143 · Cited by 51
The Silicon Ceiling: Auditing GPT’s Race and Gender Biases in Hiring
Armstrong, L., Liu, A., MacNeil, S., & Metaxa, D. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694699 · Cited by 78
“Their promise of increased efficiency has led to numerous new LLM-powered systems for tasks including writing [38, 52, 63, 94], software engineering [28, 65], and design [30, 32, 43].”
Exploring EFL students’ prompt engineering in human–AI story writing: an activity theory perspective
Woo, D. J., Guo, K., & Susanto, H. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2361381 · Cited by 67
“Students with this theme most clearly high-light the potential of human – AI story writing to re fi ne and enhance story development (Lee et al., 2022).”
Bhat, A., Agashe, S., Oberoi, P., Mohile, N., Jangir, R., & Joshi, A. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584060 · Cited by 47
“It can act as an analytical lens for interpreting qualitative and quantitative data, generating hypotheses, and providing vocabulary to articulate complex interactions observed between writers and suggestion systems (For example, interaction logs datasets from tools like CoAuthor [45]).”
BackgroundMethodWoo, D. J., Wang, Y., Susanto, H., & Guo, K. (2023) · Journal of Educational Computing Research · doi:10.1177/07356331231175999 · Cited by 44
“, 2018) and that might increase writer productivity (Lee et al., 2022).”
BackgroundWorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI
Dang, H., Brudy, F., Fitzmaurice, G. W., & Anderson, F. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606772 · Cited by 62
“Moreover, several works have suggested to also logging users’ interactions [16, 17, 42, 48] to evaluate their behavior with co-creative AI systems.”
BackgroundHelp me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing
Chakrabarty, T., Padmakumar, V., & He, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.460 · Cited by 86
Feng, K. J. K., Li, T. W., & Zhang, A. X. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581273 · Cited by 44
FigurA11y: AI Assistance for Writing Scientific Alt Text
Singh, N., Wang, L. L., & Bragg, J. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645212 · Cited by 42
“Large language models (LLMs) have recently shown promise in providing contextual suggestions for diverse writing and revision tasks [8, 10, 23].”
BackgroundBhattacharjee, A., Zeng, Y., Xu, S. Y., Kulzhabayeva, D., Ma, M., Kornfield, R., Ahmed, S. I., Mariakakis, A. T., Czerwinski, M. P., Kuzminykh, A., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642081 · Cited by 59
“In writing-intensive courses, LLMs might function as intelligent writing assistants [33, 58, 67, 117]; they can provide more than mere copy-editing, but also real-time support in enhancing the structure and flow of arguments.”
BackgroundWe’re Afraid Language Models Aren’t Modeling Ambiguity
Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.51 · Cited by 113
Moon, K., Green, A. E., & Kushlev, K. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100207 · Cited by 38
Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Zhang, S., Xu, J., & Alvero, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251327130 · Cited by 68
RELIC: Investigating Large Language Model Responses using Self-Consistency
Cheng, F., Zouhar, V., Arora, S., Sachan, M., Strobelt, H., & El‐Assady, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641904 · Cited by 55
“Recent studies in human-LLM interaction have investigated various ways humans can collaborate with LLMs in accomplishing downstream tasks, like text summarization [5], creative writing [17, 23, 53, 54], and information seeking [20, 42].”
Background'It was 80% me, 20% AI': Seeking Authenticity in Co-Writing with Large Language Models
Hwang, A. H.-C., Liao, Q. V., Blodgett, S. L., Olteanu, A., & Trischler, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711020 · Cited by 42
Chen, L., Zuo, H., Cai, Z., Yin, Y., Zhang, Y. E., Sun, L., Childs, P., & Wang, B. (2024) · Journal of Mechanical Design · doi:10.1115/1.4065562 · Cited by 41
PANDALens: Towards AI-Assisted In-Context Writing on OHMD During Travels
Cai, R., Janaka, N., Chen, Y., Wang, L., Zhao, S., & Liu, C. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642320 · Cited by 36
“For subjective measures, self-rated Post-Editing Effort , quality of Language , Creativity & Appeal , and meeting preferred Writing Style for final writing (7-point Likert scale) [49] were collected.”
MetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Sun, L., Tao, S., Hu, J., & Dow, S. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637371 · Cited by 37
“CoAuthor further explored the capabilities of LLM working as a writing collaborator and reasoned about its ability to generate fluent text, generate new ideas and work jointly with writers [50].”
BackgroundFatima, W., Rizvi, S. S. R., Ghazal, T. M., Kharma, Q. M., Ahmad, M., Abbas, S., Furqan, M., & Khan, M. A. (2025) · IEEE Access · doi:10.1109/access.2025.3575610 · Cited by 35
“It preserves the concept and context of the original content, but may exclude nuances [16].”
Fan, H., Chen, G., Wang, X., & Peng, Z. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676390 · Cited by 60
Untangling Critical Interaction with AI in Students’ Written Assessment
Shibani, A., Knight, S., Kitto, K., Karunanayake, A., & Buckingham Shum, S. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3651083 · Cited by 35
Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation
He, J., Houde, S., & Weisz, J. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713522 · Cited by 36
“Some of these factors regard characteristics of the contributions themselves, such as the types of contributions made [33, 71, 84, 92] and the amount of material contributed [44].”
InfluentialKnowNet:Guided Health Information Seeking from LLMs via Knowledge Graph Integration
Yan, Y., Hou, Y., Xiao, Y., Zhang, R., & Wang, Q. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456364 · Cited by 47
“Four authors, including two domain and two visualization experts, first collaboratively built a list of relevant papers that examined, applied, or improved LLMs for various information-seeking tasks, including but not limited to writing [22,57], coding [2,55], and healthcare [53].”
BackgroundAshkinaze, J., Mendelsohn, J., Qiwei, L., Budak, C., & Gilbert, E. (2025) · Proceedings of the ACM Collective Intelligence Conference · doi:10.1145/3715928.3737481 · Cited by 33
Guo, A., Pataranutaporn, P., & Maes, P. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650892 · Cited by 32
“Work has been done examining language models’ capabilities in creative and argumentative writing show GPT-3’s abilities and consider metrics for good collaboration including writ-ers’ feeling of ownership [20].”
Student-AI Question Cocreation for Enhancing Reading Comprehension
Liu, M., Zhang, J., Nyagoga, L. M., & Liu, L. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3333439 · Cited by 40
“Inspired by the recent success of human-AI cocreation in writing and music [23], [48], an automatic question generator was used to provide suggestions for students to reflect and create questions efficiently.”
MethodBeyond the Chat: Executable and Verifiable Text-Editing with LLMs
Laban, P., Vig, J., Hearst, M., Xiong, C., & Wu, C.-S. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676419 · Cited by 31
“Some work has framed text editing as an iterative process [22] and collected datasets of multi-turn interaction such as IteraTER [23] or Coauthor [57].”
BackgroundMethodFrom Pen to Prompt: How Creative Writers Integrate AI into their Writing Practice
Guo, A., Sathyanarayanan, S., Wang, L., Heer, J., & Zhang, A. X. (2025) · Creativity and Cognition · doi:10.1145/3698061.3726910 · Cited by 61
Song, I., Park, S., Pendse, S. R., Schleider, J. L., Choudhury, M. D., & Kim, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713883 · Cited by 64
“At can request an AI-generated summary of the current exploration e ○ .”
Multimodal large language models for inclusive collaboration learning tasks
Lewis, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop · doi:10.18653/v1/2022.naacl-srw.26 · Cited by 30
“Recent NLP research, for example, has explored neural models equipped with dynamic knowledge graph embeddings, the use of large language models to model real world speech, and the development of collaboration datasets (Ekstedt and Skantze, 2020; He et al., 2017; Lee et al., 2022), while cognitive science has explored…”
BackgroundPatchview: LLM-powered Worldbuilding with Generative Dust and Magnet Visualization
Chung, J. J. Y., & Kreminski, M. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676352 · Cited by 49
Yeo, S., Lim, G., Gao, J., Zhang, W., & Perrault, S. T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642530 · Cited by 43
“Research on LLM and Natural Language Processing (NLP) have shed light on their role as active partners throughout the writing process [85, 136].”
Background(2024) · doi:10.1145/3613904.3642939 · Cited by 31
Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process
Reza, M., Thomas-Mitchell, J., Dushniku, P., Laundry, N., Williams, J. J., & Kuzminykh, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757566 · Cited by 37
“AI also provides support with rhetorical or stylistic elements such as forming metaphors [37] or learning vocabulary [95].”
InfluentialArakawa, R., Yakura, H., & Goto, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581133 · Cited by 29
“recent studies that demonstrate how human writers and AI-based generation techniques can collaborate to support writers [23, 35, 60], such as in creative writing [57, 63, 65, 78, 82].”
BackgroundComparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
Fu, L., Newman, B., Jakesch, M., & Kreps, S. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581351 · Cited by 31
Metadata
- Authors
- Lee, M., Liang, P., & Yang, Q.
- Affiliations
- Stanford University (US)
- Cornell University (US)
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-19
- OpenAlex
- W4220747294
- Cited by
- 345
- References
- 39
- Citations by year
- 2026: 61 · 2025: 96 · 2024: 118 · 2023: 56 · 2022: 14
- FWCI
- 56.81
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI collaboration, large language models, GPT-3, writing support, creative writing, argumentative writing, human-AI interaction, idea generation, language generation, interaction design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
