The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 26 of the 42 sources its claims cite were checked for support.
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
Open accessAbstract
Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs’ generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3’s capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3’s language, ideation, and collaboration capabilities, and reveal its contribution as a writing “collaborator” under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs’ promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
48 Own result · 24 Own interpretation · 12 Partly supported · 9 No basis given · 4 Sources not found · 3 No full text of the sources · 2 Supported · 1 Supported so far · 1 Not checked yet · 1 Own method · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Any LM's performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 1 INTRODUCTION
What the sources say
- Partly supportedcites [38] 38. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv:2107.13586 [cs.CL] sourceExperimental — accuracy not yet measured
Citation [38] is attached to the preceding-text part of the sentence, not decoding parameters, which are attributed to [27]. The survey’s “Prompt Engineering” section (p. 11) says prompts are designed to improve downstream performance, supporting the point that preceding text can matter. It does not establish that performance fluctuates significantly for any language model. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [27] 27. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1689-1701. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This matches with Dou et al. [20]'s fnding that sentences written by writers tend to have more grammar and usage errors compared to those written by GPT-3.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
What the sources say
- Partly supportedcites [20] 20. Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021. Scarecrow: A Framework for Scrutinizing Machine Text. arXiv preprint arXiv:2107.01294 (2021). sourceExperimental — accuracy not yet measured
Section E.6 (page not provided) reports more Grammar and Usage error annotations for human-authored text than for the best GPT-3 configuration tested. But the claim presents those annotations as a general finding about actual writing errors. The same section cautions that many human-text annotations were artifacts of the text format and estimates that human-authored paragraphs had far fewer legitimate issues overall. (LLM, conf=0.85)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, most previous work considered restricted interaction settings (e.g.strict turn-taking [15]) and adapted LMs and interfaces (e.g.fnd-tuned GPT-2 [58]) on specifc tasks (e.g.email writing [8]), thereby making it hard to generalize to other tasks and confgurations of the same model.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.3 Limitations of Traditional Methods.
What the sources say
- Supportedcites [15] 15. Elizabeth Clark and Noah A. Smith. 2021. Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 3566-3575. source
Section 2.1 (Writing Setup; page unavailable) describes a turn-taking process in which writers alternate between writing alone and receiving model suggestions. This supports the restricted-interaction example attached to this citation; the other examples in the sentence have separate citations. (LLM, conf=0.90)
- Supportedcites [58] 58. Lu Wang, Munif Ishad Mujib, Jake Williams, George Demiris, and Jina Huh-Yoo. 2021. An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers. arXiv:2107.13115 [cs.HC] source
The citation [58] is attached to the example of a fine-tuned GPT-2, rather than to the sentence’s broader assessment of previous log-analysis work. Section 3.1 (Dataset and Fine-tuning) says the authors applied GPT-2 and fine-tuned it for therapist–patient dialogue, so the paper is the example cited. (LLM, conf=0.90)
- Supportedcites [8] 8. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. Association for Computing Machinery, New York, NY, USA. source
The citation identifies this paper as an example of work on the specific task of email writing, not as evidence for the claim about most previous work. The Language Model section (p. 5) says its model was fine-tuned for an email use case, and the Limitations section (p. 16) describes the study’s email-composition tasks. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we show that the sentences generated by GPT-3 had less spelling and grammar errors than the sentences written by writers in CoAuthor.”
- Also stated in: § 5.1.1 Language Capabilities: Ability to Generate Fluent Text. · plain
“Figure 4 (a) shows that sentences written by writers had more spelling and grammar errors compared to GPT-3-generated sentences in fnal texts.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1 GPT-3's Generative Capabilities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § 5.1.1 Language Capabilities: Ability to Generate Fluent Text. · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the number of errors per word (averaged across all sentences in creative and argumentative writing) was 0.037 ± 0.001 for writers, 0.033 ± 0.001 for GPT-3, and 0.032 ± 0.001 for both (the number next to the average indicates the standard error of measurement).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoAuthor contains interactions between writers and LMs across multiple writing tasks, writing prompts, and writers.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The grammaticality of GPT-3-generated text varies across prompts (Figure 5).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Do prompts infuence GPT-3's grammaticality?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When we computed the distinct-2 score on all suggestions generated by GPT-3, the score was even lower (0.868 ± 0.001), further confrming the previous fndings.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We recruited 63 crowd workers (writers) from Amazon Mechanical Turk to account for individuals with potentially diferent backgrounds and writing styles.”
- Also stated in: § 4.3.1 Writers and prompts. · plain
“Among these writers, 63 writers participated in the main round, where 62 writers were native English speakers and one was not.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § 4.3.1 Writers and prompts. · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the distinct-2 score (averaged across all sentences in creative and argumentative writing) was 0.884 ± 0.001 for writers, 0.876 ± 0.001 for GPT-3, and 0.923 ± 0.001 for both.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 23
- The quote only partly states this claim. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Datasets are central to evaluating LMs' generative capabilities in NLP [12,25,28,32,49,55].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.4 Datasets in NLP
What the sources say
- Not matchedcites [12] 12. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of Text Generation: A Survey.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [25] 25. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg source
The Introduction (page not available in our copy) describes GEM datasets and challenging test sets as a way to assess model performance. The paper’s discussion of NLG benchmarks also treats dataset selection as integral to evaluation, supporting the broad claim. (LLM, conf=0.90)
- Not matchedcites [28] 28. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [32] 32. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Association for Computational Linguistics, Online, 4110-4124. sourceExperimental — accuracy not yet measured
The Background (p. 2) says task-level datasets have traditionally been used to measure progress in NLP, supporting the general point about datasets and evaluation. But Caveats and Objections (p. 8) says Dynabench currently focuses on classification and span extraction, while evaluating generative tasks requires further research. The paper therefore does not establish the claim specifically about evaluating language models’ generative capabilities. (LLM, conf=0.88)
- Not matchedcites [49] 49. Sebastian Ruder. 2021. Challenges and Opportunities in NLP Benchmarking. http://ruder.io/nlp-benchmarking
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [55] 55. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation. Association for Computational Linguistics, Tokyo, Japan, 355-368. sourceExperimental — accuracy not yet measured
Section 3.4 (Design; page unavailable) says the number of generated items matters for the statistical power of human evaluations. But our copy does not say that datasets are central to evaluating language models’ generative capabilities; searches for datasets, benchmarks, and test sets found no passage making that point. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Harnessing LMs' generative capabilities to power interaction designs begins with a holistic understanding of these capabilities [5,68]; this includes understanding what LMs can and cannot do under diverse interaction contexts.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 1 INTRODUCTION
What the sources say
- Partly supportedcites [5] 5. Sara Bly and Elizabeth F Churchill. 1999. Design through matchmaking: technol- ogy in search of users. interactions 6, 2 (1999), 23-31. sourceExperimental — accuracy not yet measured
The Summary (page unavailable) describes assessing a technology’s capabilities against work domains, and “Mapping Technology to Work Activities” (page unavailable) explicitly considers activities the technology supports easily and those it does not. The paper applies this approach to virtual environments, not language models or their generative capabilities, so the LM-specific application is an extension beyond the source. (LLM, conf=0.84)
- Partly supportedcites [68] 68. Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re- Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difcult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Comput- ing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376301 sourceExperimental — accuracy not yet measured
In “What Are AI’s Capabilities and Limits?” (page unavailable), the paper says designers need to understand a technology’s capabilities and limitations to see its design possibilities. “Capability Uncertainty” adds that users and use contexts affect AI capabilities. The paper makes these points about AI systems generally, including systems with open-ended outputs, but does not specifically establish them for language models’ generative capabilities. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Datasets can account for the subjective interpretation of capabilities and allow various interpretations.For example, Cuadra et al. [17] provided a video dataset capturing user interactions with a voice assistant without defning which interactions are good.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.3 Datasets in HCI
What the sources say
- Partly supportedcites [17] 17. Andrea Cuadra, Hansol Lee, Jason Cho, and Wendy Ju. 2021. Look at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors. arXiv preprint arXiv:2104.07153 (2021). sourceExperimental — accuracy not yet measured
Section 3.5 (p. 6) confirms that Cuadra et al. provided video clips of people interacting with a voice assistant, and Section 6.1 (p. 8) acknowledges subjectivity in classifying those interactions. But Section 3.4 (p. 6) says the authors labeled each clip using defined categories for successful responses, unsuccessful responses, omissions, and idle periods. The claim that they did so without defining which interactions were good is therefore not supported. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25,32,47,56].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 1 INTRODUCTION
What the sources say
- Supportedcites [25] 25. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg source
Section 2, “Benchmarks in NLG” (page not provided), says NLP benchmarks are common and enable researchers to compare models fairly; it calls them critical for measuring modeling progress. The paper’s discussion of selecting datasets for its own benchmark supports the citing sentence’s broad point about datasets being useful for model evaluation. (LLM, conf=0.88)
- Supportedcites [32] 32. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Association for Computational Linguistics, Online, 4110-4124. source
The Background (p. 2) says NLP progress has traditionally been measured using task-level datasets that became accepted benchmarks. The Introduction (p. 1) says benchmarks enable model comparisons and have supported advances in NLP, including the era of pre-trained language models. (LLM, conf=0.90)
- Supportedcites [47] 47. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 784-789. source
Section 3.2 describes earlier question-answering datasets used to test systems, and Section 5.2 evaluates reading-comprehension models on SQuAD and SQUADRUN. Together, these passages support the broad point that datasets have been useful for evaluating NLP models over time. (LLM, conf=0.86)
- Supportedcites [56] 56. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Brussels, Belgium, 353-355. source
The paper’s GLUE overview (page not provided) describes a benchmark built from existing datasets and an online platform for evaluating and comparing language-understanding models. Its Diagnostic Dataset section also describes using dataset examples to probe trained models. These passages support the broad claim that datasets are useful for evaluating models in NLP research. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Datasets can cover diverse contexts.For example, Theodorou et al. [54] published a dataset of photos taken by blind and low vision users, for assessing how well diferent object recognition models can serve this user population.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.3 Datasets in HCI
What the sources say
- Partly supportedcites [54] 54. Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morri- son, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021. Disability- First Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors. Association for Computing Machinery, New York, NY, USA. sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says the ORBIT project collected a dataset from blind and low vision people for developing teachable object recognisers. However, the Introduction and Conclusion specify that contributors supplied videos rather than photos; the cited passages also do not establish that the dataset was published to compare different recognition models. (LLM, conf=0.86)
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although this approach may provide less rich insights than interviews, it can cover relatively diverse contexts across tasks and writers, while allowing for a fne-grained analysis of interactions.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.3 Limitations of Traditional Methods.
What the sources say
- Supportedcites [8] 8. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. Association for Computing Machinery, New York, NY, USA. source
Study Design and Apparatus (p. 6) describes logging detailed interactions across three email scenarios. Participants and Procedure (p. 8) reports four tasks per person, and Writing Research (p. 4) identifies both native and non-native English writers and the detailed examination of interaction logs. Together, these support the claim about varied tasks and writers and fine-grained analysis. (LLM, conf=0.91)
- Partly supportedcites [48] 48. Melissa Roemmele and Andrew S. Gordon. 2018. Automated Assistance for Creative Writing with an RNN Language Model. In Proceedings of the 23rd In- ternational Conference on Intelligent User Interfaces Companion (Tokyo, Japan) (IUI '18 Companion). Association for Computing Machinery, New York, NY, USA, Article 21, 2 pages. sourceExperimental — accuracy not yet measured
The Experiment and Results section (page unavailable) reports 139 writers, 940 suggestion–modification pairs, and an edit-distance analysis, supporting fine-grained analysis across writers. Participants wrote stories on any topic, but all performed the same story-writing task; the paper does not establish coverage across different tasks or compare interaction logs with interviews. (LLM, conf=0.86)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Pre-trained on massive amounts of text on the Internet, LMs are known to produce linguistically fawed, factually incorrect, or even ethically problematic text at times [1,24].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.1 Language Models' Generative Capabilities.
What the sources say
- Partly supportedcites [1] 1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. arXiv:2101.05783 [cs.CL] sourceExperimental — accuracy not yet measured
The paper says language models are trained on large corpora including internet text (p. 2), and its Results (p. 5) document biased, violent GPT-3 completions. A figure caption (p. 7) also describes some generated descriptions as less coherent. Our copy does not establish the claim that the models produce factually incorrect text. (LLM, conf=0.88)
- Partly supportedcites [24] 24. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3356-3369. sourceExperimental — accuracy not yet measured
The Abstract and §3.1 report that pretrained language models sometimes generate toxic text, supporting the ethically problematic part of the claim. §8 (Related Work) also notes incoherence in model generations. The paper describes factually unreliable content in web-text corpora, but the passages do not establish that the models generate factually incorrect text. (LLM, conf=0.88)
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“How users adapt to systems over time is of interest for not only HCI researchers but also NLP researchers [59,60].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.2 Assess the Plausibility of Hypotheses.
What the sources say
- Partly supportedcites [59] 59. Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a Programming Language via Interactive Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 929-938. sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) studies how users’ language choices change as they interact with a natural-language system, supporting the NLP-research part of the claim. Our copy does not establish that this question is also of interest to HCI researchers. (LLM, conf=0.76)
- Supportedcites [60] 60. Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning Language Games through Interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 2368-2378. source
The Abstract identifies human adaptation to the computer as a central feature of the paper’s language-learning setting. The section “Most successful players (1st–20th)” reports how players changed their language over time, supporting this citation as an example of NLP research on user adaptation. Page numbers are not provided in our copy. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On an interaction level, good writing collaboration can exhibit diferent interaction patterns, such as diferent levels of mutuality and equality[2].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Partly supportedcites [2] 2. Zsuzsanna I Abrams. 2019. Collaborative Writing and Text Quality in Google Docs. Language Learning & Technology 23, 2 (2019), 22-42. sourceExperimental — accuracy not yet measured
The Review of the Literature describes interaction patterns using differing levels of equality and mutuality, and Research Question 1 reports several such patterns among the writing groups (page numbers unavailable). The paper does not establish that all those patterns constitute *good* collaboration; it associates greater collaboration with some better writing outcomes. (LLM, conf=0.82)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Partly supportedcites [6] 6. Joel Boch. 2007. Abdullah's blogging: A generation 1.5 student enters the blogo- sphere. Language Learning & Technology 11, 2 (2007), 128-141. sourceExperimental — accuracy not yet measured
The Introduction and Background Research (page unavailable) says that blogging can give students a greater sense of possible audiences and help them learn to respond to those audiences. The paper also describes interaction between readers and writers, but it does not establish that a human writing collaborator improves a text’s fluency. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [53] 53. Neomy Storch. 2005. Collaborative writing: Product, process, and students' refections. Journal of second language writing 14, 3 (2005), 153-173. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Not checked — no full text of the sourcecites [65] 65. Luxin Yang. 2014. Examining the mediational means in collaborative writing: Case studies of undergraduate ESL students in business courses. Journal of Second Language Writing 23 (2014), 74-89. source
- Supportedcites [70] 70. Soobin Yim and Mark Warschauer. 2017. Web-based collaborative writing in L2 contexts: Methodological insights from text mining. Language Learning & Technology 21, 1 (2017), 146-165. source
The Introduction links collaborative writing to socialization opportunities with specific discourse communities. This supports the attributed point about a human writing collaborator helping writers engage with those communities; the paper cites Yang (2014) for that point. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“HCI researchers can use CoAuthor to formulate hypotheses via replay enactment [29].”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.1 Formulate Hypotheses.
What the sources say
- Partly supportedcites [29] 29. Kenneth Holstein, Erik Harpstead, Rebecca Gulotta, and Jodi Forlizzi. 2020. Replay Enactments: Exploring Possible Futures through Historical Data. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS '20). Association for Computing Machinery, New York, NY, USA, 1607-1618. https://doi.org/10.1145/3357236.3395427 sourceExperimental — accuracy not yet measured
The Introduction describes replay enactments as a way for designers to explore possible system behaviors, and the Conclusions say they help designers play out the implications of design decisions. Our copy does not mention CoAuthor or say that researchers use it to formulate hypotheses through this method (searched: CoAuthor, formulate hypotheses, hypothes). (LLM, conf=0.82)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoAuthor contains 20 writing prompts, brief passages of text that provides a potential topic idea or starting point.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor covers creative writing and argumentative writing, which have distinct goals and require diferent sets of writing skills.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.1 Dataset Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we demonstrate that the amount of collaboration between writers and GPT-3 varied signifcantly across writers, but less depended on writing prompts and the randomness of suggestions.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1 GPT-3's Generative Capabilities
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, each writing session results in 418 words of text, contains 11.8 queries to the system, has acceptance rate of 72.3%, and 72.6% of text is written by writers.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.4 Dataset Overview
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“13% and 7% of accepted suggestions contained new named entities”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.2 Ideation Capabilities: Ability to Generate New Ideas.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We provide a tool for replaying all writing sessions in CoAuthor, giving designers a felt understanding of the interactions.”
- Also stated in: § Abstract · plain
“The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > I
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § Abstract · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We designed CoAuthor to be reusable and easily expandable in the future.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor supports multiple analytical perspectives and goals by providing various measurements in three categories that can be used to defne good collaboration: writing outcome, writer perception, and writer behavior.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor consists of a set of writing sessions.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains 830 stories written by 58 writers for creative writing and 615 essays written by 49 writers for argumentative writing.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.4 Dataset Overview
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CoAuthor preserves details of rich interactions as events and event blocks.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Support subjective interpretations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrated the feasibility of this approach and discussed insights designers can draw from the dataset.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 7 CONCLUSION
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In argumentative writing, the sessions with GPT-3 with high randomness resulted in equality of 0.29 ± 0.01 and mutuality of 0.47 ± 0.01, whereas GPT-3 with low randomness resulted in 0.22 ± 0.01 and 0.38 ± 0.01 on average (Figure 9 (b)).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, some writers had more equal and mutual collaboration with GPT-3 with high randomness (dark blue), whereas others had such collaboration with GPT-3 with low randomness (light blue).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, each writing session is 418 words long, contains 11.8 queries to the system, has acceptance rate of 72.3% (how often writers accepted suggestions from GPT-3), and results in 72.6% of fnal texts written by writers (as opposed to GPT-3).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for instance, Writer #1 scored around 0.45 and 0.75 for equality and mutuality, whereas Writer #2 scored below 0.1 and 0.2”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Example of a story in which a writer accepted a suggestion from GPT-3 with a new named entity”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Equality and mutuality (y-axis) and vary across prompts (x-axis) in creative (a) and argumentative writing (b), but to the less extent than across writers.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When aggregated by prompts, 5 out of 10 prompts (50%) and 8 out of 10 prompts (80%) had higher equality and mutuality scores with GPT-3 with low randomness.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, in creative writing, writing sessions with GPT-3 with high randomness resulted in 0.30 ± 0.01 for equality and 0.48 ± 0.01 for mutuality, whereas the sessions with GPT-3 with low randomness resulted in 0.29 ± 0.01 and 0.49 ± 0.01, respectively (Figure 9 (a)).”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in creative writing, some writers used the system to advance plots (62.9%), whereas some used it to add details to stories (51.4%) according to the survey's multiple choice question.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.3 Collaboration Capabilities: Ability to Work Jointly.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, it identifes a need for holistic understanding of LMs' generative capabilities for interaction design.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > I
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LMs' generative capabilities are highly context-dependent; therefore, it is difcult to cover all contexts.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.2 Challenges in Understanding Generative Capabilities.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LMs' generative capabilities can be subjectively interpreted; therefore, they are susceptible to varying evaluations even within a given interaction context.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.2.2 Challenges in Understanding Generative Capabilities.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we present evidence that GPT-3 is capable of providing new ideas to writers, infuencing their subsequent writing.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1 GPT-3's Generative Capabilities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
In the paper
“Appropriate interaction design for a new technology requires a deep understanding of its capabilities and limitations.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 2.1.1 Types of Understanding.
What the sources say
- Not matchedcites [9] 9. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
In the paper
“When we computed the distinct-2 score on all suggestions generated by GPT-3, the score was even lower (0.868 ± 0.001), further confrming the previous fndings.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 5.1.1 Language Capabilities: Ability to Generate Fluent Text.
What the sources say
- Not checked — no full text of the sourcecites [27] 27. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1689-1701. source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
In the paper
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
What the sources say
- Not matchedcites [19] 19. Richard Donato. 1994. Collective scafolding in a second language. Vygotskian approaches to second language research (1994), 33-56.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, datasets in NLP can provide resources to help HCI researchers reason about LMs' generative capabilities in a holistic manner.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“HCI researchers can provide analytical tools for NLP researchers to better investigate LMs's capabilities interactive settings.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argued that curating and analyzing large interaction datasets is one viable approach, as it can cover diverse interaction contexts and support various interpretations of good collaboration.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 7 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that datasets have the potential to serve as boundary objects between the HCI and NLP communities.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, large interaction datasets in HCI can embed human-centered values and practices, and further incorporate them into technical advances, when used to train or evaluate LMs.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By having datasets as boundary objects, the HCI and NLP communities can easily communicate results and have shared understanding of LMs' generative capabilities.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.1 Datasets as Boundary Objects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Concretely, researchers can replay writing sessions in CoAuthor, which materializes the dynamics of interactions between writers and GPT-3 and makes complex system behavior more tangible.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.1 Formulate Hypotheses.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The authors of this paper manually rated 201 submissions with the goal of qualifying writers who demonstrated that they could meet two requirements.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > D QUALIFICATION ROUND
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoAuthor can serve as a starting point to assess the plausibility of hypotheses.”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 6.2.2 Assess the Plausibility of Hypotheses.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
“Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing.”
“We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities”
The paper relies on this without support.
The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
“We recruited 63 crowd workers (writers) from Amazon Mechanical Turk to account for individuals with potentially diferent backgrounds and writing styles.”
“Among these writers, 63 writers participated in the main round, where 62 writers were native English speakers and one was not.”
The paper relies on this without support.
Most current language models are trained and evaluated on datasets that ignore interactive settings.
“Most current LMs are trained and evaluated on datasets that do not consider interactive settings.”
The paper relies on this without support.
High-randomness GPT-3 suggestions tend to be more diverse.
“as they tend to be more diverse.”
The paper relies on this without support.
The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
“On an interaction level, good writing collaboration can exhibit diferent interaction patterns, such as diferent levels of mutuality and equality[2].”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Systems commonly provided suggestions for revisions which could be integrated directly into the writing area [14,18,75], aiding efficiency and idea exploration in the text.”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [75]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies of interactive writing systems note that the point of contact between the writer and the model (when suggestions appear, what the writer must do to take them, etc.) shapes how much of the writing remains the writer's own (Lee et al., 2022).”
From Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions? (2026) · cites this paper as Lee et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast, prior research has found that people’s perceptions of AI authorship in co-creation are nuanced. These perceptions depend on various dimensions, including the type of contribution made by the AI, how much each party contributed, and who led the co-creative effort [33, 44, 71, 92].”
From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [44]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing (Shuster et al., 2022;Thoppilan et al., 2022;Lee et al., 2022, inter alia).”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Lee et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“More recent work on AI co-writing for creative and argumentative texts has explicitly investigated issues of agency and ownership, finding that the less a user writes, the less ownership they feel over the final text [36].”
From Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems (2026) · cites this paper as [36]
The cited source says what the claim attributes to it.
In the citing paper
“GenAI has signi cantly impacted various aspects of daily life, including enhancing productivity [68], writing [57, 58], interpersonal communication [35], and relationships [34, 76].”
From Creativity in the Age of AI: Evaluating the Impact of Generative AI on Design Outputs and Designers' Creative Thinking (2024) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“In particular, a writer's sense of ownership may diminish as the proportion of system-generated content increases (Lee et al., 2022).”
From Ink and algorithm: Exploring temporal dynamics in generative AI ‐assisted writing (2026) · cites this paper as Lee et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“More commonly, systems display suggestions in a separated interface [29,43,75,89].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [75]
The cited source says what the claim attributes to it.
In the citing paper
“Some of these factors regard characteristics of the contributions themselves, such as the types of contributions made [33, 71, 84, 92] and the amount of material contributed [44].”
From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [44]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Furthermore, students may be less inclined to disclose GenAI use, given widespread awareness of the difficulty humans face in accurately detecting AI-generated text 33,34,35 and the unreliability of current detection tools 36,37 .”
From Style, sentiment, and quality of undergraduate writing in the AI era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredIn the citing paper
“These interfaces are highly intuitive, and mimic conversational interactions between humans, but lack scaffolding for crafting prompts, which can be difficult for novice AI users [49,98].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [49]
In the citing paper
“While these approaches provide an objective representation of writer-suggestion interactions at a behavioural level, they lack insight into the inner cognitive decision-making processes that writers might engage in.”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [45]
In the citing paper
“Lee et al. [54] find that LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.”
From Homogenization Effects of Large Language Models on Human Creative Ideation (2024) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > Click here to start writing! ([ANONYMIZED])
“| | | Combined | 20 | | 63 | | | 1445 | 11.2 | 366 | 11.8 | 72.3 | 72.6 | |”
§ CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities > 4.4 Dataset Overview
“On average, each writing session results in 418 words of text”
Citations
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 2×
“The goal of Natural Language Generation (NLG) is to produce fluent text in many domains, such as machine translation [21], summarization [34], dialogue [30], style transfer [18], and programming code [13].”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“Any LM’s performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1×
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25, 32, 47, 56].”
Background5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Self-citationGaver, W. W. (1991) · Proceedings of the SIGCHI conference on Human factors in computing systems Reaching through technology - CHI '91 · doi:10.1145/108844.108856 · Cited by 1,536
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Patterns of Interaction in ESL Pair Work
Storch, N. (2002) · Language Learning · doi:10.1111/1467-9922.00179 · Cited by 1,080 · cited 2×
“• Writer interaction: We measured how much interaction writers had with GPT-3 in the writing process using the notions of equality and mutuality from Storch [52].”
Method
Research works printed but not linked: 8
- Joel Boch . 2007 . Abdullah’s blogging: A generation 1.5 student enters the blogosphere . Language Learning & Technology 11 , 2 (2007), 128 – 141 . Joel Boch. 2007. Abdullah’s blogging: A generation 1.5 student enters the blogosphere. Language Learning & Technology 11, 2 (2007), 128–141.
- Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates , Inc ., 1877 –1901. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901.
- Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann.
- Alex Calderwood , Vivian Qiu , Katy Ilonka Gero , and Lydia B Chilton . 2020. How Novelists Use Generative Language Models: An Exploratory User Study . In Proceedings of the Workshops on Human-AI Co-Creation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces . Association for Computing Machinery . Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Proceedings of the Workshops on Human-AI Co-Creation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces. Association for Computing Machinery.
- Richard Donato . 1994. Collective scaffolding in a second language. Vygotskian approaches to second language research ( 1994 ), 33–56. Richard Donato. 1994. Collective scaffolding in a second language. Vygotskian approaches to second language research (1994), 33–56.
- Rebecca Fiebrink and Perry R Cook . 2010 . The Wekinator: a system for real-time, interactive machine learning in music . In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010) (Utrecht). Rebecca Fiebrink and Perry R Cook. 2010. The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)(Utrecht).
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog 1, 8 ( 2019 ). Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog 1, 8 (2019).
- Donald A Schön. 1984. The reflective practitioner: How professionals think in action. Vol. 5126. Basic books. Donald A Schön. 1984. The reflective practitioner: How professionals think in action. Vol. 5126. Basic books.
Cited by
Sorted by citations
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Self-citationPerkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839
“…AI based technologies, there is ongoing research exploring how the intricacies of co-creation may be further be codified and explored in the fields of creative writing (M. Lee et al., 2022; Y. Lee et al., 2022), software development (Biderman & Raff, 2022) and artistic outputs (Oppenlaender, 2022).”
Alqahtani, T., Badreldin, H. A., Alrashed, M., Alshaya, A. I., Alghamdi, S. S., bin Saleh, K., Alowais, S. A., Alshaya, O. A., Rahman, I., Al Yami, M. S., et al. (2023) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2023.05.016 · Cited by 565
Marzuki, Widiati, U., Rusdin, D., Darwin, & Indrawati, I. (2023) · Cogent Education · doi:10.1080/2331186x.2023.2236469 · Cited by 524
“Works by Nazari et al. (2021), Dale and Viethen (2021), M. Lee et al. (2022), and Zhao (2022) served as valuable references.”
BackgroundSharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500
Metadata
- Authors
- Lee, M., Liang, P., & Yang, Q.
- Affiliations
- Stanford University (US)
- Cornell University (US)
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-19
- OpenAlex
- W4220747294
- Cited by
- 345
- References
- 39
- Citations by year
- 2026: 61 · 2025: 96 · 2024: 118 · 2023: 56 · 2022: 14
- FWCI
- 56.81
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI collaboration, large language models, GPT-3, writing support, creative writing, argumentative writing, human-AI interaction, idea generation, language generation, interaction design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
