CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
Open accessAbstract
Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs’ generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3’s capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3’s language, ideation, and collaboration capabilities, and reveal its contribution as a writing “collaborator” under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs’ promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
26 of 42 cited sources checked · 11 supported · 0 flagged · accuracy not yet measured
- Premisecitation
A language model's performance varies substantially with preceding text and decoding parameters.
“Any LM's performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
Quote in the paperSubstantial claim1 of 2 sources checked- cites [38] 38. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv:2107.13586 [cs.CL] source
Citation [38] is attached to the preceding-text part of the sentence, not decoding parameters, which are attributed to [27]. The survey’s “Prompt Engineering” section (p. 11) says prompts are designed to improve downstream performance, supporting the point that preceding text can matter. It does not establish that performance fluctuates significantly for any language model. (LLM, conf=0.78)
Experimental — accuracy not yet measured - cites [27] 27. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1689-1701. source
- Premisecitation
Sentences written by writers tend to have more grammar and usage errors than those written by GPT-3.
“This matches with Dou et al. [20]'s fnding that sentences written by writers tend to have more grammar and usage errors compared to those written by GPT-3.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [20] 20. Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021. Scarecrow: A Framework for Scrutinizing Machine Text. arXiv preprint arXiv:2107.01294 (2021). source
Section E.6 (page not provided) reports more Grammar and Usage error annotations for human-authored text than for the best GPT-3 configuration tested. But the claim presents those annotations as a general finding about actual writing errors. The same section cautions that many human-text annotations were artifacts of the text format and estimates that human-authored paragraphs had far fewer legitimate issues overall. (LLM, conf=0.85)
Experimental — accuracy not yet measured
- Interpretationcitation
Most previous log-analysis work used restricted interaction settings and adapted models or interfaces for specific tasks, making generalization to other tasks and configurations difficult.
“However, most previous work considered restricted interaction settings (e.g.strict turn-taking [15]) and adapted LMs and interfaces (e.g.fnd-tuned GPT-2 [58]) on specifc tasks (e.g.email writing [8]), thereby making it hard to generalize to other tasks and confgurations of the same model.”
Quote in the paperSubstantial claim3 of 3 sources checked- cites [15] 15. Elizabeth Clark and Noah A. Smith. 2021. Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 3566-3575. source
- cites [58] 58. Lu Wang, Munif Ishad Mujib, Jake Williams, George Demiris, and Jina Huh-Yoo. 2021. An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers. arXiv:2107.13115 [cs.HC] source
- cites [8] 8. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. Association for Computing Machinery, New York, NY, USA. source
- Findingown data
Sentences generated by GPT-3 had fewer spelling and grammar errors than sentences written by writers in CoAuthor.
“we show that the sentences generated by GPT-3 had less spelling and grammar errors than the sentences written by writers in CoAuthor.”
- Also stated in: § 5.1.1 Language Capabilities: Ability to Generate Fluent Text. · plain
“Figure 4 (a) shows that sentences written by writers had more spelling and grammar errors compared to GPT-3-generated sentences in fnal texts.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § 5.1.1 Language Capabilities: Ability to Generate Fluent Text. · plain
- Findingown data
The number of spelling and grammar errors per word averaged across all sentences in creative and argumentative writing was 0.037 ± 0.001 for writers, 0.033 ± 0.001 for GPT-3, and 0.032 ± 0.001 for both.
“Overall, the number of errors per word (averaged across all sentences in creative and argumentative writing) was 0.037 ± 0.001 for writers, 0.033 ± 0.001 for GPT-3, and 0.032 ± 0.001 for both (the number next to the average indicates the standard error of measurement).”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Findingown data
CoAuthor contains interactions between writers and language models across multiple writing tasks, writing prompts, and writers.
“CoAuthor contains interactions between writers and LMs across multiple writing tasks, writing prompts, and writers.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Findingown data
GPT-3's output grammaticality differs depending on the prompt used.
“The grammaticality of GPT-3-generated text varies across prompts (Figure 5).”
Quote in the paperNot a substantial claim - Findingown data
The distinct-2 score computed on all suggestions generated by GPT-3 was 0.868 ± 0.001.
“When we computed the distinct-2 score on all suggestions generated by GPT-3, the score was even lower (0.868 ± 0.001), further confrming the previous fndings.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Findingown data
CoAuthor recruited 63 crowd workers from Amazon Mechanical Turk as writers.
“We recruited 63 crowd workers (writers) from Amazon Mechanical Turk to account for individuals with potentially diferent backgrounds and writing styles.”
- Also stated in: § 4.3.1 Writers and prompts. · plain
“Among these writers, 63 writers participated in the main round, where 62 writers were native English speakers and one was not.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § 4.3.1 Writers and prompts. · plain
- Findingown data
The distinct-2 score averaged across all sentences in creative and argumentative writing was 0.884 ± 0.001 for writers, 0.876 ± 0.001 for GPT-3, and 0.923 ± 0.001 for both.
“Overall, the distinct-2 score (averaged across all sentences in creative and argumentative writing) was 0.884 ± 0.001 for writers, 0.876 ± 0.001 for GPT-3, and 0.923 ± 0.001 for both.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Findingown data
Sentences written by both writers and GPT-3 contained more diverse vocabulary than sentences written by writers alone or GPT-3 alone.
“Figure 4 (b) shows that sentences written by both writers and GPT-3 contained more diverse vocabulary compared to the sentences written by writers alone and GPT-3 alone.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Findingown data
CoAuthor contains interactions between 63 writers and four GPT-3 instances across 1445 writing sessions.
“CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions.”
- Also stated in: § I · plain
“It captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions (Section 4).”
- Also stated in: § I · plain
“It contains rich interactions between 63 writers and 4 instances of GPT-3 across 1445 writing sessions.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § I · plain
- Findingown data
In argumentative writing, 14% of GPT-3's new named entities were later reused by writers.
“Among the new named entities proposed by GPT-3, 20% and 14% of them were reused by writers in subsequent writing in creative and argumentative writing, respectively (Figure 7).”
Quote in the paperNot a substantial claim - Findingown data
The correlation with text produced was stronger for query count (0.42) and accepted suggestions (0.48) than for writing time (0.29).
“The correlation was much stronger for the number of queries and accepted suggestions (0.42 and 0.48) compared to the time writers spent writing (0.29).”
Quote in the paperSubstantial claim - Findingown data
In both creative and argumentative writing, the rate of spelling and grammar errors per word in GPT-3-generated sentences differs across writing prompts.
“For both creative (a) and argumentative (b) writing, the number of spelling and grammar errors per word (y-axis) in GPT-3-generated sentences vary across writing prompts (x-axis).”
Quote in the paperNot a substantial claim - Findingown data
Writers accepted GPT-3 suggestions at an average rate of 72.3%.
“On average, each writing session is 418 words long, contains 11.8 queries to the system, has acceptance rate of 72.3% (how often writers accepted suggestions from GPT-3), and results in 72.6% of fnal texts written by writers (as opposed to GPT-3).”
Quote in the paperNot a substantial claim - Findingown data
In creative writing, 13% of GPT-3 suggestions introduced a new named entity.
“In creative writing, 13% of suggestions (generated by GPT-3) contained new named entities, among which 20% were reused by writers in subsequent writing.”
Quote in the paperNot a substantial claim - Findingown data
In argumentative writing, high-randomness GPT-3's accepted suggestions had new named entities at 9 ± 1%, compared with 6 ± 0% for low randomness.
“Similarly, in argumentative writing, accepted suggestions from GPT-3 with high randomness contained new named entities more often (9 ± 1%), compared to the ones from GPT-3 with low randomness (6 ± 0%).”
Quote in the paperSubstantial claim - Findingown data
Collaboration equality and mutuality differed much more by writer than by prompt.
“Both collaboration equality and mutuality varied greatly depending on writers, but less on prompts.”
Quote in the paperNot a substantial claim - Findingown data
In argumentative writing, reuse rates of new named entities were similar for high randomness (13 ± 2%) and low randomness (15 ± 3%).
“Yet, the likelihoods of new named entities being reused by writers were similar for both high randomness (13 ± 2%) and low randomness (15 ± 3%).”
Quote in the paperNot a substantial claim - Findingown data
The CoAuthor dataset contains 1445 writing sessions authored by 63 writers.
“The dataset contains 1445 writing sessions written by 63 writers.”
Quote in the paperNot a substantial claim - Findingnone given
We built CoAuthor, a dataset of rich interactions from 63 writers and four GPT-3 instances across 1445 writing sessions.
“Exemplifying this approach, we created CoAuthor, a dataset containing rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions.”
Quote in the paperSubstantial claim - Findingown data
On average, 72.6% of the final text in a session was written by the human writer rather than GPT-3.
“On average, each writing session is 418 words long, contains 11.8 queries to the system, has acceptance rate of 72.3% (how often writers accepted suggestions from GPT-3), and results in 72.6% of fnal texts written by writers (as opposed to GPT-3).”
Quote in the paperNot a substantial claim - Findingown data
In CoAuthor, we found no meaningful correlation between edit counts and ownership or satisfaction scores.
“However, we did not observe meaningful correlation between the amount of edits by writers and their ownership or satisfaction scores in CoAuthor.”
Quote in the paperSubstantial claim - Findingown data
Equality and mutuality of collaboration vary substantially from writer to writer.
“Equality (a) and mutuality (b) (y-axis) vary across writers (x-axis) greatly.”
Quote in the paperNot a substantial claim- Rests on an assumption: The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
- Findingown data
Across argumentative prompts, 9 of 10 had higher equality and mutuality with high randomness.
“When aggregated by prompts, 9 out of 10 prompts (90%) had higher equality and mutuality scores with GPT-3 with high randomness.”
Quote in the paperNot a substantial claim - Findingown data
In creative writing, reuse rates of new named entities were similar for high randomness (19 ± 2%) and low randomness (22 ± 2%).
“However, the likelihoods of new named entities being reused by writers were similar for both high randomness (19 ± 2%) and low randomness (22 ± 2%).”
Quote in the paperNot a substantial claim - Findingown data
High-randomness GPT-3 produced sentences with new named entities more often than low-randomness GPT-3.
“The GPT-3 instance with high randomness was more likely to generate sentences with new named entities compared to the one with low randomness, while the likelihoods of the entities being reused by writers were similar.”
Quote in the paperSubstantial claim - Findingown data
In creative writing, high-randomness GPT-3's accepted suggestions had new named entities at 15 ± 1%, compared with 10 ± 0% for low randomness.
“In creative writing, accepted suggestions from GPT-3 with high randomness contained new named entities more often (15 ± 1%), compared to the ones from GPT-3 with low randomness (10 ± 0%).”
Quote in the paperSubstantial claim - Findingown data
A writing session averages 11.8 queries to the suggestion system.
“On average, each writing session is 418 words long, contains 11.8 queries to the system, has acceptance rate of 72.3% (how often writers accepted suggestions from GPT-3), and results in 72.6% of fnal texts written by writers (as opposed to GPT-3).”
Quote in the paperNot a substantial claim - Findingown data
Some writers appreciated and found helpful the suggestions that contained new named entities.
“Survey responses show that some writers liked and found the suggestions with new named entities helpful”
Quote in the paperNot a substantial claim - Findingown data
In creative writing, 20% of GPT-3's new named entities were later reused by writers.
“Among the new named entities proposed by GPT-3, 20% and 14% of them were reused by writers in subsequent writing in creative and argumentative writing, respectively (Figure 7).”
Quote in the paperNot a substantial claim - Findingown data
Sentences produced jointly by writers and GPT-3 used more diverse vocabulary than sentences produced by writers alone or by GPT-3 alone.
“Sentences written by both writers and GPT-3 had fewer spelling and grammatical errors (a) and contained more diverse vocabulary (b) compared to sentences written by writers alone and GPT-3 alone.”
Quote in the paperSubstantial claim - Findingown data
Of the creative-writing GPT-3 suggestions that introduced a new named entity, 20% were reused by writers in their later writing.
“In creative writing, 13% of suggestions (generated by GPT-3) contained new named entities, among which 20% were reused by writers in subsequent writing.”
Quote in the paperNot a substantial claim - Findingown data
In argumentative writing, 7% of accepted suggestions introduced new named entities.
“13% and 7% of accepted suggestions contained new named entities”
Quote in the paperNot a substantial claim - Findingown data
Sentences produced jointly by writers and GPT-3 contained fewer spelling and grammatical errors than sentences produced by writers alone or by GPT-3 alone.
“Sentences written by both writers and GPT-3 had fewer spelling and grammatical errors (a) and contained more diverse vocabulary (b) compared to sentences written by writers alone and GPT-3 alone.”
Quote in the paperSubstantial claim - Interpretationcitation
Tinkering with GPT-3 in the Playground is unlikely to cover diverse contexts.
“Similarly, tinkering with GPT-3 in the Playground (a text box where one can submit a prompt to generate a completion) [42] is unlikely to cover diverse contexts.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [42] 42. OpenAI. 2020. OpenAI GPT-3 playground: GPT-3 demo. https://gpt3demo.com/ apps/openai-gpt-3-playground
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data
CoAuthor can reveal GPT-3's contribution as a writing collaborator under various definitions of good collaboration.
“and reveal its contribution as a writing”
- Also stated in: § I · plain
“The dataset can also help researchers investigate GPT-3's contribution as a writing”
- Also stated in: § 5 DEMONSTRATING USES OF COAUTHOR · plain
“it allows designers to explore the generative capabilities of LMs holistically and to reason about its contribution as a writing”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § I · plain
- Interpretationreasoning
Contextual inquiry is more effective at capturing subjective interpretation of language models' capabilities than at covering diverse contexts.
“In other words, contextual inquiry is more efective in capturing the subjective interpretation of LMs' generative capabilities than covering diverse contexts.”
Quote in the paperSubstantial claim - Interpretationreasoning
Most NLP datasets assume full automation of tasks rather than augmentation.
“However, we argue that the underlying assumption of most datasets is the full-automation of tasks rather than augmentation.”
Quote in the paperNot a substantial claim - Interpretationcitation
Studying language models' variable capabilities requires more than interviewing users or tinkering with the model.
“Examining such variable capabilities requires more than interviewing its users [66] or tinkering with the model.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [66] 66. Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi T. Iqbal, and Jaime Teevan. 2019. Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). ACM, New York, NY, USA, Article 185, 12 pages. https://doi.org/10.1145/3290605.3300415 source
- Interpretationreasoning
Curating and analyzing large interaction datasets can help the HCI community examine language models' generative capabilities more incisively.
“In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs' generative capabilities.”
- Also stated in: § 1 INTRODUCTION · plain
“This paper proposes curating and analyzing large interaction datasets as one way to address these challenges.”
- Also stated in: § 1 INTRODUCTION · plain
“We argue that, when thoughtfully designed for HCI, datasets can also reveal what LMs can do for interaction design (Section 3).”
Quote in the paperNot a substantial claim- Rests on an assumption: GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
- Also stated in: § 1 INTRODUCTION · plain
- Interpretationreasoning
When designed thoughtfully for HCI, datasets can reveal what language models can do for interaction design.
“We argue that, when thoughtfully designed for HCI, datasets can also reveal what LMs can do for interaction design (Section 3).”
Quote in the paperNot a substantial claim- Rests on an assumption: GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
- Interpretationown data
The use of suggestions from GPT-3 may have encouraged writers to use more diverse vocabulary.
“The result may imply that the use of suggestions from GPT-3 encouraged writers to use more diverse vocabulary.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Premisecitation
Contextual inquiry reveals rich insights about how writers interpret language models' capabilities in specific contexts.
“This approach reveals rich insights about how novelists interpreted the capabilities of LMs in specifc contexts.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [10] 10. Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Pro- ceedings of the Workshops on Human-AI Co-Creation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces. Association for Computing Machinery.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
Findings about GPT-2 from contextual inquiry may not generalize to other writing contexts, nonprofessional writers, future GPT versions, or other configurations of the same model.
“However, it is unclear to what extent the fndings about GPT-2 would generalize to other writing contexts, to other nonprofessional writers, to future versions of GPT, or even to other confgurations of the same model.”
Quote in the paperSubstantial claim - Interpretationreasoning
Most NLP datasets tend not to capture the writing process, focusing instead on results.
“As a result, these datasets tend to not capture the process of writing, but rather focus on result.”
Quote in the paperSubstantial claim - Interpretationreasoning
Most NLP datasets do not consider interactive settings where users guide and correct generated outputs, instead expecting models to generate correct answers alone.
“In other words, they do not consider interactive settings where users can guide and correct systems' generated outputs, but rather expect LMs to generate correct answers alone.”
Quote in the paperNot a substantial claim - Premisecitation
Recent large language models can generate varied prose and dialogues with unprecedented fluency without task-specific adaptation.
“Recent LMs (such as GPT-2 [45], GPT-3 [7], GPT-J [57], Jurassic-1 [36], Megatron-Turing-NLG [31], and Gopher [46]) can generate a wide variety of prose and dialogues with an unprecedented level of fuency out of the box.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [45] 45. Alec Radford, Jefrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog 1, 8 (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [7] 7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jefrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato,
Not checked in the free run.
- cites [57] 57. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autore- gressive Language Model. https://github.com/kingofolz/mesh-transformer-jax.
Not checked in the free run.
- Premisecitation
Machine-generated text is less diverse than human-authored text.
“This matches with the previous fndings about machine-generated text being less diverse than human-authored counterparts [27].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [27] 27. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1689-1701. source
- Interpretationown data
CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities.
“We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities”
- Also stated in: § I · plain
“We demonstrate that CoAuthor can help to answer high-level questions about GPT-3's generative capabilities.”
- Also stated in: § 5 DEMONSTRATING USES OF COAUTHOR · boosted
“This section demonstrates the usefulness of CoAuthor for revealing GPT-3's generative capabilities in assisting creative and argumentative writing.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § I · plain
- Interpretationreasoning
Curating and analyzing large interaction datasets is a viable approach to making language models' generative capabilities more accessible to the HCI community.
“Second, it proposes curating and analyzing large interaction datasets as a viable approach to making LMs' generative capabilities more accessible to the HCI community; this opens up new research opportunities in designing and mining large interaction datasets as a research contribution.”
Quote in the paperSubstantial claim- Rests on an assumption: GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
- Interpretationown data
GPT-3 suggestions may be able to raise writers' productivity.
“This observation suggests that having suggestions from GPT-3 has the potential to increase writers' productivity.”
Quote in the paperSubstantial claim - Interpretationreasoning
Our dataset captures an interactive setting that includes in-situ generation and evaluation.
“On the other hand, our dataset considers an interactive setting, which seamlessly embodies in-situ generation and evaluation.”
Quote in the paperSubstantial claim - Interpretationreasoning
NLP researchers can use CoAuthor to train and evaluate language models for interactive writing.
“NLP researchers can train and evaluate LMs on CoAuthor to better support interactive writing.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 35
- May combine two claims. 23
- The quote only partly states this claim. 2
Classified as probably not a substantial claim.
PremisecitationDatasets are central to evaluating language models' generative capabilities in NLP.
“Datasets are central to evaluating LMs' generative capabilities in NLP [12,25,28,32,49,55].”
Quote in the paperSubstantial claim3 of 6 sources checked- cites [12] 12. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of Text Generation: A Survey.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [25] 25. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg source
- cites [28] 28. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [32] 32. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Association for Computational Linguistics, Online, 4110-4124. source
The Background (p. 2) says task-level datasets have traditionally been used to measure progress in NLP, supporting the general point about datasets and evaluation. But Caveats and Objections (p. 8) says Dynabench currently focuses on classification and span extraction, while evaluating generative tasks requires further research. The paper therefore does not establish the claim specifically about evaluating language models’ generative capabilities. (LLM, conf=0.88)
Experimental — accuracy not yet measured - cites [49] 49. Sebastian Ruder. 2021. Challenges and Opportunities in NLP Benchmarking. http://ruder.io/nlp-benchmarking
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [55] 55. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation. Association for Computational Linguistics, Tokyo, Japan, 355-368. source
Section 3.4 (Design; page unavailable) says the number of generated items matters for the statistical power of human evaluations. But our copy does not say that datasets are central to evaluating language models’ generative capabilities; searches for datasets, benchmarks, and test sets found no passage making that point. (LLM, conf=0.70)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S2 (checkable).
PremisecitationUsing language models' generative capabilities for interaction design requires a holistic understanding of what they can and cannot do across diverse interaction contexts.
“Harnessing LMs' generative capabilities to power interaction designs begins with a holistic understanding of these capabilities [5,68]; this includes understanding what LMs can and cannot do under diverse interaction contexts.”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites [5] 5. Sara Bly and Elizabeth F Churchill. 1999. Design through matchmaking: technol- ogy in search of users. interactions 6, 2 (1999), 23-31. source
The Summary (page unavailable) describes assessing a technology’s capabilities against work domains, and “Mapping Technology to Work Activities” (page unavailable) explicitly considers activities the technology supports easily and those it does not. The paper applies this approach to virtual environments, not language models or their generative capabilities, so the LM-specific application is an extension beyond the source. (LLM, conf=0.84)
Experimental — accuracy not yet measured - cites [68] 68. Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re- Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difcult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Comput- ing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376301 source
In “What Are AI’s Capabilities and Limits?” (page unavailable), the paper says designers need to understand a technology’s capabilities and limitations to see its design possibilities. “Capability Uncertainty” adds that users and use contexts affect AI capabilities. The paper makes these points about AI systems generally, including systems with open-ended outputs, but does not specifically establish them for language models’ generative capabilities. (LLM, conf=0.86)
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationDatasets can account for subjective interpretation of capabilities and allow various interpretations.
“Datasets can account for the subjective interpretation of capabilities and allow various interpretations.For example, Cuadra et al. [17] provided a video dataset capturing user interactions with a voice assistant without defning which interactions are good.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [17] 17. Andrea Cuadra, Hansol Lee, Jason Cho, and Wendy Ju. 2021. Look at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors. arXiv preprint arXiv:2104.07153 (2021). source
Section 3.5 (p. 6) confirms that Cuadra et al. provided video clips of people interacting with a voice assistant, and Section 6.1 (p. 8) acknowledges subjectivity in classifying those interactions. But Section 3.4 (p. 6) says the authors labeled each clip using defined categories for successful responses, unsuccessful responses, omissions, and idle periods. The claim that they did so without defining which interactions were good is therefore not supported. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationDatasets have long been useful for evaluating language models in natural language processing research.
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25,32,47,56].”
Quote in the paperNot a substantial claim4 of 4 sources checked- cites [25] 25. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg source
- cites [32] 32. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Association for Computational Linguistics, Online, 4110-4124. source
- cites [47] 47. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 784-789. source
- cites [56] 56. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Brussels, Belgium, 353-355. source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationDatasets can cover diverse contexts.
“Datasets can cover diverse contexts.For example, Theodorou et al. [54] published a dataset of photos taken by blind and low vision users, for assessing how well diferent object recognition models can serve this user population.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [54] 54. Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morri- son, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021. Disability- First Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors. Association for Computing Machinery, New York, NY, USA. source
The Introduction (page unavailable) says the ORBIT project collected a dataset from blind and low vision people for developing teachable object recognisers. However, the Introduction and Conclusion specify that contributors supplied videos rather than photos; the cited passages also do not establish that the dataset was published to compare different recognition models. (LLM, conf=0.86)
Experimental — accuracy not yet measured
May combine two claims.
InterpretationcitationLogging interactions and analyzing them afterward can cover relatively diverse contexts across tasks and writers while allowing fine-grained analysis.
“Although this approach may provide less rich insights than interviews, it can cover relatively diverse contexts across tasks and writers, while allowing for a fne-grained analysis of interactions.”
Quote in the paperSubstantial claim2 of 2 sources checked- cites [8] 8. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. Association for Computing Machinery, New York, NY, USA. source
- cites [48] 48. Melissa Roemmele and Andrew S. Gordon. 2018. Automated Assistance for Creative Writing with an RNN Language Model. In Proceedings of the 23rd In- ternational Conference on Intelligent User Interfaces Companion (Tokyo, Japan) (IUI '18 Companion). Association for Computing Machinery, New York, NY, USA, Article 21, 2 pages. source
The Experiment and Results section (page unavailable) reports 139 writers, 940 suggestion–modification pairs, and an edit-distance analysis, supporting fine-grained analysis across writers. Participants wrote stories on any topic, but all performed the same story-writing task; the paper does not establish coverage across different tasks or compare interaction logs with interviews. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
PremisecitationPre-trained language models are known to sometimes produce linguistically flawed, factually incorrect, or ethically problematic text.
“Pre-trained on massive amounts of text on the Internet, LMs are known to produce linguistically fawed, factually incorrect, or even ethically problematic text at times [1,24].”
Quote in the paperSubstantial claim2 of 2 sources checked- cites [1] 1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. arXiv:2101.05783 [cs.CL] source
The paper says language models are trained on large corpora including internet text (p. 2), and its Results (p. 5) document biased, violent GPT-3 completions. A figure caption (p. 7) also describes some generated descriptions as less coherent. Our copy does not establish the claim that the models produce factually incorrect text. (LLM, conf=0.88)
Experimental — accuracy not yet measured - cites [24] 24. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3356-3369. source
The Abstract and §3.1 report that pretrained language models sometimes generate toxic text, supporting the ethically problematic part of the claim. §8 (Related Work) also notes incoherence in model generations. The paper describes factually unreliable content in web-text corpora, but the passages do not establish that the models generate factually incorrect text. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
PremisecitationHow users adapt to systems over time matters to both HCI and NLP researchers.
“How users adapt to systems over time is of interest for not only HCI researchers but also NLP researchers [59,60].”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites [59] 59. Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a Programming Language via Interactive Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 929-938. source
The Introduction (page unavailable) studies how users’ language choices change as they interact with a natural-language system, supporting the NLP-research part of the claim. Our copy does not establish that this question is also of interest to HCI researchers. (LLM, conf=0.76)
Experimental — accuracy not yet measured - cites [60] 60. Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning Language Games through Interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 2368-2378. source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationGood writing collaboration can show varied interaction patterns, including differing levels of mutuality and equality.
“On an interaction level, good writing collaboration can exhibit diferent interaction patterns, such as diferent levels of mutuality and equality[2].”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [2] 2. Zsuzsanna I Abrams. 2019. Collaborative Writing and Text Quality in Google Docs. Language Learning & Technology 23, 2 (2019), 22-42. source
The Review of the Literature describes interaction patterns using differing levels of equality and mutuality, and Research Question 1 reports several such patterns among the writing groups (page numbers unavailable). The paper does not establish that all those patterns constitute *good* collaboration; it associates greater collaboration with some better writing outcomes. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationA human writing collaborator can improve fluency and the sense of audience in a text.
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites [6] 6. Joel Boch. 2007. Abdullah's blogging: A generation 1.5 student enters the blogo- sphere. Language Learning & Technology 11, 2 (2007), 128-141. source
The Introduction and Background Research (page unavailable) says that blogging can give students a greater sense of possible audiences and help them learn to respond to those audiences. The paper also describes interaction between readers and writers, but it does not establish that a human writing collaborator improves a text’s fluency. (LLM, conf=0.78)
Experimental — accuracy not yet measured - cites [53] 53. Neomy Storch. 2005. Collaborative writing: Product, process, and students' refections. Journal of second language writing 14, 3 (2005), 153-173. source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA human writing collaborator can better exploit socialization opportunities with discourse communities.
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites [65] 65. Luxin Yang. 2014. Examining the mediational means in collaborative writing: Case studies of undergraduate ESL students in business courses. Journal of Second Language Writing 23 (2014), 74-89. source
- cites [70] 70. Soobin Yim and Mark Warschauer. 2017. Web-based collaborative writing in L2 contexts: Methodological insights from text mining. Language Learning & Technology 21, 1 (2017), 146-165. source
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationHCI researchers can use CoAuthor to generate hypotheses through replay enactment.
“HCI researchers can use CoAuthor to formulate hypotheses via replay enactment [29].”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [29] 29. Kenneth Holstein, Erik Harpstead, Rebecca Gulotta, and Jodi Forlizzi. 2020. Replay Enactments: Exploring Possible Futures through Historical Data. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS '20). Association for Computing Machinery, New York, NY, USA, 1607-1618. https://doi.org/10.1145/3357236.3395427 source
The Introduction describes replay enactments as a way for designers to explore possible system behaviors, and the Conclusions say they help designers play out the implications of design decisions. Our copy does not mention CoAuthor or say that researchers use it to formulate hypotheses through this method (searched: CoAuthor, formulate hypotheses, hypothes). (LLM, conf=0.82)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataCoAuthor contains 20 writing prompts.
“CoAuthor contains 20 writing prompts, brief passages of text that provides a potential topic idea or starting point.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
Classified as probably not a substantial claim. May combine two claims.
Findingnone givenCoAuthor covers creative writing and argumentative writing.
“CoAuthor covers creative writing and argumentative writing, which have distinct goals and require diferent sets of writing skills.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe dataset and a replay interface are publicly available at https://coauthor.stanford.edu.
“The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
May combine two claims.
Findingown dataThe amount of collaboration between writers and GPT-3 varied significantly across writers, but depended less on writing prompts and the randomness of suggestions.
“we demonstrate that the amount of collaboration between writers and GPT-3 varied signifcantly across writers, but less depended on writing prompts and the randomness of suggestions.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
May combine two claims.
Findingown dataOn average, each writing session results in 418 words of text, contains 11.8 queries to the system, has an acceptance rate of 72.3%, and 72.6% of text is written by writers.
“On average, each writing session results in 418 words of text, contains 11.8 queries to the system, has acceptance rate of 72.3%, and 72.6% of text is written by writers.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
The quote only partly states this claim.
Findingown dataIn creative writing, 13% of accepted suggestions introduced new named entities.
“13% and 7% of accepted suggestions contained new named entities”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataCoAuthor provides a replay tool that gives designers a felt understanding of the interactions.
“We provide a tool for replaying all writing sessions in CoAuthor, giving designers a felt understanding of the interactions.”
- Also stated in: § Abstract · plain
“The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
- Also stated in: § Abstract · plain
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenCoAuthor was designed to be reusable and easily expandable.
“We designed CoAuthor to be reusable and easily expandable in the future.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
Classified as probably not a substantial claim.
Findingnone givenCoAuthor supports multiple analytical perspectives and goals by providing measurements in three categories: writing outcome, writer perception, and writer behavior.
“CoAuthor supports multiple analytical perspectives and goals by providing various measurements in three categories that can be used to defne good collaboration: writing outcome, writer perception, and writer behavior.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenCoAuthor consists of a set of writing sessions.
“CoAuthor consists of a set of writing sessions.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
May combine two claims.
Findingown dataThe dataset contains 830 stories written by 58 writers for creative writing and 615 essays written by 49 writers for argumentative writing.
“The dataset contains 830 stories written by 58 writers for creative writing and 615 essays written by 49 writers for argumentative writing.”
Quote in the paperNot a substantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
Classified as probably not a substantial claim.
Findingnone givenCoAuthor preserves details of rich interactions as events and event blocks.
“CoAuthor preserves details of rich interactions as events and event blocks.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
May combine two claims.
Findingown dataWe showed the approach is feasible and discussed insights for designers from the dataset.
“We demonstrated the feasibility of this approach and discussed insights designers can draw from the dataset.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn argumentative writing, high-randomness sessions averaged equality 0.29 ± 0.01 and mutuality 0.47 ± 0.01, while low-randomness sessions averaged 0.22 ± 0.01 and 0.38 ± 0.01.
“In argumentative writing, the sessions with GPT-3 with high randomness resulted in equality of 0.29 ± 0.01 and mutuality of 0.47 ± 0.01, whereas GPT-3 with low randomness resulted in 0.22 ± 0.01 and 0.38 ± 0.01 on average (Figure 9 (b)).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataSome writers collaborated more equally and mutually with the high-randomness GPT-3, while others did so with the low-randomness GPT-3.
“Also, some writers had more equal and mutual collaboration with GPT-3 with high randomness (dark blue), whereas others had such collaboration with GPT-3 with low randomness (light blue).”
Quote in the paperNot a substantial claim- Rests on an assumption: The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataA CoAuthor writing session averages 418 words in length.
“On average, each writing session is 418 words long, contains 11.8 queries to the system, has acceptance rate of 72.3% (how often writers accepted suggestions from GPT-3), and results in 72.6% of fnal texts written by writers (as opposed to GPT-3).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWriter #1 had equality and mutuality around 0.45 and 0.75, while Writer #2 had values below 0.1 and 0.2.
“for instance, Writer #1 scored around 0.45 and 0.75 for equality and mutuality, whereas Writer #2 scored below 0.1 and 0.2”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataOne story example shows a writer accepting a GPT-3 suggestion that introduced the named entity 'Dr.John' and then reusing that entity in later writing.
“Example of a story in which a writer accepted a suggestion from GPT-3 with a new named entity”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataEquality and mutuality vary across prompts in both creative and argumentative writing, though less than they vary across writers.
“Equality and mutuality (y-axis) and vary across prompts (x-axis) in creative (a) and argumentative writing (b), but to the less extent than across writers.”
Quote in the paperNot a substantial claim- Rests on an assumption: The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
May combine two claims.
Findingown dataAcross creative prompts, 5 of 10 had higher equality and 8 of 10 had higher mutuality with low randomness.
“When aggregated by prompts, 5 out of 10 prompts (50%) and 8 out of 10 prompts (80%) had higher equality and mutuality scores with GPT-3 with low randomness.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn creative writing, high-randomness sessions averaged equality 0.30 ± 0.01 and mutuality 0.48 ± 0.01, while low-randomness sessions averaged 0.29 ± 0.01 and 0.49 ± 0.01.
“On the other hand, in creative writing, writing sessions with GPT-3 with high randomness resulted in 0.30 ± 0.01 for equality and 0.48 ± 0.01 for mutuality, whereas the sessions with GPT-3 with low randomness resulted in 0.29 ± 0.01 and 0.49 ± 0.01, respectively (Figure 9 (a)).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn creative writing, 62.9% of writers used the system to advance plots and 51.4% used it to add story details.
“For example, in creative writing, some writers used the system to advance plots (62.9%), whereas some used it to add details to stories (51.4%) according to the survey's multiple choice question.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone givenThe paper identifies a need for holistic understanding of language models' generative capabilities for interaction design.
“First, it identifes a need for holistic understanding of LMs' generative capabilities for interaction design.”
Quote in the paperSubstantial claim- Rests on an assumption: GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
Classified as probably not a substantial claim. May combine two claims.
Premisenone givenLanguage models' generative capabilities are highly context-dependent, so covering all contexts is difficult.
“LMs' generative capabilities are highly context-dependent; therefore, it is difcult to cover all contexts.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
Premisenone givenLanguage models' generative capabilities can be subjectively interpreted, so evaluations can vary even within one interaction context.
“LMs' generative capabilities can be subjectively interpreted; therefore, they are susceptible to varying evaluations even within a given interaction context.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataGPT-3 is capable of providing new ideas to writers and influencing their subsequent writing.
“we present evidence that GPT-3 is capable of providing new ideas to writers, infuencing their subsequent writing.”
Quote in the paperSubstantial claim- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
Probably not a substantial claim: may not pass S2 (checkable).
PremisecitationAppropriate interaction design for a new technology requires a deep understanding of its capabilities and limitations.
“Appropriate interaction design for a new technology requires a deep understanding of its capabilities and limitations.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [9] 9. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.
Interpretationown dataThe lower distinct-2 score on all GPT-3 suggestions confirms previous findings that machine-generated text is less diverse than human-authored text.
“When we computed the distinct-2 score on all suggestions generated by GPT-3, the score was even lower (0.868 ± 0.001), further confrming the previous fndings.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [27] 27. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1689-1701. source
- Rests on an assumption: The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA human writing collaborator can broaden the pool of knowledge and ideas.
“For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level)[6,53], can expand the pool of knowledge and ideas (ideation and thinking)[19], can better harness the socialization opportunity with the discourse communities (socialization)[65,70], and more.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [19] 19. Richard Donato. 1994. Collective scafolding in a second language. Vygotskian approaches to second language research (1994), 33-56.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
InterpretationreasoningNLP datasets can help HCI researchers study language model capabilities in a holistic way.
“For example, datasets in NLP can provide resources to help HCI researchers reason about LMs' generative capabilities in a holistic manner.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenHCI researchers can supply analytical tools that help NLP researchers study language models in interactive settings.
“HCI researchers can provide analytical tools for NLP researchers to better investigate LMs's capabilities interactive settings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningCurating and analyzing large interaction datasets is a viable approach because it covers diverse interaction contexts and supports multiple interpretations of good collaboration.
“We argued that curating and analyzing large interaction datasets is one viable approach, as it can cover diverse interaction contexts and support various interpretations of good collaboration.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningDatasets can act as boundary objects linking HCI and NLP communities.
“We argue that datasets have the potential to serve as boundary objects between the HCI and NLP communities.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
InterpretationreasoningLarge HCI interaction datasets can carry human-centered values into technical work when used to train or evaluate language models.
“Moreover, large interaction datasets in HCI can embed human-centered values and practices, and further incorporate them into technical advances, when used to train or evaluate LMs.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningUsing datasets as boundary objects lets HCI and NLP communities share results and build common understanding of language model capabilities.
“By having datasets as boundary objects, the HCI and NLP communities can easily communicate results and have shared understanding of LMs' generative capabilities.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningReplaying CoAuthor sessions makes writer-GPT-3 interaction dynamics concrete and complex system behavior easier to grasp.
“Concretely, researchers can replay writing sessions in CoAuthor, which materializes the dynamics of interactions between writers and GPT-3 and makes complex system behavior more tangible.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe authors manually rated 201 submissions to qualify writers who met two requirements.
“The authors of this paper manually rated 201 submissions with the goal of qualifying writers who demonstrated that they could meet two requirements.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningCoAuthor can be a starting point for judging how plausible hypotheses are.
“CoAuthor can serve as a starting point to assess the plausibility of hypotheses.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
GPT-3's behavior in CoAuthor is representative enough of language models' generative capabilities for interactive writing to inform interaction design.
“Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing.”
“We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities”
The paper relies on this without support.
The recruited crowd workers are adequate proxies for writers in creative and argumentative writing.
“We recruited 63 crowd workers (writers) from Amazon Mechanical Turk to account for individuals with potentially diferent backgrounds and writing styles.”
“Among these writers, 63 writers participated in the main round, where 62 writers were native English speakers and one was not.”
The paper relies on this without support.
Most current language models are trained and evaluated on datasets that ignore interactive settings.
“Most current LMs are trained and evaluated on datasets that do not consider interactive settings.”
The paper relies on this without support.
High-randomness GPT-3 suggestions tend to be more diverse.
“as they tend to be more diverse.”
The paper relies on this without support.
The notions of equality and mutuality developed for human-human writing collaboration carry over to human-AI writing collaboration.
“On an interaction level, good writing collaboration can exhibit diferent interaction patterns, such as diferent levels of mutuality and equality[2].”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the General context, systems commonly provided suggestions for revisions which could be integrated directly into the writing area, aiding efficiency and idea exploration in the text.
“Systems commonly provided suggestions for revisions which could be integrated directly into the writing area [14,18,75], aiding efficiency and idea exploration in the text.”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [75]
Section 4.2.3 says accepted suggestions were automatically appended to the current text, and sections 5.1.2 and 5.2.1 discuss idea generation and potential productivity benefits. The paper describes suggestions for continuing writing, not specifically revisions to existing text, and its account of CoAuthor does not establish that systems commonly use this design. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe point of contact between the writer and the model shapes how much of the writing remains the writer's own.
“Studies of interactive writing systems note that the point of contact between the writer and the model (when suggestions appear, what the writer must do to take them, etc.) shapes how much of the writing remains the writer's own (Lee et al., 2022).”
From Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions? (2026) · cites this paper as Lee et al., 2022
Section 4.2.2 describes when suggestions appear and how writers accept them. Section 5.2.2 reports a correlation between the writer-authored fraction of the final text and writers’ feelings of ownership. The paper does not show that changing those interface choices shapes the writer-authored fraction; its observation about receiving more suggestions is tentative. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPrior research has found that people's perceptions of AI authorship in co-creation are nuanced and depend on the type of contribution made by the AI, how much each party contributed, and who led the co-creative effort.
“In contrast, prior research has found that people’s perceptions of AI authorship in co-creation are nuanced. These perceptions depend on various dimensions, including the type of contribution made by the AI, how much each party contributed, and who led the co-creative effort [33, 44, 71, 92].”
From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [44]
Section 5.2.2 reports a correlation between writers’ perceived ownership of GPT-3-assisted texts and the fraction of the text they wrote, supporting the claim that relative contributions matter. The paper does not establish that perceptions of AI authorship depend on the type of AI contribution or who led the co-creative effort. No page number is available in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredLanguage models are used in conversational interfaces, search and summarization, and collaborative writing.
“Language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing (Shuster et al., 2022;Thoppilan et al., 2022;Lee et al., 2022, inter alia).”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Lee et al., 2022
The less a user writes, the less ownership they feel over the final text.
“More recent work on AI co-writing for creative and argumentative texts has explicitly investigated issues of agency and ownership, finding that the less a user writes, the less ownership they feel over the final text [36].”
From Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems (2026) · cites this paper as [36]
GenAI has significantly impacted various aspects of daily life, including enhancing productivity, writing, interpersonal communication, and relationships.
“GenAI has signi cantly impacted various aspects of daily life, including enhancing productivity [68], writing [57, 58], interpersonal communication [35], and relationships [34, 76].”
From Creativity in the Age of AI: Evaluating the Impact of Generative AI on Design Outputs and Designers' Creative Thinking (2024) · cites this paper as [57]
A writer's sense of ownership may diminish as the proportion of system-generated content increases.
“In particular, a writer's sense of ownership may diminish as the proportion of system-generated content increases (Lee et al., 2022).”
From Ink and algorithm: Exploring temporal dynamics in generative <scp>AI</scp> ‐assisted writing (2026) · cites this paper as Lee et al., 2022
More commonly, systems display suggestions in a separated interface.
“More commonly, systems display suggestions in a separated interface [29,43,75,89].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [75]
Prior work has identified factors that affect people's sense of ownership over work produced with AI assistance, including types of contributions, amount of material contributed, process factors such as critical integration and control, and the role of AI in the writing process.
“Some of these factors regard characteristics of the contributions themselves, such as the types of contributions made [33, 71, 84, 92] and the amount of material contributed [44].”
From Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation (2025) · cites this paper as [44]
Students may be less inclined to disclose GenAI use, given widespread awareness of the difficulty humans face in accurately detecting AI-generated text and the unreliability of current detection tools.
“Furthermore, students may be less inclined to disclose GenAI use, given widespread awareness of the difficulty humans face in accurately detecting AI-generated text 33,34,35 and the unreliability of current detection tools 36,37 .”
From Style, sentiment, and quality of undergraduate writing in the AI era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports (2025) · cites this paper as [33]
The paper’s section 2.2.2 discusses the difficulty of evaluating language models’ contributions to writing, not whether people can identify AI-generated text. Searches for human detection, detection tools, student disclosure, and related terms found no passage addressing the claim. (LLM, conf=0.73)
Experimental — accuracy not yet measuredChat-based interfaces are highly intuitive, mimic conversational interactions between humans, but lack scaffolding for crafting prompts, which can be difficult for novice AI users.
“These interfaces are highly intuitive, and mimic conversational interactions between humans, but lack scaffolding for crafting prompts, which can be difficult for novice AI users [49,98].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [49]
Interaction logs and writing session replays give an objective behavioural representation of writer-suggestion interactions but do not reveal inner cognitive decision-making.
“While these approaches provide an objective representation of writer-suggestion interactions at a behavioural level, they lack insight into the inner cognitive decision-making processes that writers might engage in.”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [45]
LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.
“Lee et al. [54] find that LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.”
From Homogenization Effects of Large Language Models on Human Creative Ideation (2024) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Combined average length conflicts with the reported session average
MinorTable 3 reports a combined average of 366 words per session, while the text reports 418. The table's combined value is also lower than both category averages, 446 and 380 words.
“| | | Combined | 20 | | 63 | | | 1445 | 11.2 | 366 | 11.8 | 72.3 | 72.6 | |”
“On average, each writing session results in 418 words of text”
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 2× · see citing works
“The goal of Natural Language Generation (NLG) is to produce fluent text in many domains, such as machine translation [21], summarization [34], dialogue [30], style transfer [18], and programming code [13].”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2× · see citing works
“Any LM’s performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1× · see citing works
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25, 32, 47, 56].”
Background5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Self-citationA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 1,981 · see citing works
Gaver, W. W. (1991) · Proceedings of the SIGCHI conference on Human factors in computing systems Reaching through technology - CHI '91 · doi:10.1145/108844.108856 · Cited by 1,536 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Beyond English-Centric Multilingual Machine TranslationPreprint
Fan, A., Bhosale, S., Schwenk, H., Ma, Z.-Y., El-Kishky, A., Goyal, S., Baines, M., Çelebi, O., Wenzek, G., Chaudhary, V., et al. (2020) · Journal of machine learning research · doi:10.48550/arxiv.2010.11125 · Cited by 1,208 · see citing works
Patterns of Interaction in ESL Pair Work
Storch, N. (2002) · Language Learning · doi:10.1111/1467-9922.00179 · Cited by 1,080 · see citing works
Collaborative writing: Product, process, and students’ reflections
Storch, N. (2005) · Journal of Second Language Writing · doi:10.1016/j.jslw.2005.05.002 · Cited by 1,052 · cited 1× · see citing works
“2For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level) [6, 53], can expand the pool of knowledge and ideas (ideation and thinking) [19], can better harness the socialization opportunity with the discourse communities (sociali…”
BackgroundRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · cited 2× · see citing works
“Some have started leveraging of-the-shelf LMs for rapid prototyping of novel natural language interactions [64]; others have started crafting enduser-facing applications with fne-tuned LMs directly1[69], even though how soon such applications can become production-ready remain highly disputable [1, 24].”
BackgroundLim, Y.-K., Stolterman, E., & Tenenberg, J. (2008) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1375761.1375762 · Cited by 629 · cited 1× · see citing works
“By “tinkering with a scale” (observing how systems react to diferent user inputs) and potentially replaying many interactions, designers can develop a more holistic understanding of what the technology can and cannot do broadly [37, 50].”
BackgroundRe-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · see citing works
Self-citationPlug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508 · see citing works
Persistent Anti-Muslim Bias in Large Language Models
Abid, A., Farooqi, M., & Zou, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462624 · Cited by 476 · cited 2× · see citing works
“Pre-trained on massive amounts of text on the Internet, LMs are known to produce linguistically flawed, factually incorrect, or even ethically problematic text at times [1, 24].”
BackgroundA Simple Language Model for Task-Oriented DialoguePreprint
Hosseini-Asl, E., McCann, B., Wu, C.-S., Yavuz, S., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00796 · Cited by 368 · see citing works
Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification
Carney, M., Webster, B., Alvarado, I., Phillips, K., Howell, N., Griffith, J., Jongejan, J., Pitaru, A., & Chen, A. J. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382839 · Cited by 326 · see citing works
Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257 · cited 1× · see citing works
“The most traditional approach is contextual inquiry, inviting writers to write with an LM and interviewing them afterward [10, 14, 26, 63, 64, 66].”
BackgroundDynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
Investigating How Experienced UX Designers Effectively Work with Machine Learning
Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. M. (2018) · Designing Interactive Systems Conference · doi:10.1145/3196709.3196730 · Cited by 230 · cited 1× · see citing works
“In understanding advances in technologies, a diferent approach has emerged: developing interaction datasets and providing tools for data analyses [17, 51, 54, 61, 67].”
MethodSelf-citationBest practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · cited 12× · see citing works
“Then, we describe potential use cases of CoAuthor in HCI and NLP research, exemplifying the potential of CoAuthor serving as a boundary object.”
BackgroundInfluentialAbdullah's Blogging: A generation 1.5 student enters the blogosphere
Boch, J. (2007) · Language Learning & Technology · doi:10.64152/10125/44107 · Cited by 173 · cited 1× · see citing works
“2For example, a human writing collaborator can enhance the fuency and the sense of audience in the writing (contribution at a text production level) [6, 53], can expand the pool of knowledge and ideas (ideation and thinking) [19], can better harness the socialization opportunity with the discourse communities (sociali…”
BackgroundUnifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · cited 2× · see citing works
“Any LM’s performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
BackgroundSelf-citationThe GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · cited 12× · see citing works
“Then, we describe potential use cases of CoAuthor in HCI and NLP research, exemplifying the potential of CoAuthor serving as a boundary object.”
BackgroundInfluentialMetaphoria: An Algorithmic Companion for Metaphor Creation
Gero, K. I., & Chilton, L. B. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300526 · Cited by 146 · cited 1× · see citing works
“The most traditional approach is contextual inquiry, inviting writers to write with an LM and interviewing them afterward [10, 14, 26, 63, 64, 66].”
BackgroundLearning Language Games through Interaction
Wang, S. I., Liang, P., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1224 · Cited by 132 · see citing works
Self-citationCollaborative writing and text quality in Google Docs
Abrams, Z. I. (2019) · Language Learning & Technology · doi:10.64152/10125/44681 · Cited by 124 · see citing works
Buschek, D., Zürn, M., & Eiband, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445372 · Cited by 108 · cited 4× · see citing works
“email writing [8]), thereby making it hard to generalize to other tasks and confgurations of the same model.”
BackgroundInfluentialWeb-based collaborative writing in L2 contexts: Methodological insights from text mining
Yim, S., & Warschauer, M. (2017) · Language Learning & Technology · doi:10.64152/10125/44599 · Cited by 100 · see citing works
Teach Me to Explain: A Review of Datasets for Explainable Natural Language ProcessingPreprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12060 · Cited by 93 · see citing works
Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
Yang, Q., Cranshaw, J., Amershi, S., Iqbal, S. T., & Teevan, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300415 · Cited by 90 · cited 4× · see citing works
“The most traditional approach is contextual inquiry, inviting writers to write with an LM and interviewing them afterward [10, 14, 26, 63, 64, 66].”
BackgroundInfluentialSelf-citationDou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N. A., & Choi, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.01294 · Cited by 90 · see citing works
How to support designers in getting hold of the immaterial material of software
Ozenc, F. K., Kim, M., Zimmerman, J., Oney, S., & Myers, B. A. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753707 · Cited by 75 · see citing works
Yang, L. (2014) · Journal of Second Language Writing · doi:10.1016/j.jslw.2014.01.003 · Cited by 61 · cited 1× · see citing works
“2For example, a human writing collaborator can enhance the fluency and the sense of audience in the writing (contribution at a text production level) [6, 53], can expand the pool of knowledge and ideas (ideation and thinking) [19], can better harness the socialization opportunity with the discourse communities (social…”
BackgroundNaturalizing a Programming Language via Interactive Learning
Wang, S. I., Ginn, S., Liang, P., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1086 · Cited by 59 · see citing works
Self-citationSemmens, R., Martelaro, N., Kaveti, P., Stent, S., & Ju, W. G. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300867 · Cited by 57 · see citing works
Automated Assistance for Creative Writing with an RNN Language Model
Roemmele, M., & Gordon, A. S. (2018) · Companion Proceedings of the 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3180308.3180329 · Cited by 55 · cited 5× · see citing works
“Some existing work uses the amount of text written by the system or the number of edits made by writers to estimate the system’s “usefulness” or degree of contribution [15, 48].”
BackgroundMethodInfluentialTheodorou, L., Massiceti, D., Zintgraf, L., Stumpf, S., Morrison, C., Cutrell, E., Harris, M. T., & Hofmann, K. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471225 · Cited by 42 · cited 2× · see citing works
“[54] published a dataset of photos taken by blind and low vision users, for assessing how well diferent object recognition models can serve this user population.”
BackgroundMethodBirnholtz, J. P., Steinhardt, S. B., & Pavese, A. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466123 · Cited by 42 · cited 1× · see citing works
“[3] suggested that the quantity of collaboration (e.”
BackgroundWordcraft: a Human-AI Collaborative Editor for Story Writing
Coenen, A., Davis, L., Ippolito, D., Reif, E., & Yuan, A. (2021) · arXiv · doi:10.48550/arxiv.2107.07430 · Cited by 35 · see citing works
Design through matchmaking: technology in search of users
Bly, S., & Churchill, E. F. (1999) · Interactions · doi:10.1145/296165.296174 · Cited by 34 · see citing works
Wu, T., Terry, M., & Cai, C. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01691 · Cited by 26 · see citing works
Clark, E., & Smith, N. A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.279 · Cited by 23 · see citing works
An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
Wang, L., Mujib, M. I., Williams, J., Demiris, G., & Huh-Yoo, J. (2021) · arXiv · doi:10.48550/arxiv.2107.13115 · Cited by 21 · see citing works
Importance-Aware Learning for Neural Headline Editing
Wu, Q., Li, L., Zhou, H., Zeng, Y., & Yu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6467 · Cited by 20 · cited 1× · see citing works
“The most traditional approach is contextual inquiry, inviting writers to write with an LM and interviewing them afterward [10, 14, 26, 63, 64, 66].”
BackgroundReplay Enactments: Exploring Possible Futures through Historical Data
Holstein, K., Harpstead, E., Gulotta, R., & Forlizzi, J. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395427 · Cited by 19 · see citing works
On the Generation of Medical Dialogues for COVID-19Preprint
Yang, W., Zeng, G., Tan, B., Ju, Z., Chakravorty, S., He, X., Chen, S., Yang, X., Wu, Q., Yu, Z., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05442 · Cited by 18 · see citing works
Persistent Anti-Muslim Bias in Large Language ModelsPreprint
Abid, A., Farooqi, M., & Zou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05783 · Cited by 17 · see citing works
On the Generation of Medical Dialogues for COVID-19Preprint
Yang, W., Zeng, G., Tan, B., Ju, Z., Chakravorty, S., He, X., Chen, S., Yang, X., Wu, Q., Yu, Z., et al. (2020) · medRxiv · doi:10.1101/2020.05.08.20095810 · Cited by 15 · cited 1× · see citing works
“Some have started leveraging of-the-shelf LMs for rapid prototyping of novel natural language interactions [64]; others have started crafting enduser-facing applications with fne-tuned LMs directly1[69], even though how soon such applications can become production-ready remain highly disputable [1, 24].”
BackgroundLook at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors
Cuadra, A., Lee, H., Cho, J., & Ju, W. (2021) · arXiv · doi:10.48550/arxiv.2104.07153 · Cited by 5 · see citing works
Make This! Introduction to Electronics Prototyping Using Arduino
Martelaro, N., Zamfirescu-Pereria, J. D., Goedicke, D., et al. (2020) · doi:10.1145/3334480.3375052 · Cited by 3 · cited 1× · see citing works
“For example, Arduino made accessible the interaction design possibilities of sensors and motors [39]; Wekinator [22] and Teachable Machine [11] made accessible the otherwise abstract capabilities of supervised machine learning classifiers.”
BackgroundStanza: A Python Natural Language Processing Toolkit for Many Human Languages
Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.07082 · cited 2×
“• Engine: davinci • Response length (word piece): 30 • Temperature: 0.2, 0.3, 0.75, 0.9 • Top P: 1 • Frequency penalty: 0, 0.5, 1 • Presence penalty: 0 • Best of: 1 A.3 Suggestions Once GPT-3 generated an output given a prompt, it was parsed as a list of sentences using Stanza [44], and only the first sentence was use…”
MethodInfluential
Flagged first, then by citations
Research works printed but not linked: 9
- Joel Boch . 2007 . Abdullah’s blogging: A generation 1.5 student enters the blogosphere . Language Learning & Technology 11 , 2 (2007), 128 – 141 . Joel Boch. 2007. Abdullah’s blogging: A generation 1.5 student enters the blogosphere. Language Learning & Technology 11, 2 (2007), 128–141.
- Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates , Inc ., 1877 –1901. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901.
- Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann.
- Alex Calderwood , Vivian Qiu , Katy Ilonka Gero , and Lydia B Chilton . 2020. How Novelists Use Generative Language Models: An Exploratory User Study . In Proceedings of the Workshops on Human-AI Co-Creation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces . Association for Computing Machinery . Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Proceedings of the Workshops on Human-AI Co-Creation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces. Association for Computing Machinery.
- Richard Donato . 1994. Collective scaffolding in a second language. Vygotskian approaches to second language research ( 1994 ), 33–56. Richard Donato. 1994. Collective scaffolding in a second language. Vygotskian approaches to second language research (1994), 33–56.
- Rebecca Fiebrink and Perry R Cook . 2010 . The Wekinator: a system for real-time, interactive machine learning in music . In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010) (Utrecht). Rebecca Fiebrink and Perry R Cook. 2010. The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)(Utrecht).
- Dan Hendrycks , Collin Burns , Steven Basart , Andy Zou , Mantas Mazeika , Dawn Song , and Jacob Steinhardt . 2021 . Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021 , Virtual Event, Austria , May 3-7, 2021. OpenReview.net. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog 1, 8 ( 2019 ). Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog 1, 8 (2019).
- Donald A Schön. 1984. The reflective practitioner: How professionals think in action. Vol. 5126. Basic books. Donald A Schön. 1984. The reflective practitioner: How professionals think in action. Vol. 5126. Basic books.
Cited by
Sorted by citations
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · see citing works
Self-citationPerkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839 · see citing works
“…AI based technologies, there is ongoing research exploring how the intricacies of co-creation may be further be codified and explored in the fields of creative writing (M. Lee et al., 2022; Y. Lee et al., 2022), software development (Biderman & Raff, 2022) and artistic outputs (Oppenlaender, 2022).”
Alqahtani, T., Badreldin, H. A., Alrashed, M., Alshaya, A. I., Alghamdi, S. S., bin Saleh, K., Alowais, S. A., Alshaya, O. A., Rahman, I., Al Yami, M. S., et al. (2023) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2023.05.016 · Cited by 565 · see citing works
Marzuki, Widiati, U., Rusdin, D., Darwin, & Indrawati, I. (2023) · Cogent Education · doi:10.1080/2331186x.2023.2236469 · Cited by 524 · see citing works
“Works by Nazari et al. (2021), Dale and Viethen (2021), M. Lee et al. (2022), and Zhao (2022) served as valuable references.”
BackgroundSharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500 · see citing works
Impact of Artificial Intelligence on Dental Education: A Review and Guide for Curriculum Update
Thurzo, A., Strunga, M., Urban, R., Surovková, J., & Afrashtehfar, K. I. (2023) · Education Sciences · doi:10.3390/educsci13020150 · Cited by 345 · see citing works
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265 · see citing works
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · see citing works
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240 · see citing works
“Some studies consulted with expert linguists [31], university students in the sciences [42] or humanities [123], and amateur writers [2, 21, 62, 100, 125].”
BackgroundKar, A. K., Varsha, P. S., & Rajan, S. (2023) · Global Journal of Flexible Systems Management · doi:10.1007/s40171-023-00356-x · Cited by 209 · see citing works
“…technologies bring flexibility in firms to tackle the co-creation complexities which can be codified and located in the domains of creative writing (M Lee et al., 2022a, 2022b; Y Lee et al., 2022a, 2022b), information technology (Bider-man & Raff, 2022) and artistic outputs (Oppenlaender, 2022).”
BackgroundHelp Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 196 · see citing works
Draxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3637875 · Cited by 186 · see citing works
“The levels of control are derived from [29] and [70].”
BackgroundMethodInfluentialHomogenization Effects of Large Language Models on Human Creative Ideation
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656204 · Cited by 188 · see citing works
Jo, E., Epstein, D. A., Jung, H., & Kim, Y.-H. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581503 · Cited by 174 · see citing works
From explainable to interactive AI: A literature review on current trends in human-AI interaction
Raees, M., Meijerink, I., Lykourentzou, I., Khan, V.-J., & Papangelis, K. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103301 · Cited by 180 · see citing works
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 179 · see citing works
“Coauthor [50] provides a dataset of rich interactions between GPT-3 and writers.”
BackgroundEnabling Conversational Interaction with Mobile UI using Large Language Models
Wang, B., Li, G., & Li, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580895 · Cited by 156 · see citing works
“[25] present CoAuthor, a dataset designed to reveal GPT-3’s capabilities in assisting creative and argumentative writing.”
BackgroundOniani, D., Hilsman, J., Peng, Y., Poropatich, R. K., Pamplin, J. C., Legault, G. L., & Wang, Y. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00965-x · Cited by 143 · see citing works
Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671573 · Cited by 142 · see citing works
“…advances in large language models (LLMs) [5, 11, 44, 55, 60, 67, 68] have enabled a wide range of real-world applications across various domains, such as chatbots [43], writing assistants [24, 31], search engines (e.g., New Bing), tool/API retriever [46, 49] and multimodal systems [19, 23, 63].”
McGuire, J., De Cremer, D., & Van de Cruys, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-69423-2 · Cited by 150 · see citing works
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136 · see citing works
“Outputs that are coherent in terms of grammar, content, and tone”
Self-citationArt or Artifice? Large Language Models and the False Promise of Creativity
Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., & Wu, C.-S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642731 · Cited by 135 · see citing works
“In prior work, Baer [5] argued how divergent thinking remains the most frequently used indicator of creativity in both creativity research and educational practice, and divergent thinking theory has a strong hold on everyday conceptions of what it means to be creative.”
BackgroundSocial Dynamics of AI Support in Creative Writing
Gero, K. I., Long, T., & Chilton, L. B. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580782 · Cited by 129 · see citing works
“[22] looks at interactions with a GPT-3 autocomplete system across 63 writers.”
BackgroundInfluentialGenerative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking
Sharma, N., Liao, Q. V., & Xiao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642459 · Cited by 132 · see citing works
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 131 · see citing works
“, 2021) and show potential for creative writing tasks (Lee et al., 2022).”
BackgroundZhang, Z., Gao, J., Dhaliwal, R. S., & Li, T. J.-J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606800 · Cited by 108 · see citing works
“Similarly, CoAuthor [33] provides writers with context-based argumentative suggestions, inspiring them with the next piece of content to write.”
BackgroundMethodInfluentialOn the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113 · see citing works
“Indeed, we believe that LLMs can also foster human-AI co-creativity (Lee et al. 2022), since they can be used to write portions of stories to serve specific purposes, e.g., they can typify all the dialogues from a character, or they can provide more detailed descriptions of scenes (Calderwood et…”
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Cheng, M., Durmus, E., & Jurafsky, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.84 · Cited by 152 · see citing works
Han, A., Zhou, X., Cai, Z., Han, S., Ko, R., Corrigan, S., & Peppler, K. A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642438 · Cited by 114 · see citing works
Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models
Dhillon, P. S., Molaei, S., Li, J., Golub, M., Zheng, S., & Robert, L. P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642134 · Cited by 111 · see citing works
“Writing Prompts: We utilized 10 argumentative writing prompts originally selected by the CoAuthor study [41] from a set provided by The New York Times [48].”
BackgroundMethodResultInfluentialAI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
Agarwal, D., Naaman, M., & Vashistha, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11360 · Cited by 99
AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
Agarwal, D., Naaman, M., & Vashistha, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713564 · Cited by 132 · see citing works
“Beyond creative writing, HCI researchers have also proposed tools for argumentative writing, including jour-nalism [65], scientific writing [30], email writing [46], and crafting self-introductions for networking [41].”
Investigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87 · see citing works
“Given that trust is deeply embedded in contextual factors, the factors identified in prior empirical studies might change for AI-powered code-generation tools as system properties and the social and organizational contexts of usage shift.”
BackgroundBeyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries
Dang, H., Benharrak, K., Lehmann, F., & Buschek, D. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545672 · Cited by 85 · see citing works
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637364 · Cited by 84 · see citing works
Park, H., & Ahn, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642785 · Cited by 84 · see citing works
“Accordingly, recent HCI research has expanded LLMs’ application to tasks such as co-writing with LLMs [15, 32, 49] and providing health interventions for socially isolated or marginalized individuals [37, 90].”
BackgroundA systematic review of artificial intelligence technologies used for story writing
Fang, X., Ng, D. T. K., Leung, J. K. L., & Chu, S. K. W. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11741-5 · Cited by 82 · see citing works
“Similarly, Lee et al. (2022) mentioned that although AI was incorporated into story generators, they still have multiple aspects that need to be improved, such as language technicalities and story suggestions.”
BackgroundMethodInfluentialUnderstanding EFL students’ chatbot-assisted argumentative writing: An activity theory perspective
Guo, K., Li, Y., Li, Y., & Chu, S. K. W. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12230-5 · Cited by 83 · see citing works
“First, due to its human-like design features and interactivity, Argumate was perceived by the student users as a writing partner (Lee et al., 2022; Yuan et al., 2022) that provided support rather than as a mere tool that they used.”
Dang, H., Goller, S., Lehmann, F., & Buschek, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580969 · Cited by 83 · see citing works
“Single text suggestions can be shown inline [4, 8, 17, 48], whereas multiple suggestions are shown as popup lists of about three to six entries [6, 26].”
BackgroundResultInfluentialLiao, Q. V., Subramonyam, H., Wang, J., & Vaughan, J. W. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 97 · see citing works
“Similarly, to enable exploring GPT-3’s promises for interaction design, Lee et al. [65] created a dataset of instances from writers working with GPT-3.”
BackgroundQin, H. X., Jin, S., Gao, Z., Fan, M., & Hui, P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642105 · Cited by 82 · see citing works
The Value, Benefits, and Concerns of Generative AI-Powered Assistance in Writing
Li, Z., Liang, C., Peng, J., & Yin, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642625 · Cited by 81 · see citing works
“This assistance could also tailor to diverse writing tasks including stories [12, 13, 29, 41, 43, 45, 46], slogans [13], argumentative essays [29], emails [9, 19], and script writing [33].”
BackgroundCummings, R. E., Monroe, S. M., & Watkins, M. (2024) · Computers and Composition · doi:10.1016/j.compcom.2024.102827 · Cited by 83 · see citing works
Instruction Tuning for Large Language Models: A Survey
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wang, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3777411 · Cited by 640 · see citing works
“As depicted in Figure 1, it involves further training LLMs using (instruction, output) pairs, where instruction denotes the human instruction for the model, and output denotes the desired output that follows the instruction.”
Kim, T., Shin, D., Kim, Y.-H., & Hong, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642693 · Cited by 75 · see citing works
“While LLMs have been shown to enhance general writing abilities—by promoting varied vocabulary use and clarifying ambiguous ideas [10, 17, 29]—their application in the realm of personal journaling remains under-explored.”
InfluentialGenerative Image AI Using Design Sketches as input: Opportunities and Challenges
Zhang, C., Wang, W., Pangaro, P., Martelaro, N., & Byrne, D. (2023) · Creativity and Cognition · doi:10.1145/3591196.3596820 · Cited by 82 · see citing works
“Emerging research has demonstrated the potential of generative AI models in providing creative writing suggestions [5] as well as in generating advertisement posters [3], illustrative images for news [7], and graphical user interfaces[16].”
BackgroundBuilding machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 93 · see citing works
Self-citationHerding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3
Zamfirescu-Pereira, J. D., Wei, H., Xiao, A., Gu, K., Jung, G., Lee, M. G., Hartmann, B., & Yang, Q. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596138 · Cited by 70 · see citing works
“Our dialogue annotation and error analyses included: • Basic text-generation quality metrics, such as factualness [15]; • The extent to which the best-performing prompt can enable a GPT bot to exhibit the four interaction qualities listed in §3.1.2; • The severity of the interaction breakdowns,…”
MethodInfluentialSelf-citationThe Idea Machine: LLM-based Expansion, Rewriting, Combination, and Suggestion of Ideas
Fede, G. D., Rocchesso, D., Dow, S. P., & Andolina, S. (2022) · Creativity and Cognition · doi:10.1145/3527927.3535197 · Cited by 68 · see citing works
“With its design grounded in creativity and Human-Centered AI literature [15], the Idea Machine will enable the collection of valuable interaction data in LLM-based augmented ideation, similar to what has been done in other domains [11].”
ResultWriter-Defined AI Personas for On-Demand Feedback Generation
Benharrak, K., Zindulka, T., Lehmann, F., Heuer, H., & Buschek, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642406 · Cited by 63 · see citing works
“More recently, systems such asWordcraft [59] or CoAuthor [39] provide co-writing experiences with AI.”
BackgroundKim, J., Suh, S., Chilton, L. B., & Xia, H. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595996 · Cited by 61 · see citing works
“To evaluate mapping, fve criteria were used: fuency [11], scientifc precision, relatedness [27], willingness to adopt [45], and inspirational efect [24, 57].”
BackgroundMethodLiu, Y., Chen, S., Cheng, H., Yu, M., Ran, X., Mo, A., Tang, Y., & Huang, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642698 · Cited by 61 · see citing works
“Previous HCI studies suggest that while users might sometimes view unexpected outputs favorably [15], they can also find them unhelpful at times [35].”
Chakrabarty, T., Padmakumar, V., Brahman, F., & Muresan, S. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656201 · Cited by 61 · see citing works
Evolving Roles and Workflows of Creative Practitioners in the Age of Generative AI
Palani, S., & Ramos, G. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656190 · Cited by 61 · see citing works
Kobiella, C., Flores López, Y. S., Waltenberger, F., Draxler, F., & Schmidt, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641964 · Cited by 57 · see citing works
“GenAI tools have been used in the hope of bolstering the productivity of knowledge workers [9, 11, 26] and providing valuable support in problem-solving, idea generation, and conceptual development [20, 23, 53], key facets of knowledge-intensive careers.”
MethodLiu, J. Q. J., Hui, K. T. K., Al Zoubi, F., Zhou, Z. Z. X., Samartzis, D., Yu, C. C. H., Chang, J. R., & Wong, A. Y. L. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00155-6 · Cited by 56 · see citing works
A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
Gao, J., Gebreegziabher, S. A., Choo, K. T. W., Li, T. J.-J., Perrault, S. T., & Malone, T. W. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650786 · Cited by 57 · see citing works
“One example is the OpenAI Playground; similarly, literature such as CoAuthor leverages LLMs to ofer writing suggestions based on the user’s input text [40].”
A Redhead Walks into a Bar: Experiences of Writing Fiction with Artificial Intelligence
Ghajargar, M., Bardzell, J., & Lagerkvist, L. (2022) · Proceedings of the 25th International Academic Mindtrek Conference · doi:10.1145/3569219.3569418 · Cited by 54 · see citing works
“CoAuthor is a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities, which assists humans in writing creative and argumentative pieces [47].”
BackgroundReza, M., Laundry, N. M., Musabirov, I., Dushniku, P., Yu, Z. Y. “., Mittal, K., Grossman, T., Liut, M., Kuzminykh, A., & Williams, J. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641899 · Cited by 52 · see citing works
“There has been an explosion of recent work [4, 9, 14, 16, 26–28, 31, 47, 48, 50, 56, 67, 82, 84, 97] on interfaces for writing with LLMs in tandem with rapid advancements in AI writing capabilities using LLMs [8, 18, 66, 76, 91], examining various aspects of the AI-assisted writing process…”
InfluentialPang, R. Y., Schroeder, H., Smith, K. S., Barocas, S., Xiao, Z., Tseng, E., & Bragg, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713726 · Cited by 68 · see citing works
Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
Kumar, H., Vincentius, J., Jordan, E., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714198 · Cited by 51 · see citing works
“Lee et al. [37] introduced a dataset for analyzing GPT-3’s use in creative and argumentative writing, suggesting that the HCI community could foster more detailed examinations of LLMs’ generative capabilities through such datasets.”
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Wang, R., Ribeiro, A., Robinson, C., Loeb, S., & Demszky, D. (2024) · doi:10.21203/rs.3.rs-5363154/v1 · Cited by 53 · see citing works
Evaluating language models for mathematics through interactions
Collins, K. M., Jiang, A. Q., Frieder, S., Wong, L., Zilka, M., Bhatt, U., Lukasiewicz, T., Wu, Y.-H., Tenenbaum, J., Hart, W., et al. (2023) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2318124121 · Cited by 130 · see citing works
Hoque, M. N., Mashiat, T., Ghai, B., Shelton, C. D., Chevalier, F., Kraus, K., & Elmqvist, N. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641895 · Cited by 48 · see citing works
“Visualization has long been used to provide explainable machine learning [73] and NLP models [11, 45], and these ideas have also recently begun to be applied to LLMs. Recently, Jiang et al. [39] proposed Graphologue, that converts responses from LLMs to interactive graphs for fast and non-linear…”
BackgroundImran, M., Almusharraf, N., Abdellatif, M. S., & Abbasova, M. Y. (2024) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v18i18.49143 · Cited by 51 · see citing works
The Silicon Ceiling: Auditing GPT’s Race and Gender Biases in Hiring
Armstrong, L., Liu, A., MacNeil, S., & Metaxa, D. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694699 · Cited by 59 · see citing works
“Their promise of increased efficiency has led to numerous new LLM-powered systems for tasks including writing [38, 52, 63, 94], software engineering [28, 65], and design [30, 32, 43].”
Exploring EFL students’ prompt engineering in human–AI story writing: an activity theory perspective
Woo, D. J., Guo, K., & Susanto, H. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2361381 · Cited by 67 · see citing works
“Students with this theme most clearly high-light the potential of human – AI story writing to re fi ne and enhance story development (Lee et al., 2022).”
Bhat, A., Agashe, S., Oberoi, P., Mohile, N., Jangir, R., & Joshi, A. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584060 · Cited by 47 · see citing works
“It can act as an analytical lens for interpreting qualitative and quantitative data, generating hypotheses, and providing vocabulary to articulate complex interactions observed between writers and suggestion systems (For example, interaction logs datasets from tools like CoAuthor [45]).”
BackgroundMethodWoo, D. J., Wang, Y., Susanto, H., & Guo, K. (2023) · Journal of Educational Computing Research · doi:10.1177/07356331231175999 · Cited by 44 · see citing works
“, 2018) and that might increase writer productivity (Lee et al., 2022).”
BackgroundWorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI
Dang, H., Brudy, F., Fitzmaurice, G. W., & Anderson, F. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606772 · Cited by 49 · see citing works
“Moreover, several works have suggested to also logging users’ interactions [16, 17, 42, 48] to evaluate their behavior with co-creative AI systems.”
BackgroundHelp me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing
Chakrabarty, T., Padmakumar, V., & He, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.460 · Cited by 56 · see citing works
Feng, K. J. K., Li, T. W., & Zhang, A. X. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581273 · Cited by 44 · see citing works
FigurA11y: AI Assistance for Writing Scientific Alt Text
Singh, N., Wang, L. L., & Bragg, J. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645212 · Cited by 42 · see citing works
“Large language models (LLMs) have recently shown promise in providing contextual suggestions for diverse writing and revision tasks [8, 10, 23].”
BackgroundBhattacharjee, A., Zeng, Y.-C., Xu, S., Kulzhabayeva, D., Ma, M., Kornfield, R., Ahmed, S. I., Mariakakis, A., Czerwinski, M. P., Kuzminykh, A., et al. (2023) · International Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642081 · Cited by 59 · see citing works
“In writing-intensive courses, LLMs might function as intelligent writing assistants [33, 58, 67, 117]; they can provide more than mere copy-editing, but also real-time support in enhancing the structure and flow of arguments.”
BackgroundWe’re Afraid Language Models Aren’t Modeling Ambiguity
Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.51 · Cited by 63 · see citing works
Moon, K., Green, A. E., & Kushlev, K. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100207 · Cited by 38 · see citing works
Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Zhang, S., Xu, J., & Alvero, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251327130 · Cited by 48 · see citing works
RELIC: Investigating Large Language Model Responses using Self-Consistency
Cheng, F., Zouhar, V., Arora, S., Sachan, M., Strobelt, H., & El‐Assady, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641904 · Cited by 40 · see citing works
“Recent studies in human-LLM interaction have investigated various ways humans can collaborate with LLMs in accomplishing downstream tasks, like text summarization [5], creative writing [17, 23, 53, 54], and information seeking [20, 42].”
Background'It was 80% me, 20% AI': Seeking Authenticity in Co-Writing with Large Language Models
Hwang, A. H.-C., Liao, Q. V., Blodgett, S. L., Olteanu, A., & Trischler, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711020 · Cited by 42 · see citing works
Chen, L., Zuo, H., Cai, Z., Yin, Y., Zhang, Y. E., Sun, L., Childs, P., & Wang, B. (2024) · Journal of Mechanical Design · doi:10.1115/1.4065562 · Cited by 38 · see citing works
PANDALens: Towards AI-Assisted In-Context Writing on OHMD During Travels
Cai, R., Janaka, N., Chen, Y., Wang, L., Zhao, S., & Liu, C. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642320 · Cited by 36 · see citing works
“For subjective measures, self-rated Post-Editing Effort , quality of Language , Creativity & Appeal , and meeting preferred Writing Style for final writing (7-point Likert scale) [49] were collected.”
MetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Sun, L., Tao, S., Hu, J., & Dow, S. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637371 · Cited by 37 · see citing works
“CoAuthor further explored the capabilities of LLM working as a writing collaborator and reasoned about its ability to generate fluent text, generate new ideas and work jointly with writers [50].”
BackgroundFatima, W., Rizvi, S. S. R., Ghazal, T. M., Kharma, Q. M., Ahmad, M., Abbas, S., Furqan, M., & Khan, M. A. (2025) · IEEE Access · doi:10.1109/access.2025.3575610 · Cited by 35 · see citing works
“It preserves the concept and context of the original content, but may exclude nuances [16].”
Fan, H., Chen, G., Wang, X., & Peng, Z. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676390 · Cited by 38 · see citing works
Untangling Critical Interaction with AI in Students’ Written Assessment
Shibani, A., Knight, S., Kitto, K., Karunanayake, A., & Buckingham Shum, S. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3651083 · Cited by 35 · see citing works
Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creation
He, J., Houde, S., & Weisz, J. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713522 · Cited by 36 · see citing works
“Some of these factors regard characteristics of the contributions themselves, such as the types of contributions made [33, 71, 84, 92] and the amount of material contributed [44].”
InfluentialKnowNet:Guided Health Information Seeking from LLMs via Knowledge Graph Integration
Yan, Y., Hou, Y., Xiao, Y., Zhang, R., & Wang, Q. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456364 · Cited by 37 · see citing works
“Four authors, including two domain and two visualization experts, first collaboratively built a list of relevant papers that examined, applied, or improved LLMs for various information-seeking tasks, including but not limited to writing [22,57], coding [2,55], and healthcare [53].”
BackgroundAshkinaze, J., Mendelsohn, J., Qiwei, L., Budak, C., & Gilbert, E. (2025) · Proceedings of the ACM Collective Intelligence Conference · doi:10.1145/3715928.3737481 · Cited by 33 · see citing works
Guo, A., Pataranutaporn, P., & Maes, P. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650892 · Cited by 32 · see citing works
“Work has been done examining language models’ capabilities in creative and argumentative writing show GPT-3’s abilities and consider metrics for good collaboration including writ-ers’ feeling of ownership [20].”
Student-AI Question Cocreation for Enhancing Reading Comprehension
Liu, M., Zhang, J., Nyagoga, L. M., & Liu, L. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3333439 · Cited by 34 · see citing works
“Inspired by the recent success of human-AI cocreation in writing and music [23], [48], an automatic question generator was used to provide suggestions for students to reflect and create questions efficiently.”
MethodBeyond the Chat: Executable and Verifiable Text-Editing with LLMs
Laban, P., Vig, J., Hearst, M., Xiong, C., & Wu, C.-S. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676419 · Cited by 31 · see citing works
“Some work has framed text editing as an iterative process [22] and collected datasets of multi-turn interaction such as IteraTER [23] or Coauthor [57].”
BackgroundMethodFrom Pen to Prompt: How Creative Writers Integrate AI into their Writing Practice
Guo, A., Sathyanarayanan, S., Wang, L., Heer, J., & Zhang, A. X. (2025) · Creativity and Cognition · doi:10.1145/3698061.3726910 · Cited by 38 · see citing works
Song, I., Park, S., Pendse, S. R., Schleider, J. L., Choudhury, M. D., & Kim, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713883 · Cited by 42 · see citing works
“At can request an AI-generated summary of the current exploration e ○ .”
Multimodal large language models for inclusive collaboration learning tasks
Lewis, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop · doi:10.18653/v1/2022.naacl-srw.26 · Cited by 30 · see citing works
“Recent NLP research, for example, has explored neural models equipped with dynamic knowledge graph embeddings, the use of large language models to model real world speech, and the development of collaboration datasets (Ekstedt and Skantze, 2020; He et al., 2017; Lee et al., 2022), while cognitive science has explored…”
BackgroundPatchview: LLM-powered Worldbuilding with Generative Dust and Magnet Visualization
Chung, J. J. Y., & Kreminski, M. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676352 · Cited by 30 · see citing works
Yeo, S., Lim, G., Gao, J., Zhang, W., & Perrault, S. T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642530 · Cited by 35 · see citing works
“Research on LLM and Natural Language Processing (NLP) have shed light on their role as active partners throughout the writing process [85, 136].”
Background(2024) · doi:10.1145/3613904.3642939 · Cited by 31 · see citing works
Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process
Reza, M., Thomas-Mitchell, J., Dushniku, P., Laundry, N., Williams, J. J., & Kuzminykh, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757566 · Cited by 37 · see citing works
“AI also provides support with rhetorical or stylistic elements such as forming metaphors [37] or learning vocabulary [95].”
InfluentialArakawa, R., Yakura, H., & Goto, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581133 · Cited by 29 · see citing works
“recent studies that demonstrate how human writers and AI-based generation techniques can collaborate to support writers [23, 35, 60], such as in creative writing [57, 63, 65, 78, 82].”
BackgroundComparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
Fu, L., Newman, B., Jakesch, M., & Kreps, S. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581351 · Cited by 31 · see citing works
- Authors
- Lee, M., Liang, P., & Yang, Q.
- Affiliations
- Stanford University (US)
- Cornell University (US)
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-19
- OpenAlex
- W4220747294
- Cited by
- 345
- References
- 39
- Citations by year
- 2026: 61 · 2025: 96 · 2024: 118 · 2023: 56 · 2022: 14
- FWCI
- 56.81
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI collaboration, large language models, GPT-3, writing support, creative writing, argumentative writing, human-AI interaction, idea generation, language generation, interaction design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-30.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-30.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
