The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers
Open accessThe DOI matches its Crossref record; 80 of the 81 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
We present an in-depth analysis of the impact of multi-word suggestion choices from a neural language model on user behaviour regarding input and text composition in email writing. Our study for the first time compares different numbers of parallel suggestions, and use by native and non-native English writers, to explore a trade-off of “efficiency vs ideation”, emerging from recent literature. We built a text editor prototype with a neural language model (GPT-2), refined in a prestudy with 30 people. In an online study (N=156), people composed emails in four conditions (0/1/3/6 parallel suggestions). Our results reveal (1) benefits for ideation, and costs for efficiency, when suggesting multiple phrases; (2) that non-native speakers benefit more from more suggestions; and (3) further insights into behaviour patterns. We discuss implications for research, the design of interactive suggestion systems, and the vision of supporting writers with AI instead of replacing them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Writing with suggestions correlates with shorter and more predictable texts being written [4], along with increased use of standard phrases when writing with a language model [17, 27].”
From Co-Writing with Opinionated Language Models Affects Users’ Views (2023) · cites this paper as [27]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Minimal AI assistance, such as simple word suggestions or basic grammatical corrections, may not offer substantial benefits to writers.This is particularly true for experienced writers who may already possess a rich vocabulary and a strong grasp of syntax.”
From Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models (2024) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Planning and translating support are intermingled due to the frequent use of longer AI outputs that directly ideate and write for the user, though these are generally presented as suggestions in order to preserve the user's engagement with the text [15,65].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, most previous work considered restricted interaction settings (e.g.strict turn-taking [15]) and adapted LMs and interfaces (e.g.fnd-tuned GPT-2 [58]) on specifc tasks (e.g.email writing [8]), thereby making it hard to generalize to other tasks and confgurations of the same model.”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“Although this approach may provide less rich insights than interviews, it can cover relatively diverse contexts across tasks and writers, while allowing for a fne-grained analysis of interactions.”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [8]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“[12]) have improved many benchmarks, including language models for understanding and generating text [12, 21].”
BackgroundlmerTest Package: Tests in Linear Mixed Effects Models
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017) · Journal of Statistical Software · doi:10.18637/jss.v082.i13 · Cited by 24,346
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · cited 10×
“Columns list the involved HMM states (cf. Figure 3), describe the behaviour, and outline how this interpretation was derived from the state parameters. the rest for evaluation (11 states scored best, considering log-likelihood and BIC [15]).”
BackgroundInfluentialKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Research works printed but not linked: 4
- [14] N. Carr. 2015. The Glass Cage: Where Automation is Taking Us . Random House, London, UK. -
- 10.18653/v1 · DOI
- [46] Eva Lindgren and Kirk Sullivan. 2019. Observing Writing . Brill, Leiden, Niederlande. https://doi.org/10.1163/9789004392526 - · DOI
- [57] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9. -
Cited by
Items 101–161 of 161
Lehmann, F., Kornecki, I., Buschek, D., & Feit, A. M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3604276 · Cited by 6
Self-citationChen, Y., & Liu, Z. (2024) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.1145/3690712.3690715 · Cited by 6
Zindulka, T., Goller, S., Lehmann, F., & Buschek, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713890 · Cited by 6
Self-citationLi, G., Teng, Y., Yue, J., Ahmed, U., Tantiwongse, A. S., Liang, J. Y., Zhang, D., Smith, K. S., Long, T., Lee, M., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.05287 · Cited by 6
Human-AI Interaction Patterns in Creative Domains and Their Time-Based Visualization
Leger, R., & Buschek, D. (2025) · Mensch und Computer (MuC) · doi:10.1145/3743049.3743081 · Cited by 5
Self-citationNLP Privacy Risk Identification in Social Media (NLP-PRISM): A Survey
Goswami, D., Kumar, J. K. N., & Das, S. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.78 · Cited by 5
Who Owns the Text? Design Patterns for Preserving Authorship in AI-Assisted WritingPreprint
Zhang, B., Bu, C., & Dhillon, P. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.10236 · Cited by 5
Mapping the Design Space of Human-AI Interaction in Text Summarization
Cheng, R., Smith-Renner, A., Zhang, K., Tetreault, J. R., & Jaimes, A. (2022) · arXiv · doi:10.48550/arxiv.2206.14863 · Cited by 4
“Writing with Model Assistance Generating & Editing Yes Human Auto-completion: (Green et al., 2014; Torregrosa et al., 2014; Gero and Chilton, 2019; Santy et al., 2019; Buschek et al., 2021; Ippolito et al., 2019; Calderwood et al., 2020; Bhat et al., 2021; Clark and Smith, 2021); Substitution dropdowns: (Green et al.”
BackgroundInfluentialReimagining Human-Thing Interactions Through Iterative Posthumanist Design Speculations
Cho, H. (2024) · Designing Interactive Systems Conference · doi:10.1145/3656156.3665125 · Cited by 4
What Happens When Reviewers Receive AI Feedback in Their Reviews?
Chen, S., Zhong, S., Brumby, D. P., & Cox, A. L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791431 · Cited by 4
From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants
Rismani, S., Blodgett, S. L., Liao, Q. V., Olteanu, A., & Moon, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791670 · Cited by 4
Semantic Prompting: Agentic Incremental Narrative Refinement through Spatial Semantic Interaction
Tang, X., Tahmid, I. A., Krokos, E., Whitley, K. N., Wang, X., & North, C. L. (2026) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3811427.3811444 · Cited by 4
Lehmann, F., & Buschek, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3733053 · Cited by 3
Self-citationMobile Typing with Intelligent Text Entry: A Large-Scale Dataset and ResultsPreprint
Leino, K., Laine, M., Kurimo, M., & Oulasvirta, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4654512/v1 · Cited by 3
LingoQ: Bridging the Gap between EFL Learning and Work through AI-Generated Work-Related Quizzes
Yang, Y., Lee, S. W., Song, J. Y., Yun, S., & Kim, Y.-H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791342 · Cited by 2
Wei, J., & Nishimoto, K. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791355 · Cited by 2
Framing an AI with Values Reduces AI Reliance in AI-supported Writing Tasks
Gao, A., Meltzoff, A. N., Sap, M., & Reinecke, K. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812369 · Cited by 2
Are Word Suggestions Beneficial? The Effect of Typing Efficiency and Suggestion Accuracy
Roy, Q., Casiez, G., & Vogel, D. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3772716 · Cited by 2
The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank
Ippolito, D., Dugan, L. D., Reif, E., Yuan, A., Coenen, A., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.185 · Cited by 2
Chen, J., Brandimarte, L., & Susarla, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4898166 · Cited by 2
Transforming the Reading Experience of Scientific Documents with Polymorphism
Masson, D. (2023) · UWSpace (University of Waterloo) · Cited by 2
Identity, Agency, and Human–AI Mediation in L2 Academic Writing: An Interactional Study
Aldamen, H., Almashour, M., & Jarrah, M. (2026) · Sage Open · doi:10.1177/21582440261463001 · Cited by 1
SnapSound: Empowering everyone to customize sound experience with Generative AI
Oh, J., & Jeong, J.-W. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103673 · Cited by 1
What smartphones change about writing: the impact of word suggestions on orthographic processing
Anastaseni, A., Roy, Q., Perret, C., Romano, A., & Kandel, S. (2025) · Cognitive Neuropsychology · doi:10.1080/02643294.2025.2604374 · Cited by 1
Augmenting Human Cognition Through Generative AI
Rogers, Y., Reicherts, L., Zhang, Z. T., & Hassib, M. (2026) · Springer eBooks · doi:10.1007/978-3-031-61050-9_10-1 · Cited by 1
Exploring Collaborative Writing Between Human and ChatGPT 3.5
Azhar, N. A. B., & Bunari, G. (2025) · LSP International Journal · doi:10.11113/lspi.v12.25999 · Cited by 1
Users Mispredict What Drives Their Preferences for AI Writing Assistance
Lai, V., Buçinca, Z., Kang, H. B., Houtti, M., Akpinar, N.-J., Chian, K., Suh, N., & Williams, A. C. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1622 · Cited by 1
Simulating Word Suggestion Usage in Mobile Typing to Guide Intelligent Text Entry Design
Li, Y., & Feit, A. M. (2026) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.48550/arxiv.2602.06489 · Cited by 1
Quantifying Co-writing with AI across Datasets from the HCI Community
Zindulka, T., Richter, S., & Buschek, D. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798711 · Cited by 1
Self-citationWriting with AI Can Reduce Gender Bias in Hiring Evaluations
Liu, A. T. H., Lee, M., & Bai, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791136 · Cited by 1
Exploring Email-Prompted Information Needs
Hsiao, J., & Bentley, F. (2021) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3479861 · Cited by 1
How different mental models of AI-based writing assistants impact writers’ interactions with them
Rismani, S., Blodgett, S. L., Olteanu, A., Liao, Q. V., & Moon, A. (2024) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.1145/3690712.3690722 · Cited by 1
Designing mixed-initiative video games
Yang, D. (2023) · Digital Repository Service (Northeastern University) · doi:10.17760/d20618670 · Cited by 1
UnWEIRDing LLM Entity RecommendationsPreprint
Kumar, A., & Mhatre, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.18403 · Cited by 1
Solipsistic Superintelligence is Unlikely to be CooperativePreprint
Trivedi, R., Jaques, N., Cross, L., Vezhnevets, A. S., & Leibo, J. Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03237 · Cited by 1
Designing Proactive Thought Partners for WritingPreprint
Zhang, C., Davis, A., Chen, C.-W., & Hsu, C.-C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.01588 · Cited by 1
Jiang, X., & Ashraf, M. A. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1003522
Some Determinations as to Whether or Not Academic Texts Are Produced by Artificial Intelligence
Uzun, C. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1007724
Designing AI-Based Language Tools for Non-Native Speakers' Language Use and Development
Xiao, Y. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3721094
The Effect of Response Suggestion on Dialog Flow: Analysis Based on Dialogue Act and Initiative
Cognitive Science Society 2023, Imai, M., Inoue, M., Maekawa, T., & Shibata, R. (2023) · eScholarship (California Digital Library) · doi:10.48448/568y-2b23
Heuer, H., Goller, S., & Buschek, D. (2026) · i-com · doi:10.1515/icom-2025-0024
Self-citationAI Based Collaborative Research Paper Management System
Subbarao, G. V., Durga, B. S. S., Lakshmi, D. B. S., Nagavalli, B. P., & Satyasri, M. V. (2026) · Journal of Emerging Technologies and Innovative Research · doi:10.56975/jetir.v13i3.577242
Ito, T. (2023) · Institutional Repositories DataBase (IRDB)
Singh, S., Baraka, K., Hindriks, K. V., & Heylen, D. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798698
Jang, Y., Lyu, Z., Arsenault, A. C., Kreps, S. E., & Yang, Q. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3813085
Nguyen, H. L. N. (2026) · University of Birmingham Institutional Research Archive (University of Birmingham)
Rani, I., & Zafar, A. (2026) · Dialogica · doi:10.5709/dialogica-02.s1.2026-11
BEYOND FAIRNESS METRICS: HUMAN EXPERIENCES AND DIFFERENTIAL OUTCOMES IN AI-MEDIATED WORK SETTINGS
Kadoma, K. (2026) · eCommons (Cornell University) · doi:10.7298/4h41-w092
Li, Z. "., Freeman, B., Raman, A., Rajkumar, K. A., Fang, X., Srivastava, R., Feiner, S., & Thakoor, K. A. (2026) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3830398.3830722
Towards Cognitive Process-Aware Proactive Writing SupportPreprint
Yoshida, M., Kobayashi, A., Tateno, K., & Chen, X. '. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30424
Apprentissage, évaluation et rééducation de l’écriture
Véron-Delor, L., Pinto, S., Eusebio, A., et al. (2025) · De Boeck Supérieur eBooks · doi:10.3917/dbu.conna.2025.01.0233
Anik, A. I. (2026) · Mspace (University of Manitoba)
Kalving, M., Väänänen, K., & Olsson, T. (2026) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3829807.3829865
Partial computational replication and preprocessing sensitivity in AI assisted email compositionPreprint
ni, W. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.23187195
Liang, Q. (., Adnin, R., & Das, M. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3829052
Rewriting the Self: User Perceptions of AI-Mediated Cognitive ReframingPreprint
Yang, Y.-W., Yessenbayev, O., Meter, A. V., Choudhury, T., & Adler, D. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.34647
Matsuyama, H., & Higuchi, K. (2026) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3830398.3830558
IntentWeaver: Sustaining Coherence across Evolving Intents in Human-AI Co-WritingPreprint
Zhang, Y., Cheng, X., Zhang, B., Zhang, X., Zhunis, A., Ma, S., & Ma, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.08214
Interpretive Proactivity: Expanding the Design Space of Proactive InteractionPreprint
Wang, P., & Kuang, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.09904
Who Shapes the Stance? A Stases-Based Analysis of Writer-LLM Negotiation with Coauthor
Estes, H., & Stolee, K. T. (2026) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc71707.2026.11772955
Singh, S., Hindriks, K. V., Heylen, D., & Baraka, K. (2026) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3841580.3841607
Metadata
- Authors
- Buschek, D., Zürn, M., & Eiband, M.
- Affiliations
- University of Bayreuth (DE)
- Ludwig-Maximilians-Universität München (DE)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-13
- OpenAlex
- W3123978464
- Cited by
- 108
- References
- 78
- Citations by year
- 2026: 13 · 2025: 28 · 2024: 26 · 2023: 21 · 2022: 14 · 2021: 6
- FWCI
- 12.52
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- ERef Bayreuth (University of Bayreuth) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Bavarian State Ministry of Science and the Arts
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- email writing, neural language models, GPT-2, idea generation, writing efficiency, native vs non-native writers, non-native English speakers, AI-assisted writing, suggestion systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.