CoChat
Sign Up
Cited by 31Usability and User Interface DesignNatural Language Processing TechniquesPersonal Information Management and User Behavior

Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication

Open access
Fu, Liye, Newman, Benjamin, Jakesch, Maurice, Kreps, Sarah·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3581351
Open this paper in CoChat — free accountOpen PDF

Abstract

Traditionally, writing assistance systems have focused on short or even single-word suggestions. Recently, large language models like GPT-3 have made it possible to generate significantly longer natural-sounding suggestions, offering more advanced assistance opportunities. This study explores the trade-offs between sentence- vs. message-level suggestions for AI-mediated communication. We recruited 120 participants to act as staffers from legislators’ offices who often need to respond to large volumes of constituent concerns. Participants were asked to reply to emails with different types of assistance. The results show that participants receiving message-level suggestions responded faster and were more satisfied with the experience, as they mainly edited the suggested drafts. In addition, the texts they wrote were evaluated as more helpful by others. In comparison, participants receiving sentence-level assistance retained a higher sense of agency, but took longer for the task as they needed to plan the flow of their responses and decide when to use suggestions. Our findings have implications for designing task-appropriate communication assistance systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3544548.3581351

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 26

  1. Sinan Aral and Dean Eckles. 2019. Protecting elections from social media manip- ulation. Science 365, 6456 (2019), 858-861.
  2. Nikola Banovic, Ticha Sethapakdi, Yasasvi Hari, Anind K. Dey, and Jennifer Mankof. 2019. The Limits of Expert Text Entry Speed on Mobile Keyboards with Autocorrect. In Proceedings of the 21st International Conference on Human- Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, Article 15, 12 pages. https://doi.org/10.1145/3338286.3340126 · DOI
  3. Pablo Barberá, Andreu Casas, Jonathan Nagler, Patrick J Egan, Richard Bonneau, John T Jost, and Joshua A Tucker. 2019. Who leads? Who follows? Measuring issue attention and agenda setting by legislators and the mass public using social media data. American Political Science Review 113, 4 (2019), 883-901.
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
  5. Axel Bruns. 2019. Are flter bubbles real? John Wiley & Sons.
  6. Daniel Buschek, Benjamin Bisinger, and Florian Alt. 2018. ResearchIME: A Mobile Keyboard Application for Studying Free Typing Behaviour in the Wild. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA. https://doi.org/10.1145/3173574.3173829 event-place: Montreal, Quebec, CA. · DOI
  7. Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 732, 13 pages. https: //doi.org/10.1145/3411764.3445372 · DOI · Paper page
  8. Joshua Goodman, Gina Venolia, Keith Steury, and Chauncey Parker. 2002. Lan- guage modeling for soft keyboards. (Jan. 2002), 194-195. https://doi.org/10.1145/ 502716.502753 · DOI
  9. Christian R. Grose, Neil Malhotra, and Robert Parks Van Houweling. 2015. Explaining Explanations: How Legislators Explain their Pol- icy Positions and How Citizens React. American Journal of Politi- cal Science 59, 3 (2015), 724-743. https://doi.org/10.1111/ajps.12164 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/ajps.12164 · DOI
  10. Jefrey T. Hancock, Mor Naaman, and Karen Levy. 2020. AI-Mediated Commu- nication: Defnition, Research Agenda, and Ethical Considerations. Journal of Computer-Mediated Communication 25, 1 (March 2020). https://doi.org/10.1093/ jcmc/zmz022 · DOI · Paper page
  11. ALEXANDER HERTEL-FERNANDEZ, MATTO MILDENBERGER, and LEAH C. STOKES. 2019. Legislative Staf and Representation in Congress. American Politi- cal Science Review 113, 1 (2019), 1-18. https://doi.org/10.1017/S0003055418000606 · DOI
  12. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2019. Reducing sen- timent bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064 (2019). · arXiv
  13. Maurice Jakesch, Jefrey Hancock, and Mor Naaman. 2022. Human Heuristics for AI-Generated Language Are Flawed. arXiv preprint arXiv:2206.07271 (2022). · arXiv
  14. Christina James and Michael Longé. 2000. Bringing Text Input beyond the Desk- top. In CHI '00 Extended Abstracts on Human Factors in Computing Systems (The Hague, The Netherlands) (CHI EA '00). Association for Computing Machinery, New York, NY, USA, 49-50. https://doi.org/10.1145/633292.633324 · DOI
  15. Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016. Smart Reply: Automated Response Suggestion for Email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD '16). Association for Computing Machinery, New York, NY, USA, 955-964. https://doi.org/10.1145/2939672.2939801 · DOI
  16. Habibul Haque Khondker. 2011. Role of the new media in the Arab Spring. Globalizations 8, 5 (2011), 675-679.
  17. Sarah Kreps, R Miles McCain, and Miles Brundage. 2022. All the news that's ft to fabricate: AI-generated text as a tool of media misinformation. Journal of Experimental Political Science 9, 1 (2022), 104-117.
  18. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. https://doi.org/10.18653/v1/N16-1014 · DOI
  19. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 (2021). · arXiv
  20. Kseniia Palin, Anna Maria Feit, Sunjun Kim, Per Ola Kristensson, and Antti Oulasvirta. 2019. How do People Type on Mobile Devices? Observations from a Study with 37,000 Volunteers. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, 1-12. https: //doi.org/10.1145/3338286.3340120 · DOI
  21. Philip Quinn and Shumin Zhai. 2016. A Cost-Beneft Study of Text Entry Sugges- tion Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Com- puting Machinery, New York, NY, USA, 83-88. https://doi.org/10.1145/2858036. 2858305 · DOI
  22. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243 (2019). · arXiv
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
  24. Keith Vertanen, Haythem Memmi, Justin Emge, Shyam Reyal, and Per Ola Kristensson. 2015. VelociTap: Investigating Fast Mobile Text Entry using Sentence-Based Decoding of Touchscreen Keyboard Input. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). Association for Computing Machinery, Seoul, Republic of Korea, 659-668. https://doi.org/10.1145/2702123.2702135 · DOI
  25. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Grifn, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021). · arXiv
  26. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural Text Generation with Unlikelihood Training. ArXiv abs/1908.04319 (2020). · arXiv

Cited by

Sorted by citations

Authors
Fu, L., Newman, B., Jakesch, M., & Kreps, S.
Affiliations
  • Thomson Reuters (Canada) (CA)
  • Allen Institute for Artificial Intelligence (US)
  • Cornell University (US)
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-13
DOI
10.1145/3544548.3581351
OpenAlex
W4322716930
Cited by
31
References
17
Citations by year
2026: 6 · 2025: 13 · 2024: 10 · 2023: 2
FWCI
7.18
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Decision Sciences › Information Systems and Management › Personal Information Management and User Behavior
Keywords
AI-mediated communication, writing support, large language models, GPT-3, constituent correspondence, legislative offices, user agency, reaction time (psychology), user satisfaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review