The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Abstract
In email interfaces, providing users with reply suggestions may simplify or accelerate correspondence. While the “success” of such systems is typically quantified using the number of suggestions selected by users, this ignores the impact of social context, which can change how suggestions are perceived. To address this, we developed a mixed-methods framework involving qualitative interviews and crowdsourced experiments to characterize problematic email reply suggestions. Our interviews revealed issues with over-positive, dissonant, cultural, and gender-assuming replies, as well as contextual politeness. In our experiments, crowdworkers assessed email scenarios that we generated and systematically controlled, showing that contextual factors like social ties and the presence of salutations impacts users’ perceptions of email correspondence. These assessments created a novel dataset of human-authored corrections for problematic email replies. Our study highlights the social complexity of providing suggestions for email correspondence, raising issues that may apply to all social messaging systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Indeed, various AIMC studies have shown that AI assistance (e.g., word-by-word suggestions or short-reply suggestions) can shift the characteristics of writing content, such as a more positive tone [22,48,59], or even affect topics addressed in writing [57].”
From 'It was 80% me, 20% AI': Seeking Authenticity in Co-Writing with Large Language Models (2025) · cites this paper as [59]
In the citing paper
“This is a surprising result given that previous AIMC work has found readers to hold negative opinions about messages and their writers once they become aware of the message being written with AI assistance [31,44,59].”
From 'It was 80% me, 20% AI': Seeking Authenticity in Co-Writing with Large Language Models (2025) · cites this paper as [59]
Citations
Cites
Politeness: Some Universals in Language Usage
Brown, P., Levinson, S. C., & Gumperz, J. J. (1987) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813085 · Cited by 10,438
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774
(2017) · Macat Library eBooks · doi:10.4324/9781912281732 · Cited by 4,703
Tit for Tat? The Spiraling Effect of Incivility in the Workplace
Andersson, L. M., & Pearson, C. M. (1999) · The Academy of Management Review · doi:10.2307/259136 · Cited by 3,256
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Course in General Linguistics: Ferdinand de Saussure
(2013) · Bloomsbury Publishing Plc eBooks · doi:10.5040/9781350284968 · Cited by 2,695
Crystal, D. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9781139164771 · Cited by 2,233
Preston, C. C., & Colman, A. M. (2000) · Acta Psychologica · doi:10.1016/s0001-6918(99)00050-5 · Cited by 2,106
Incivility in the workplace: Incidence and impact.
Cortina, L. M., Magley, V. J., Williams, J. H., & Langhout, R. D. (2001) · Journal of Occupational Health Psychology · doi:10.1037//1076-8998.6.1.64 · Cited by 2,084
Research works printed but not linked: 24
- 10.5465/amr.1999.2202131 · doi:10.5465/amr.1999.2202131 · Paper page
- Boomerang. 2020. Respondable: Write Better Email. https://www.boomeranggmail.com/respondable.
- Robin Campbell and Roger Wales. 1970. The Study of Language Acquisition. In New Horizons in Linguistics, J. Lyons (Ed.). Penguin, Harmondsworth, 242–260.
- Christina Cauterucci. 2015. New Chrome App Helps Women Stop Saying “Just” and “Sorry” in Emails. https://slate.com/human-interest/2015/12/new-chrome-app-helps-women-stop-saying-just-and-sorry-in-emails.html.
- Mia Xu Chen, Benjamin N. Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019. Gmail Smart Compose: Real-Time Assisted Writing. arXiv:1906.00080 [cs] (May 2019). arxiv:1906.00080 [cs] · arXiv
- Noam Chomsky. 1965. Aspects of the Theory of Syntax. Number 11 in Massachusetts Institute of Technology (Cambridge, Mass.). Research Laboratory of Electronics. Special Technical Report. MIT Press, Cambridge, MA. OCLC: 245876504.
- 10.1037/1076-8998.6.1.64 · doi:10.1037/1076-8998.6.1.64 · Paper page
- Budhaditya Deb, Peter Bailey, and Milad Shokouhi. 2019. Diversifying Reply Suggestions Using a Matching-Conditional Variational Autoencoder. arXiv:1903.10630 [cs, stat] (March 2019). arxiv:1903.10630 [cs, stat] · arXiv
- Christa Dürscheid, Carmen Frehner, Susan C Herring, Dieter Stein, and Tuija Virtanen. 2013. Email communication. Handbooks of Pragmatics [HOPS]9 (2013), 35–54.
- William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: The Uses and Impacts of Search in Britain, France, Germany, Italy, Poland, Spain, and the United States. SSRN Electronic Journal(2017).
- Anne Edstrom and Jennifer D. Ewald. 2017. ”Out of the Office”: Conveying Politeness through Auto-Reply Email Messages.Language@Internet 14, 4 (2017).
- Anne Edstrom and Jennifer D. Ewald. 2019. Characteristics of Effective Auto-Reply Emails: Politeness and Perceptions. Technology in Society (Jan. 2019), S0160791X1730218X.
- Brenden Gallagher. 2018. Gmail’s Smart Reply blurs the line between people and brands. https://www.dailydot.com/debug/thanks-for-letting-me-know/. [Online; accessed Sept-2019].
- Gary W. Giumetti, Andrea L. Hatfield, Jenna L. Scisco, Amber N. Schroeder, Eric R. Muth, and Robin M. Kowalski. 2013. What a Rude E-Mail! Examining the Differential Effects of Incivility versus Support on Mood, Energy, Engagement, and Performance in an Online Context.Journal of Occupational Health Psychology 18, 3 (July 2013), 297–309.
- Paul H. Grice. 1975. Logic and Conversation. In Syntax and Semantics: Speech Acts, Peter Cole and J. Morgan (Eds.). Vol. 3. Academic Press, New York, 41–58.
- Danielle N. Gunraj, April M. Drumm-Hewitt, Erica M. Dashow, Sri Siddhi N. Upadhyay, and Celia M. Klin. 2016. Texting Insincerely: The Role of the Period in Text Messaging. Computers in Human Behavior 55 (Feb. 2016), 1067–1075.
- Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient Natural Language Response Suggestion for Smart Reply. arXiv:1705.00652 [cs] (May 2017). arxiv:1705.00652 [cs] · arXiv
- Susan C Herring. 1994. Politeness in computer culture: Why women thank and men flame. In Cultural Performances: Proceedings of the Third Berkeley Women and Language Conference. Berkeley Women and Language Group Berkeley, CA, 278–294.
- Kenneth J. Houghton, Sri Siddhi N. Upadhyay, and Celia M. Klin. 2018. Punctuation in Text Messages May Convey Abruptness. Period. Computers in Human Behavior 80 (March 2018), 112–121.
- Dell Hymes. 1972. On Communicative Competence. In Sociolinguistics: Selected Readings, J.B. Pride and J. Holmes (Eds.). Penguin, Harmondsworth, 269–293.
- Jon A. Krosnick and Stanley Presser. 2010. Question and Questionnaire Design. In Handbook of Survey Research, Second Edition (2 edition ed.), Peter V. Marsden and James D. Wright (Eds.). Emerald Publishing, Bingley, UK.
- Séamas O’Reilly. 2019. How Smart Are Gmail’s ‘Smart Replies’?The Observer (Feb. 2019).
- Erin L. Spottswood and Jeffrey T. Hancock. 2016. The Positivity Bias and Prosocial Deception on Facebook. Computers in Human Behavior 65 (Dec. 2016), 252–259.
- Jocelyne Vincent. 2008. Netiquette Rules OK!...OK?: speculating on rhetorical cleansing and English linguistic and cultural imperialism through email netiquette style guides. In Threads in the complex fabric of language: linguistic and literary studies in honour of Lavinia Merlini Barbaresi Lavinia Merlini Barbaresi Marcella Bertuccelli-Papi Antonio Bertacca and Silvia Bruti (Eds.). Felici San Giuliano Terme (Pisa) 409–443.
Cited by
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136
Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511599 · Cited by 119
Will AI Console Me when I Lose my Pet? Understanding Perceptions of AI-Mediated Email Writing
Liu, Y., Mittal, A., Yang, D., & Bruckman, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517731 · Cited by 92
'It was 80% me, 20% AI': Seeking Authenticity in Co-Writing with Large Language Models
Hwang, A. H.-C., Liao, Q. V., Blodgett, S. L., Olteanu, A., & Trischler, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711020 · Cited by 42
Self-citationCo-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process
Reza, M., Thomas-Mitchell, J., Dushniku, P., Laundry, N., Williams, J. J., & Kuzminykh, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757566 · Cited by 37
“Clark’s [24] work examining user experiences with AI writing prototypes revealed that while participants found AI collaboration satisfying, the resulting text quality did not surpass that of unaided human writers.”
Metadata
- Authors
- Robertson, R. E., Olteanu, A., Diaz, F., Shokouhi, M., & Bailey, P.
- Affiliations
- Northeastern University (US)
- Microsoft (United States) (US)
- Microsoft Research, Canada
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-18
- OpenAlex
- W3134747254
- Cited by
- 43
- References
- 103
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 10 · 2023: 4 · 2022: 9 · 2021: 3
- FWCI
- 4.85
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Linguistics and Language › Digital Communication and Language
- Social Sciences › Decision Sciences › Information Systems and Management › Personal Information Management and User Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- social environment, politeness, gender assumptions, cultural differences, social ties, greetings, crowdsourced experiments, email correspondence, social messaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions. https://app.cochat.ai/papers/10.1145/3411764.3445557
- The paper
- Robertson, R. E., Olteanu, A., Diaz, F., Shokouhi, M., & Bailey, P. (2021). “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445557
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.