Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.

“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions

Robertson, Ronald E, Olteanu, Alexandra, Diaz, Fernando, Shokouhi, Milad, Bailey, Peter·2021·Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Abstract

In email interfaces, providing users with reply suggestions may simplify or accelerate correspondence. While the “success” of such systems is typically quantified using the number of suggestions selected by users, this ignores the impact of social context, which can change how suggestions are perceived. To address this, we developed a mixed-methods framework involving qualitative interviews and crowdsourced experiments to characterize problematic email reply suggestions. Our interviews revealed issues with over-positive, dissonant, cultural, and gender-assuming replies, as well as contextual politeness. In our experiments, crowdworkers assessed email scenarios that we generated and systematically controlled, showing that contextual factors like social ties and the presence of salutations impacts users’ perceptions of email correspondence. These assessments created a novel dataset of human-authored corrections for problematic email replies. Our study highlights the social complexity of providing suggestions for email correspondence, raising issues that may apply to all social messaging systems.

Overview

Integrity signals

Registry

Crossref · 10.1145/3411764.3445557

Cites

103 of 103 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 103 references

Research works printed but not linked: 24

  1. 10.5465/amr.1999.2202131 · doi:10.5465/amr.1999.2202131 · Paper page
  2. Boomerang. 2020. Respondable: Write Better Email. https://www.boomeranggmail.com/respondable.
  3. Robin Campbell and Roger Wales. 1970. The Study of Language Acquisition. In New Horizons in Linguistics, J. Lyons (Ed.). Penguin, Harmondsworth, 242–260.
  4. Christina Cauterucci. 2015. New Chrome App Helps Women Stop Saying “Just” and “Sorry” in Emails. https://slate.com/human-interest/2015/12/new-chrome-app-helps-women-stop-saying-just-and-sorry-in-emails.html.
  5. Mia Xu Chen, Benjamin N. Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019. Gmail Smart Compose: Real-Time Assisted Writing. arXiv:1906.00080 [cs] (May 2019). arxiv:1906.00080 [cs] · arXiv
  6. Noam Chomsky. 1965. Aspects of the Theory of Syntax. Number 11 in Massachusetts Institute of Technology (Cambridge, Mass.). Research Laboratory of Electronics. Special Technical Report. MIT Press, Cambridge, MA. OCLC: 245876504.
  7. 10.1037/1076-8998.6.1.64 · doi:10.1037/1076-8998.6.1.64 · Paper page
  8. Budhaditya Deb, Peter Bailey, and Milad Shokouhi. 2019. Diversifying Reply Suggestions Using a Matching-Conditional Variational Autoencoder. arXiv:1903.10630 [cs, stat] (March 2019). arxiv:1903.10630 [cs, stat] · arXiv
  9. Christa Dürscheid, Carmen Frehner, Susan C Herring, Dieter Stein, and Tuija Virtanen. 2013. Email communication. Handbooks of Pragmatics [HOPS]9 (2013), 35–54.
  10. William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: The Uses and Impacts of Search in Britain, France, Germany, Italy, Poland, Spain, and the United States. SSRN Electronic Journal(2017).
  11. Anne Edstrom and Jennifer D. Ewald. 2017. ”Out of the Office”: Conveying Politeness through Auto-Reply Email Messages.Language@Internet 14, 4 (2017).
  12. Anne Edstrom and Jennifer D. Ewald. 2019. Characteristics of Effective Auto-Reply Emails: Politeness and Perceptions. Technology in Society (Jan. 2019), S0160791X1730218X.
  13. Brenden Gallagher. 2018. Gmail’s Smart Reply blurs the line between people and brands. https://www.dailydot.com/debug/thanks-for-letting-me-know/. [Online; accessed Sept-2019].
  14. Gary W. Giumetti, Andrea L. Hatfield, Jenna L. Scisco, Amber N. Schroeder, Eric R. Muth, and Robin M. Kowalski. 2013. What a Rude E-Mail! Examining the Differential Effects of Incivility versus Support on Mood, Energy, Engagement, and Performance in an Online Context.Journal of Occupational Health Psychology 18, 3 (July 2013), 297–309.
  15. Paul H. Grice. 1975. Logic and Conversation. In Syntax and Semantics: Speech Acts, Peter Cole and J. Morgan (Eds.). Vol. 3. Academic Press, New York, 41–58.
  16. Danielle N. Gunraj, April M. Drumm-Hewitt, Erica M. Dashow, Sri Siddhi N. Upadhyay, and Celia M. Klin. 2016. Texting Insincerely: The Role of the Period in Text Messaging. Computers in Human Behavior 55 (Feb. 2016), 1067–1075.
  17. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient Natural Language Response Suggestion for Smart Reply. arXiv:1705.00652 [cs] (May 2017). arxiv:1705.00652 [cs] · arXiv
  18. Susan C Herring. 1994. Politeness in computer culture: Why women thank and men flame. In Cultural Performances: Proceedings of the Third Berkeley Women and Language Conference. Berkeley Women and Language Group Berkeley, CA, 278–294.
  19. Kenneth J. Houghton, Sri Siddhi N. Upadhyay, and Celia M. Klin. 2018. Punctuation in Text Messages May Convey Abruptness. Period. Computers in Human Behavior 80 (March 2018), 112–121.
  20. Dell Hymes. 1972. On Communicative Competence. In Sociolinguistics: Selected Readings, J.B. Pride and J. Holmes (Eds.). Penguin, Harmondsworth, 269–293.
  21. Jon A. Krosnick and Stanley Presser. 2010. Question and Questionnaire Design. In Handbook of Survey Research, Second Edition (2 edition ed.), Peter V. Marsden and James D. Wright (Eds.). Emerald Publishing, Bingley, UK.
  22. Séamas O’Reilly. 2019. How Smart Are Gmail’s ‘Smart Replies’?The Observer (Feb. 2019).
  23. Erin L. Spottswood and Jeffrey T. Hancock. 2016. The Positivity Bias and Prosocial Deception on Facebook. Computers in Human Behavior 65 (Dec. 2016), 252–259.
  24. Jocelyne Vincent. 2008. Netiquette Rules OK!...OK?: speculating on rhetorical cleansing and English linguistic and cultural imperialism through email netiquette style guides. In Threads in the complex fabric of language: linguistic and literary studies in honour of Lavinia Merlini Barbaresi Lavinia Merlini Barbaresi Marcella Bertuccelli-Papi Antonio Bertacca and Silvia Bruti (Eds.). Felici San Giuliano Terme (Pisa) 409–443.

Cited by

See all 43

Metadata

Authors
Robertson, R. E., Olteanu, A., Diaz, F., Shokouhi, M., & Bailey, P.
Affiliations
  • Northeastern University (US)
  • Microsoft (United States) (US)
  • Microsoft Research, Canada
Venue
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2021-05-06
Pages
1-18
OpenAlex
W3134747254
Cited by
43
References
103
Citations by year
2026: 8 · 2025: 9 · 2024: 10 · 2023: 4 · 2022: 9 · 2021: 3
FWCI
4.85
Type
conference paper
Language
en
Open access
closed
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion)
Topics
  • Social Sciences › Linguistics and Language › Digital Communication and Language
  • Social Sciences › Decision Sciences › Information Systems and Management › Personal Information Management and User Behavior
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
social environment, politeness, gender assumptions, cultural differences, social ties, greetings, crowdsourced experiments, email correspondence, social messaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions. https://app.cochat.ai/papers/10.1145/3411764.3445557
The paper
Robertson, R. E., Olteanu, A., Diaz, F., Shokouhi, M., & Bailey, P. (2021). “I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445557

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.