To Err is AI: Imperfect Interventions and Repair in a Conversational Agent Facilitating Group Chat Discussions
Open accessThe DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Abstract
Conversational agents (CAs) can analyze online conversations using natural language techniques and effectively facilitate group discussions by sending supervisory messages. However, if a CA makes imperfect interventions, users may stop trusting the CA and discontinue using it. In this study, we demonstrate how inaccurate interventions of a CA and a conversational repair strategy can influence user acceptance of the CA, members' participation in the discussion, perceived discussion experience between the members, and group performance. We built a CA that encourages the participation of members with low contributions in an online chat discussion in which a small group (3-6 members) performs a decision-making task. Two types of errors can occur when detecting under-contributing members: 1) false-positive (FP) errors happen when the CA falsely identifies a member as under-contributing and 2) false-negative (FN) errors occur when the CA misses detecting an under-contributing member. We designed a conversational repair strategy that gives users a chance to contest the detection results and the agent sends a correctional message if an error is detected. Through an online study with 175 participants, we found that participants who received FN error messages reported higher acceptance of the CA and better discussion experience, but participated less compared to those who received FP error messages. The conversational repair strategy moderated the effect of errors such as improving the perceived discussion experience of participants who received FP error messages. Based on our findings, we offer design implications for which model should be selected by practitioners between high precision (i.e., fewer FP errors) and high recall (i.e., fewer FN errors) models depending on the desired effects. When frequent FP errors are expected, we suggest using the conversational repair strategy to improve the perceived discussion experience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, our findings are similar to Do et al. [16], which found that people who interacted with a conversational assistant that made factual errors were less likely to continue participating in group discussions.”
From The AI Double Standard: Humans Judge All AIs for the Actions of One (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mixed-Effects Models in Sand S-PLUS
Pinheiro, J. C., & Bates, D. M. (2000) · Statistics and Computing · doi:10.1007/978-1-4419-0318-1 · Cited by 11,187
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
The Preference for Self-Correction in the Organization of Repair in Conversation
Schegloff, E. A., Jefferson, G., & Sacks, H. (1977) · Language · doi:10.2307/413107 · Cited by 5,150
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 2×
“However, Dietvorst et al. argued that a system acknowledging mistakes can make errors stand out and people may lose trust and likability in algorithmic decisions (i.e., algorithmic aversion) [24].”
BackgroundMany Hands Make Light the Work: The Causes and Consequences of Social Loafing
Latané, B., Williams, K., & Harkins, S. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.6.822 · Cited by 2,216
Nass, C., Steuer, J., & Tauber, E. R. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191703 · Cited by 2,148 · cited 2×
“This perspective supports the Computers Are Social Actors (CASA) paradigm [65] where people respond in the same manner regardless of whether they are interacting with a human facilitator or a computer.”
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
Araujo, T. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.03.051 · Cited by 1,156
Nowak, K. L., & Biocca, F. (2003) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474603322761289 · Cited by 1,141
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Research works printed but not linked: 19
- Martin Adam, Michael Wessel, and Alexander Benlian. 2020. AI-based chatbots in customer service and their effects on user compliance. Electronic Markets (2020), 1-19.
- Saul Albert and Jan P de Ruiter. 2018. Repair: the interface between interaction and cognition. Topics in cognitive science 10, 2 (2018), 279-313.
- Dennis Benner, Edona Elshan, Sofia Schöbel, and Andreas Janson. 2021. What do you mean? a review on recovery strategies to overcome conversational breakdowns of conversational agents. In International Conference on Information Systems (ICIS).
- Anthony D Bergstrom. 2011. Social mirrors: visualization as conversation feedback. Ph.D. Dissertation. University of Illinois at Urbana-Champaign.
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77-101.
- The Concord Consortium. 2017. Teaching Teamwork. Retrieved Jan 22, 2021 from https://learn.concord.org/resources/ 565/teaching-teamwork-adder
- Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly (1989), 319-340.
- Gerardine DeSanctis and R Brent Gallupe. 1987. A foundation for the study of group decision support systems. Management science 33, 5 (1987), 589-609.
- James Fishkin, Nikhil Garg, Lodewijk Gelauff, Ashish Goel, Kamesh Munagala, Sukolsak Sakshuwong, Alice Siu, and Sravya Yandamuri. 2018. Deliberative democracy with the Online Deliberation platform. In The 7th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2019). HCOMP.
- Asbjørn Følstad and Cameron Taylor. 2019. Conversational repair in chatbots for customer service: the effect of expressing uncertainty and suggesting alternatives. In International Workshop on Chatbot Research and Design. Springer, 201-214.
- Taemie Kim, Agnes Chang, Lindsey Holland, and Alex Sandy Pentland. 2008. Meeting mediator: enhancing group collaborationusing sociometric feedback. In Proceedings of the 2008 ACM conference on Computer supported cooperative work. 457-466.
- Ben Lafreniere, Tanya R. Jonker, Stephanie Santosa, Mark Parent, Michael Glueck, Tovi Grossman, Hrvoje Benko, and Daniel Wigdor. 2021. False Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs on Input Error Preference. In The 34th Annual ACM Symposium on User Interface Software and Technology. 54-68.
- J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159-174.
- Christopher H Morrell. 1998. Likelihood ratio testing of variance components in the linear mixed-effects model using restricted maximum likelihood. Biometrics (1998), 1560-1568.
- Sherry L Piezon and William D Ferree. 2007. Perceptions of social loafing in online learning groups. In 23 rd Annual Conference on Distance Teaching & Learning, http://www. uwex. edu.
- Rachel Quednau. 2016. Top Strong Citizen Discussions this Week. Retrieved Jan 22, 2021 from https://www.strongtowns. org/journal/2016/4/13/top-strong-citizens-discussions?rq=slack
- Sandhya Saisubramanian, Shannon C Roberts, and Shlomo Zilberstein. 2021. Understanding User Attitudes Towards Negative Side Effects of AI Systems. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1-6.
- James A Shepperd. 1993. Productivity loss in performance groups: A motivation analysis. Psychological bulletin 113, 1 (1993), 67.
- Sam R Wilson, William C Barley, Luisa Ruge-Jones, and Marshall Scott Poole. 2020. Tacking Amid Tensions: Using Oscillation to Enable Creativity in Diverse Teams. The Journal of Applied Behavioral Science (2020), 0021886320960245.
Cited by
“As an AI language model, I cannot”: Investigating LLM Denials of User Requests
Wester, J., Schrills, T., Pohl, H., & van Berkel, N. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642135 · Cited by 36
Cai, Z., Park, S., Nixon, N., & Doroudi, S. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650868 · Cited by 56
Are We On Track? AI-Assisted Active and Passive Goal Reflection During Meetings
Chen, X., Tankelevitch, L., Vanukuru, R., Scott, A. E., Panda, P., & Rintel, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714052 · Cited by 52
Fok, R., Lipka, N., Sun, T., & Siu, A. F. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641969 · Cited by 37
Rayan, J. A., Kanetkar, D., Gong, Y., Yang, Y., Palani, S., Xia, H., & Dow, S. P. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656184 · Cited by 37
Metadata
- Authors
- Do, H. J., Kong, H.-K., Tetali, P., Lee, J., & Bailey, B. P.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Seattle University (US)
- University of Washington (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-04-14
- Volume
- 7
- Issue
- CSCW1
- Pages
- 1-23
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4366003147
- Cited by
- 32
- References
- 74
- Citations by year
- 2026: 12 · 2025: 12 · 2024: 8
- FWCI
- 9.47
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, conversational repair, false positives, false negative errors, user acceptance, group participation, decision-making groups, precision-recall trade-off, trust in AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.