CoChat
Sign Up
Cited by 83Social Robot Interaction and HRIHuman-Automation Interaction and SafetyMulti-Agent Systems and Negotiation

Trust repair in human-agent teams: the effectiveness of explanations and expressing regret

Open access
Kox, E. S., Kerstholt, J. H., Hueting, T. F., de Vries, P. W.·2021·Autonomous Agents and Multi-Agent Systems
DOI 10.1007/s10458-021-09515-9
Open this paper in CoChat — free accountOpen PDF

Abstract

The role of intelligent agents becomes more social as they are expected to act in direct interaction, involvement and/or interdependency with humans and other artificial entities, as in Human-Agent Teams (HAT). The highly interdependent and dynamic nature of teamwork demands correctly calibrated trust among team members. Trust violations are an inevitable aspect of the cycle of trust and since repairing damaged trust proves to be more difficult than building trust initially, effective trust repair strategies are needed to ensure durable and successful team performance. The aim of this study was to explore the effectiveness of different trust repair strategies from an intelligent agent by measuring the development of human trust and advice taking in a Human-Agent Teaming task. Data for this study were obtained using a task environment resembling a first-person shooter game. Participants carried out a mission in collaboration with their artificial team member. A trust violation was provoked when the agent failed to detect an approaching enemy. After this, the agent offered one of four trust repair strategies, composed of the apology components explanation and expression of regret (either one alone, both or neither). Our results indicated that expressing regret was crucial for effective trust repair. After trust declined due to the violation by the agent, trust only significantly recovered when an expression of regret was included in the apology. This effect was stronger when an explanation was added. In this context, the intelligent agent was the most effective in its attempt of rebuilding trust when it provided an apology that was both affective, and informational. Finally, the implications of our findings for the design and study of Human-Agent trust repair are discussed.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10458-021-09515-9

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Employing multiple repair strategies is more effective than relying on a single repair strategy.

    “Importantly, the comparison and combination of different repair strategies have gained attention as researchers have discovered that employing multiple repair strategies is more effective than relying on a single repair strategy [37, 77].”

    From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [37]

    The Discussion reports that combining an expression of regret with an explanation was the most effective approach tested for repairing trust after an agent’s error. These were components of an apology, however, not a general comparison showing that multiple repair strategies outperform a single strategy across settings. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Research shows the effectiveness of system-repair strategies is stronger when combined with other strategies as a multiple-repair strategy.

    “Research shows the effectiveness of system-repair strategies is stronger when combined with other strategies as a multiple-repair strategy [37, 77].”

    From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [37]

    The Results (§3.2 Trust) and Discussion (§4) report stronger trust recovery when the agent combined an explanation with an expression of regret than when it expressed regret alone. The paper tested those specific apology components in a human–agent team; it does not establish that system-repair strategies generally become more effective whenever they are combined with other strategies. (LLM, conf=0.81)

    Experimental — accuracy not yet measured

  • Some authors use correlations between perceived trustworthiness and system capabilities to measure calibration; larger correlation indicates better calibration.

    “Some authors work with correlations to get a better sense of perceived trustworthiness and system capabilities [5,12,24,39,66,67,81,94,130,139,142].Here the larger the correlation between perceived trustworthiness and system capabilities, the better the calibration.”

    From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [67]

    Section 3.3 (page unavailable) reports correlations between trust and participants’ perceptions that the agent was intelligent or useful. Sections 1 and 1.2 describe calibrated trust as trust warranted by the agent’s actual capabilities, but the reported correlations do not compare trust with actual capabilities or establish that a larger correlation means better calibration. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Kox et al. found trust repair strategies profited most when accompanied by an expression of regret, while including different levels of uncertainty had no effect.

    “Examining possible moderating variables, Kox et al. [67] found that trust repair strategies profted the most when accompanied by an expression of regret, while including diferent levels of uncertainty had no efect on the trust repair [67].”

    From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [67]

    Section 3.2 (page unavailable) reports that trust increased significantly after repair only when the agent expressed regret; the Discussion says regret plus an explanation was most effective. The study compared regret and explanation conditions, not different levels of uncertainty, so it does not support the claimed null effect of uncertainty. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Akgun, M., Cagiltay, K., & Zeyrek, D. (2010). The effect of apologetic error messages and mood 'states on computer users' self-appraisal of performance. Journal of Pragmatics, 42(9), 2430-2448. https:// doi. org/ 10. 1016/j. pragma. 2009. 12. 011 · DOI
  2. Barnes, M. J. et al. (2014). Designing for humans in autonomous systems: military applications. no. January, pp. 1-30.
  3. Brave, S., Nass, C., & Hutchinson, K. (2005). Computers that care: Investigating the effects of orienta- tion of emotion exhibited by an embodied computer agent. International Journal of Human-Computer Studies, 62(2), 161-178.
  4. Costa, A. C. (2003). Work team trust and effectiveness. Personnel Review, 32(5), 605-622. https:// doi. org/ 10. 1108/ 00483 48031 04883 60 · DOI
  5. Culley, K. E., & Madhavan, P. (2013). A note of caution regarding anthropomorphism in HCI agents. Computers in Human Behavior, 29(3), 577-579. https:// doi. org/ 10. 1016/j. chb. 2012. 11. 023 · DOI
  6. Ferguson, G., & Allen, J. (2011). A cognitive model for collaborative agents. In AAAI Fall Sympo- sium-Technical Report, vol. FS-11-01, pp. 112-120.
  7. Fine, G. A., & Holyfield, L. (2006). Secrecy, trust, and dangerous leisure: Generating group cohesion in voluntary organizations. Social Psychology Quarterly. https:// doi. org/ 10. 2307/ 27871 17 · DOI
  8. Freedy, A., DeVisser, E., Weltman, G., & Coeyman, N. (2007). Measurement of trust in human-robot collaboration. In Proceedings of the 2007 international symposium on collaboration technology sys- tem. CTS, pp. 106-114. https:// doi. org/ 10. 1109/ CTS. 2007. 46217 45 · DOI
  9. Gambetta, D. (2000). Can we trust trust? In Trust: Making and breaking cooperative relations, elec- tronic (pp. 212-237). Department of Sociology, University of Oxford.
  10. Haselhuhn, M. P., Schweitzer, M. E., & Wood, A. M. (2010). How implicit beliefs influence trust recovery. Psychological Science, 21(5), 645-648. https:// doi. org/ 10. 1177/ 09567 97610 367752 · DOI
  11. Hidalgo, C. A., Orghian, D., Albo-Canals, J., de Almeida, F., & Martin, N. (2021). How humans judge machines. Massachusetts Institute of Technology: The MIT Press Cambridge.
  12. Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407-434. https:// doi. org/ 10. 1177/ 00187 20814 547570 · DOI · Paper page
  13. Kim, P. H., Cooper, C. D., Ferrin, D. L., & Dirks, K. T. (2004). Removing the shadow of suspicion: The effects of apology versus denial for repairing competence-versus integrity-based trust violations. The Journal of Applied Psychology, 89(1), 104-118.
  14. Kim, P. H., Dirks, K. T., Cooper, C. D., & Ferrin, D. L. (2006). When more blame is better than less: The implications of internal vs. external attributions for the repair of trust after a competence-vs. integrity-based trust violation. Organizational Behavior and Human Decision Processes. https:// doi. org/ 10. 1016/j. obhdp. 2005. 07. 002 · DOI
  15. Lee, A. Y., Bond, G. D., Russell, D. C., Tost, J., González, C., & Scarbrough, P. S. (2010). Team perceived trustworthiness in a complex military peacekeeping simulation. Military Psychology, 22(3), 237-261. https:// doi. org/ 10. 1080/ 08995 605. 2010. 492676 · DOI
  16. Lee, J., & Moray, N. (1992). Trust, control strategies and allocation of function in human-machine systems. Ergonomics, 35(10), 1243-1270. https:// doi. org/ 10. 1080/ 00140 13920 89673 92 · DOI · Paper page
  17. Lee, J. E. R., & Nass, C. I. (2010). Trust in computers: The computers-are-social-actors (CASA) par- adigm and trustworthiness perception in human-computer communication, Trust Technol. a Ubiqui- tous Mod. Environ. Theor. Methodol. Perspect., pp. 1-15. https:// doi. org/ 10. 4018/ 978-1-61520-901-9. ch001. · DOI
  18. Lee, M. K., Kiesler, S., Forlizzi, J., Srinivasa, S., & Rybski, P. (2010). Gracefully mitigating break- downs in robotic services. In 2010 5th ACM/IEEE International Conference on Human-Robot Interac- tion, pp. 203-210. https:// doi. org/ 10. 1109/ HRI. 2010. 54531 95. · DOI
  19. Lewicki, R. J., Polin, B., & Lount, R. B. (2016). An exploration of the structure of effective apologies. Negotiation and Conflict Management Research, 9(2), 177-196. https:// doi. org/ 10. 1111/ ncmr. 12073 · DOI
  20. Loewenstein, G. F., Hsee, C. K., Weber, E. U., & Welch, N. (2001). Risk as feelings. Psychological Bulletin. https:// doi. org/ 10. 1037/ 0033-2909. 127.2. 267 · DOI · Paper page
  21. Lyons, J. B. (2013). Being transparent about transparency: A model for human-robot interaction. In AAAI Spring Symposium-Technical Report, vol. SS-13-07, pp. 48-53.
  22. Madhavan, P., & Wiegmann, D. A. (2007). Similarities and differences between human-human and human-automation trust: An integrative review. Theoretical Issues in Ergonomics Science, 8(4), 277- 301. https:// doi. org/ 10. 1080/ 14639 22050 03377 08 · DOI · Paper page
  23. Madhavan, P., Wiegmann, D. A., & Lacson, F. C. (2006). Automation failures on tasks easily per- formed by operators undermine trust in automated aids. Human Factors, 48(2), 241-256. https:// doi. org/ 10. 1518/ 00187 20067 77724 408 · DOI · Paper page
  24. Madsen, M., & Gregor, S. (2000). Measuring human-computer trust. In Proceedings of the Elev. Aus- tralas. Conf. Inf. Syst., pp. 6-8, 2000. http://books.google.com/books?hl=en&lr=&id=b0yalwi1HDM C&oi=fnd&pg=PA102&dq=The+Big+Five+Trait+Taxonomy:+History,+measurement,+and+The oretical+Perspectives&ots=758BNaTvOi&sig=L52e79TS6r0Fp2m6xQVESnGt8mw%5Cnhttp://cit- eseerx.ist.psu.edu/viewdoc/download?doi=.
  25. Matthews, G., Lin, J., Panganiban, A. R., & Long, M. D. (2019). Individual differences in trust in autonomous robots: Implications for transparency. IEEE Transactions on Human-Machine Systems, PP(November), 1-11. https:// doi. org/ 10. 1109/ THMS. 2019. 29475 92 · DOI
  26. McKnight, D. H., & Chervany, N. L. (2000) What is trust ? A conceptual analysis and an interdisci- plinary model. In Proceedings of the 2000 American Conference on Information System. AMCI2000 AIS Long Beach CA August 2000, vol. 346, p. 382. http:// aisel. aisnet. org/ cgi/ viewc ontent. cgi? artic le= 1876& amp; conte xt= amcis 2000.
  27. Melo, C. M. D., Gratch, J., & Carnevale, P. J. (2015). Humans versus computers: Impact of emotion expressions on people's decision making. IEEE Transactions on Affective Computing, 6(2), 127-136. https:// doi. org/ 10. 1109/ TAFFC. 2014. 23324 71 · DOI
  28. Melo, C. M. D., Zheng, L., & Gratch, J. (2009). Expression of moral emotions in cooperating agents. In Lecture Notes on Computer Science (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinfor- matics), vol. 5773 LNAI, pp. 301-307. https:// doi. org/ 10. 1007/ 978-3-642-04380-2_ 32. · DOI
  29. Merritt, S. M., Heimbaugh, H., Lachapell, J., & Lee, D. (2013). I trust it, but i don't know why: Effects of implicit attitudes toward automation on trust in an automated system. Human Factors, 55(3), 520- 534. https:// doi. org/ 10. 1177/ 00187 20812 465081 · DOI
  30. Merritt, S. M., Unnerstall, J. L., Lee, D., & Huber, K. (2015). Measuring individual differences in the perfect automation schema. Human Factors, 57(5), 740-753. https:// doi. org/ 10. 1177/ 00187 20815 581247 · DOI
  31. Muir, B. M. (1987). Trust between humans and machines, and the design of decision aids. Interna- tional Journal of Man-Machine Studies, 27(5-6), 527-539.
  32. Olshtain, E., & Cohen, A. (1983). Apology: A speech act set."Socioling. Lang. Acquis., pp. 18-35.
  33. Ososky, S. et al. (2012). The importance of shared mental models and shared situation awareness for transforming robots from tools to teammates. Unmanned Syst. Technol. XIV, vol. 8387, no. May 2012, p. 838710. https:// doi. org/ 10. 1117/ 12. 923283. · DOI
  34. Parasuraman, R., De Visser, E., Wiese, E., & Madhavan, P. (2014). Human trust in other humans, automation, robots, and cognitive agents: Neural correlates and design implications. In Proceedings of the Human Factors Ergon. Soc., vol. 2014-Janua, pp. 340-344. https:// doi. org/ 10. 1177/ 15419 31214 581070. · DOI
  35. Pop, V. L., Shrewsbury, A., & Durso, F. T. (2015). Individual differences in the calibration of trust in automation. Human Factors, 57(4), 545-556. https:// doi. org/ 10. 1177/ 00187 20814 564422 · DOI · Paper page
  36. Raue, M., D'Ambrosio, L. A., Ward, C., Lee, C., Jacquillat, C., & Coughlin, J. F. (2019). The influ- ence of feelings while driving regular cars on the perception and acceptance of self-driving cars. Risk Analysis, 39(2), 358-374. https:// doi. org/ 10. 1111/ risa. 13267 · DOI
  37. Riek, L. D., Rabinowitch, T.-C., Chakrabarti, B., & Robinson, P. (2009). How anthropomorphism affects empathy toward robots. In Proceedings of the 4th ACM/IEEE International Conference on Human and Robotics Interaction-HRI '09, p. 245. https:// doi. org/ 10. 1145/ 15140 95. 15141 58. · DOI
  38. Russell, S. J., & Norvig, P. (2003). Artificial intelligence: A modern approach (2nd ed.). Prentice Hall.
  39. Scher, S. J., & Darley, J. M. (1997). How effective are the things people say to apologize? Effects of the realization of the apology speech act. Journal of Psycholinguistic Research, 26(1), 127-140. https:// doi. org/ 10. 1023/A: 10250 68306 386 · DOI
  40. Sebo, S. S., Krishnamurthi, P., & Scassellati, B. (2019). 'I Don't Believe You': Investigating the effects of robot trust violation and repair. In ACM/IEEE international conference on human-robot interaction, vol. 2019-March, pp. 57-65. https:// doi. org/ 10. 1109/ HRI. 2019. 86731 69. · DOI · Paper page
  41. Serholt, S., & Barendregt, W. (2016). Robots tutoring children: Longitudinal evaluation of social engagement in child-robot interaction. In ACM international conference proceeding series, vol. 23-27- Octo, 2016. https:// doi. org/ 10. 1145/ 29714 85. 29715 36. · DOI · Paper page
  42. Teo, G., Wohleber, R., Lin, J., & Reinerman-jones, L. (2019). The relevance of theory to human-robot teaming research and development, vol. 784, no. January. https:// doi. org/ 10. 1007/ 978-3-319-94346-6. · DOI
  43. Tomlinson, E. C., Dineen, B. R., & Lewicki, R. J. (2004). The road to reconciliation: Antecedents of victim willingness to reconcile following a broken promise. Journal of Management. https:// doi. org/ 10. 1016/j. jm. 2003. 01. 003 · DOI
  44. Tzeng, J. Y. (2004). Toward a more civilized design: Studying the effects of computers that apologize. International Journal of Human-Computer Studies. https:// doi. org/ 10. 1016/j. ijhcs. 2004. 01. 002 · DOI
  45. Visser, E. J. D., Pak, R., & Shaw, T. H. (2018). From 'automation' to 'autonomy': The importance of trust repair in human-machine interaction. Ergonomics, 61(10), 1409-1427. https:// doi. org/ 10. 1080/ 00140 139. 2018. 14577 25 · DOI
  46. Visser, E. J. D., et al. (2019). Towards a theory of longitudinal trust calibration in human-robot teams. International Journal of Social Robotics. https:// doi. org/ 10. 1007/ s12369-019-00596-x · DOI · Paper page
  47. Vries, P. W. D., Van Den Berg, S. M., & Midden, C. (2015). Assessing technology in the absence of proof: Trust based on the interplay of others opinions and the interaction process. Human Factors, 57(8), 1378-1402. https:// doi. org/ 10. 1177/ 00187 20815 598604 · DOI
  48. Waytz, A., Cacioppo, J., & Epley, N. (2008). Who sees human? The stability and importance of individual differences in anthropomorphism. Bone, 23(1), 1-7. https:// doi. org/ 10. 1177/ 17456 91610 369336. Who · DOI
  49. Wright, J. L., Chen, J. Y. C., Barnes, M. J., & Hancock, P. A. (2016). The effect of agent reasoning transparency on automation bias: An analysis of response performance, vol. 9740, pp. 465-477.
  50. Xie, Y., & Peng, S. (2009). How to repair customer trust after negative publicity: The roles of compe- tence, integrity, benevolence, and forgiveness. Psychology and Marketing, 26(7), 572-589. https:// doi. org/ 10. 1002/ mar. 20289 · DOI

Cited by

Sorted by citations

Authors
Kox, E. S., Kerstholt, J. H., Hueting, T. F., & de Vries, P. W.
Affiliations
Netherlands Organisation for Applied Scientific Research (NL)
Corresponding author
Kox, E. S.
ORCID
Venue
Autonomous Agents and Multi-Agent Systems
Publisher
Springer Science and Business Media LLC
Published
2021-06-18
Volume
35
Issue
2
ISSN
  • 1387-2532 (ISSN-L)
  • 1573-7454
DOI
10.1007/s10458-021-09515-9
OpenAlex
W3176386948
Cited by
83
References
11
Citations by year
2026: 17 · 2025: 19 · 2024: 23 · 2023: 12 · 2022: 12
FWCI
11.51
Type
journal article
Language
en
Open access
Licence
Locations
  • Autonomous Agents and Multi-Agent Systems (publishedVersion, cc-by)
  • University of Twente Research Information (publishedVersion, cc-by)
  • University of Twente Research Information (publishedVersion, other-oa)
  • TNO Repository (submittedVersion, cc-by)
  • TNO Repository (submittedVersion)
  • University of Twente Research Information (submittedVersion, cc-by)
  • University of Twente Research Information (submittedVersion, other-oa)
Funders
Ministry of Defence
Topics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
trust repair, human-agent teams, breach of trust, apology, explanation, advice taking, human-agent collaboration, emotional communication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review