CoChat
Sign Up
Cited by 52Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIHuman-Automation Interaction and Safety

Artificial agents’ explainability to support trust: considerations on timing and context

Open access
Papagni, Guglielmo, de Pagter, Jesse, Zafari, Setareh, Filzmoser, Michael, Koeszegi, Sabine T.·2023·AI & SOCIETY
DOI 10.1007/s00146-022-01462-7
Open this paper in CoChat — free accountOpen PDF

Abstract

Strategies for improving the explainability of artificial agents are a key approach to support the understandability of artificial agents’ decision-making processes and their trustworthiness. However, since explanations are not inclined to standardization, finding solutions that fit the algorithmic-based decision-making processes of artificial agents poses a compelling challenge. This paper addresses the concept of trust in relation to complementary aspects that play a role in interpersonal and human–agent relationships, such as users’ confidence and their perception of artificial agents’ reliability. Particularly, this paper focuses on non-expert users’ perspectives, since users with little technical knowledge are likely to benefit the most from “post-hoc”, everyday explanations. Drawing upon the explainable AI and social sciences literature, this paper investigates how artificial agent’s explainability and trust are interrelated at different stages of an interaction. Specifically, the possibility of implementing explainability as a trust building, trust maintenance and restoration strategy is investigated. To this extent, the paper identifies and discusses the intrinsic limits and fundamental features of explanations, such as structural qualities and communication strategies. Accordingly, this paper contributes to the debate by providing recommendations on how to maximize the effectiveness of explanations for supporting non-expert users’ understanding and trust.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00146-022-01462-7

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 91

  1. Alipour K, Schulze JP, Yao Y, Ziskind A, Burachas G (2020) A study on multimodal and interactive explanations for visual question answering. arXiv preprint arXiv:200300431
  2. Andras P, Esterle L, Guckert M, Han TA, Lewis PR, Milanovic K, Payne T, Perret C, Pitt J, Powers ST, Urquhart N, Wells S (2018) Trusting intelligent machines: deepening trust within socio-tech- nical systems. IEEE Technol Soc Mag 37(4):76-83. https:// doi. org/ 10. 1109/ MTS. 2018. 28761 07 · DOI · Paper page
  3. Anjomshoae S, Najjar A, Calvaresi D, Främling K (2019) Explain- able Agents and Robots: Results from a Systematic Literature Review. In: Proceedings of the 18th InternationalConference on Autonomous Agents and MultiAgent Systems, pp 1078-1088
  4. Bellamy RK, Dey K, Hind M, Hoffman SC, Houde S, Kannan K, Lohia P, Martino J, Mehta S, Mojsilovic A, et al (2018) Ai fairness 360: an extensible toolkit for detecting, understanding, and mitigat- ing unwanted algorithmic bias. arXiv preprint arXiv:181001943
  5. Berland LK, Reiser BJ (2009) Making sense of argumentation and explanation. Sci Educ 93(1):26-55
  6. Cawsey A (1993) User modelling in interactive explanations. User Model User-Adap Inter 3(3):221-247
  7. Chen Y, Wu F, Shuai W, Wang N, Chen R, Chen X (2015) Kejia robot- an attractive shopping mall guider. In: International Conference on social robotics, Springer, pp 145-154
  8. Coeckelbergh M (2018) How to describe and evaluate "deception" phenomena: recasting the metaphysics, ethics, and politics of icts in terms of magic and performance and taking a relational and narrative turn. Ethics Inf Technol 20(2):71-85
  9. De Graaf MM, Malle BF (2017) How people explain action (and autonomous intelligent systems should too). In: 2017 AAAI Fall Symposium Series, pp 19-26
  10. de Graaf MM, Malle BF, Dragan A, Ziemke T (2018) Explainable robotic systems. In: Companion of the 2018 ACM/IEEE Inter- national Conference on Human-Robot Interaction, pp 387-388
  11. De Visser EJ, Peeters MM, Jung MF, Kohn S, Shaw TH, Pak R, Neer- incx MA (2020) Towards a theory of longitudinal trust calibra- tion in human-robot teams. Int J Soc Robot 12(2):459-478
  12. Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpret- able machine learning. arXiv preprint arXiv:170208608
  13. Dunne PE, Doutre S, Bench-Capon T (2005) Discovering inconsistency through examination dialogues. In: Proceedings of the 19th Inter- national Joint Conference on artificial intelligence, pp 1680-1681
  14. Elangovan A, Auer-Rizzi W, Szabo E (2007) Why don't I trust you now? An attributional approach to erosion of trust. Journal of Managerial Psychology, 22(1), 4-24
  15. Elia J (2009) Transparency rights, technology, and trust. Ethics Inf Technol 11(2):145-153
  16. Engle RA (1998) Not channels but composite signals: Speech, gesture, diagrams and object demonstrations are integrated in multimodal explanations. In: Proceedings of the twentieth annual conference of the cognitive science society, pp 321-326
  17. Ferreira JJ, Monteiro MdS (2020) Do ml experts discuss explainability for ai systems? a discussion case in the industry for a domain- specific solution. arXiv preprint arXiv:200212450
  18. Fulmer CA, Gelfand MJ (2012) At what level (and in whom) we trust: trust across multiple organizational levels. J Manag 38(4):1167-1230
  19. Gefen D (2000) E-commerce: the role of familiarity and trust. Omega 28(6):725-737
  20. Goodman B, Flaxman S (2017) European union regulations on algo- rithmic decision-making and a "right to explanation." AI Mag 38(3):50-57
  21. Gunning D, Aha DW (2019) Darpa's explainable artificial intelligence program. AI Mag 40(2):44-58
  22. Harman GH (1965) The inference to the best explanation. Philos Rev 74(1):88-95
  23. Hellström T, Bensch S (2018) Understandable robots -What, Why, and How. Paladyn J Behav Robot 9(1):110-123. https:// doi. org/ 10. 1515/ pjbr2 018-0009 · DOI
  24. Hilton DJ (1990) Conversational processes and causal explanation. Psychol Bull 107(1):65
  25. Hilton DJ, Slugoski BR (1986) Knowledge-based causal attribution: The abnormal conditions focus model. Psychol Rev 93(1):75
  26. Hilton DJ, McClure J, Sutton RM (2010) Selecting explanations from causal chains: Do statistical principles explain preferences for voluntary causes? Eur J Soc Psychol 40(3):383-400
  27. Hleg A (2019) Ethics guidelines for trustworthy AI, Publications Office. https:// data. europa. eu/ doi/ 10. 2759/ 346720 · DOI
  28. Ho N, Sadler GG, Hoffmann LC, Zemlicka K, Lyons J, Fergueson W, Richardson C, Cacanindin A, Cals S, Wilkins M (2017) A longitudinal field study of auto-gas acceptance and trust: First-year results and implications. J Cogn Eng Decis Making 11(3):239-251
  29. Hois J, Theofanou-Fuelbier D, Junk AJ (2019) How to Achieve Explainability and Transparency in Human AI Interaction. In: Stephanidis C (ed) HCI International 2019 -Posters, vol 1033, Springer International Publishing, Cham, pp 177-183, DOI https:// doi. org/ 10. 1007/ 978-3-030-23528-425 · DOI
  30. Holliday D, Wilson S, Stumpf S (2016) User trust in intelligent sys- tems: A journey over time. In: Proceedings of the 21st Inter- national Conference on intelligent user interfaces, pp 164-168
  31. Huk Park D, Anne Hendricks L, Akata Z, Rohrbach A, Schiele B, Dar- rell T, Rohrbach M (2018) Multimodal explanations: Justifying decisions and pointing to the evidence. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 8779-8788
  32. Im I, Hong S, Kang MS (2011) An international comparison of tech- nology adoption: Testing the Utaut model. Inf Manag 48(1):1-8
  33. Jacovi A, Marasović A, Miller T, Goldberg Y (2021) Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in ai. In: Proceedings of the 2021 ACM Conference on fair- ness, accountability, and transparency, pp 624-635
  34. Kaur H, Nori H, Jenkins S, Caruana R, Wallach H, Wortman Vaughan J (2020) Interpreting interpretability: Understanding data sci- entists' use of interpretability tools for machine learning. In: Proceedings of the 2020 CHI Conference on human factors in computing systems, pp 1-14
  35. Keil FC (2003) Folkscience: Coarse interpretations of a complex real- ity. Trends Cogn Sci 7(8):368-373
  36. Kulesza T, Stumpf S, Burnett M, Yang S, Kwan I, Wong WK (2013) Too much, too little, or just right? Ways explanations impact end users' mental models. In: 2013 IEEE Symposium on Visual Languages and Human Centric Computing, IEEE, pp 3-10
  37. Lakkaraju H, Bastani O (2020) "how do i fool you?" manipulating user trust via misleading black box explanations. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp 79-85
  38. Lamche B, Adıgüzel U, Wörndl W (2014) Interactive explanations in mobile shopping recommender systems. In: Joint Workshop on Interfaces and Human Decision Making in Recommender Sys- tems, vol 14
  39. Lankton NK, McKnight DH, Tripp J (2015) Technology, human- ness, and trust: Rethinking trust in technology. J Assoc Inf Syst 16(10):1
  40. Lee JD, See KA (2004) Trust in automation: designing for appropriate reliance. Hum Factors 46(1):50-80
  41. Leimstoll U, Stormer H (2007) Collaborative recommender systems for online shops. In: 13th Americas Conference on Information Systems, AMCIS 2007, Keystone, Colorado, USA, August 9-12, 2007
  42. Lewis D (1986) Causal Explanation. Philosophical Papers Vol ii, Oxford University Press, 214-240
  43. Li X, Hess TJ, Valacich JS (2008) Why do we trust new technology? A study of initial trust formation with organizational information systems. J Strateg Inf Syst 17(1):39-71
  44. Lipton ZC (2016) The mythos of model interpretability. arXiv:160603490 [cs, stat] 1606.03490
  45. Lipton ZC, Steinhardt J (2018) Troubling Trends in Machine Learning Scholarship. arXiv https:// arxiv. org/ abs/ 1807. 03341, 1807.03341
  46. Lockey S, Gillespie N, Holm D, Someh IA (2021) A review of trust in artificial intelligence: challenges, vulnerabilities and future directions. In: Proceedings of the 54th Hawaii International Con- ference on system sciences, pp 5463-5472
  47. Lomas M, Chevalier R, Cross EV, Garrett RC, Hoare J, Kopack M (2012) Explaining robot actions. In: Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction, pp 187-188
  48. Lombrozo T (2006) The structure and function of explanations. Trends Cogn Sci 10(10):464-470. https:// doi. org/ 10. 1016/j. tics. 2006. 08. 004 · DOI
  49. Lombrozo T (2007) Simplicity and probability in causal explanation. Cogn Psychol 55(3):232-257
  50. Luhmann N (2000) Familiarity, confidence, trust: problems and alter- natives. Trust Making Break Cooper Relat 6(1):94-107
  51. Luhmann N (2018) Trust and power. Wiley Lyon F, Möllering G, Saunders MN (2015) Introduction. Researching trust: the ongoing challenge of matching objectives and meth- ods. In: Handbook of research methods on trust, Edward Elgar Publishing
  52. Madumal P, Miller T, Vetere F, Sonenberg L (2018) Towards a grounded dialog model for explainable artificial intelligence. arXiv preprint arXiv:180608055
  53. Madumal P, Miller T, Sonenberg L, Vetere F (2019) A grounded inter- action protocol for explainable artificial intelligence. arXiv pre- print arXiv:190302409
  54. Malle BF, Knobe JM, Nelson SE (2007) Actor-observer asymmetries in explanations of behavior: New answers to an old question. J Pers Soc Psychol 93(4):491
  55. Matsuno F, Tadokoro S (2004) Rescue robots and systems in japan. In: 2004 IEEE International Conference on robotics and biomimet- ics, IEEE, pp 12-20
  56. Mohseni S, Zarei N, Ragan ED (2018) A survey of evaluation methods and measures for interpretable machine learning. arXiv:181111839 [cs] 1811.11839
  57. Mollering G (2006) Trust: reason, routine, reflexivity. Emerald Group Publishing
  58. Niemelä M, Heikkilä P, Lammi H (2017) A social service robot in a shopping mall: expectations of the management, retailers and consumers. In: Proceedings of the Companion of the 2017 ACM/ IEEE International Conference on human-robot interaction, pp 227-228
  59. O'Leary DE (2019) Google's duplex: pretending to be human. Intell Syst Account Finance Manag 26(1):46-53
  60. O'neill O (2002) Autonomy and trust in bioethics. Cambridge Univer- sity Press, Berlin
  61. Papagni G, Koeszegi S (2020) Understandable and trustworthy explain- able robots: A sensemaking perspective. Paladyn J Behav Robot 12(1):13-30
  62. Papagni G, Koeszegi S (2021) A pragmatic approach to the inten- tional stance semantic, empirical and ethical considerations for the design of artificial agents. Mind Mach 31(4):505-534
  63. Peirce CS (1997) Pragmatism as a principle and method of right think- ing: the 1903 Harvard lectures on pragmatism. SUNY Press
  64. Preece A, Harborne D, Braines D, Tomsett R, Chakraborty S (2018) Stakeholders in explainable ai. arXiv preprint arXiv:181000184
  65. Pu P, Chen L (2007) Trust-inspiring explanation interfaces for recom- mender systems. Knowl-Based Syst 20(6):542-556
  66. Quinn DB, Pak R, de Visser EJ (2017) Testing the efficacy of human- human trust repair strategies with machines. In: Proceedings of the Human Factors and Ergonomics Society Annual Meet- ing, SAGE Publications Sage CA: Los Angeles, CA, vol 61, pp 1794-1798
  67. Riedl MO (2019) Human-centered artificial intelligence and machine learning. Human Behav Emerg Technolos 1(1):33-36
  68. Rossi A, Dautenhahn K, Koay KL, Walters ML, Holthaus P (2020) Evaluating people's perceptions of trust in a robot in a repeated interactions study. In: International Conference on social robot- ics, Springer, pp 453-465
  69. Rotter JB (1971) Generalized expectancies for interpersonal trust. Am Psychol 26(5):443
  70. Rudin C (2018) Please stop explaining black box models for high stakes decisions. arXiv URL https:// arxiv. org/ abs/ 1811. 10154, 1811.10154
  71. Salem M, Lakatos G, Amirabdollahian F, Dautenhahn K (2015) Would you trust a (faulty) robot? effects of error, task type and per- sonality on human-robot cooperation and trust. In: 2015 10th ACM/IEEE International Conference on human-robot interaction (HRI), IEEE, pp 141-148
  72. Schoorman FD, Mayer RC, Davis JH (2007) An integrative model of organizational trust: past, present, and future. Acad Manag Rev 32(2):344-354
  73. Siau K, Wang W (2018) Building trust in artificial intelligence, machine learning, and robotics. Cutter Bus Technol J 31(2):47-53
  74. Sood K (2018) The ultimate black box: The thorny issue of program- ming moral standards in machines [industry view]. IEEE Technol Soc Mag 37(2):27-29
  75. Taddeo M, Floridi L (2011) The case for e-trust. Ethics Inf Technol 13(1):1-3
  76. Thagard P (1989) Explanatory coherence. Behav Brain Sci 12(3):435-502
  77. van Maris A, Lehmann H, Natale L, Grzyb B (2017) The influence of a robot's embodiment on trust: A longitudinal study. In: Proceed- ings of the Companion of the 2017 ACM/IEEE International Conference on human-robot interaction, pp 313-314
  78. Wagner AR, Borenstein J, Howard A (2018) Overtrust in the robotic age. Commun ACM 61(9):22-24
  79. Wagoner A, Jagadish A, Matson ET, EunSeop L, Nah Y, Tae KK, Lee DH, Joeng JE (2015) Humanoid robots rescuing humans and extinguishing fires for cooperative fire security system using harms. In: 2015 6th International Conference on automation, robotics and applications (ICARA), IEEE, pp 411-415
  80. Walton D (2011) A dialogue system specification for explanation. Syn- these 182(3):349-374
  81. Wang T (2019) Gaining free or low-cost interpretability with interpret- able partial substitute. In: International Conference on machine learning, PMLR, pp 6505-6514
  82. Wang T, Lin Q (2021) Hybrid predictive models: When an interpret- able model collaborates with a black-box model. J Mach Learn Res 22(137):1-38
  83. Weick KE, Sutcliffe KM, Obstfeld D (2005) Organizing and the process of sensemaking. Organ Sci 16(4):409-421
  84. Wiegand G, Schmidmaier M, Weber T, Liu Y, Hussmann H (2019) I drive-you trust: Explaining driving behavior of autonomous cars. In: Extended Abstracts of the 2019 Chi Conference on human factors in computing systems, pp 1-6
  85. Wilkenfeld DA, Lombrozo T (2015) Inference to the best explana- tion (ibe) versus explaining for the best inference (ebi). Sci Educ 24(9-10):1059-1077
  86. Woodcock C, Mittelstadt B, Busbridge D, Blank G et al (2021) The impact of explanations on layperson trust in artificial intelli- gence-driven symptom checker apps: experimental study. J Med Internet Res 23(11):e29386
  87. Zafari S, Koeszegi ST (2018) Machine agency in socio-technical sys- tems: a typology of autonomous artificial agents. In: 2018 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), IEEE, pp 125-130
  88. Zaheer A, McEvily B, Perrone V (1998) Does trust matter? Explor- ing the effects of interorganizational and interpersonal trust on performance. Organ Sci 9(2):141-159
  89. Zemla JC, Sloman S, Bechlivanidis C, Lagnado DA (2017) Evaluat- ing everyday explanations. Psychon Bull Rev 24(5):1488-1500
  90. Zou J, Schiebinger L (2018) AI can be sexist and racist-it's time to make it fair, 559(7714):324-326
  91. Zucker LG (1987) Institutional theories of organization. Ann Rev Sociol 13(1):443-464

Cited by

Sorted by citations

Authors
Papagni, G., de Pagter, J., Zafari, S., Filzmoser, M., & Koeszegi, S. T.
Affiliations
TU Wien (AT)
Corresponding author
Papagni, Guglielmo
ORCID
Venue
AI & SOCIETY
Publisher
Springer Science and Business Media LLC
Published
2022-06-27
Volume
38
Issue
2
Pages
947-960
ISSN
  • 0951-5666 (ISSN-L)
  • 1435-5655
DOI
10.1007/s00146-022-01462-7
OpenAlex
W4283588026
Cited by
52
References
11
Citations by year
2026: 22 · 2025: 17 · 2024: 7 · 2023: 6
FWCI
2.34
Type
journal article
Language
en
Open access
Licence
Locations
  • AI & Society (publishedVersion, cc-by)
  • reposiTUm (TU Wien) (submittedVersion, cc-by)
Funders
TU Wien
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
Keywords
trust in AI, explainability, non-expert users, user confidence, agent reliability, post-hoc explanation, trust building, trust maintenance, trust repair, human-agent interaction, communication strategies, transparency in decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review