CoChat

It's complicated: the relationship between user trust, model accuracy and explanations in AI

Open access
Papenmeier, Andrea, Kern, Dagmar, Englebienne, Gwenn, Seifert, Christin · 2022 · ACM Transactions on Computer-Human Interaction

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

Abstract

Automated decision-making systems become increasingly powerful due to higher model complexity. While powerful in prediction accuracy, Deep Learning models are black boxes by nature, preventing users from making informed judgments about the correctness and fairness of such an automated system. Explanations have been proposed as a general remedy to the black box problem. However, it remains unclear if effects of explanations on user trust generalise over varying accuracy levels. In an online user study with 959 participants, we examined the practical consequences of adding explanations for user trust: We evaluated trust for three explanation types on three classifiers of varying accuracy. We find that the influence of our explanations on trust differs depending on the classifier’s accuracy. Thus, the interplay between trust and explanations is more complex than previously reported. Our findings also reveal discrepancies between self-reported and behavioural trust, showing that the choice of trust measure impacts the results.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3495013

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Screenshot of the decision-support system used in our user study: We evaluated explanations for automated decision-making with regard to three explanation types. The user study is based on a scenario in which participants judge Tweets (top) with the help of a support system (middle) and indicate the offensiveness of the Tweet (buttons, bottom).
Fig. 1 Screenshot of the decision-support system used in our user study: We evaluated explanations for automated decision-making with regard to three explanation types. The user study is based on a scenario in which participants judge Tweets (top) with the help of a support system (middle) and indicate the offensiveness of the Tweet (buttons, bottom).CC-BY-NC-SA · From the PDF
Fig. 2 Screenshot of the graphical user interface for classification without the support system.
Fig. 2 Screenshot of the graphical user interface for classification without the support system.CC-BY-NC-SA · From the PDF
Fig. 3 Screenshot of the graphical user interface with the support system in condition C _ HF .
Fig. 3 Screenshot of the graphical user interface with the support system in condition C _ HF .CC-BY-NC-SA · From the PDF
Fig. 4 Overview of the experiment design.
Fig. 4 Overview of the experiment design.CC-BY-NC-SA · From the PDF
Image 5
CC-BY-NC-SA · From the PDF
Fig. 6 Likert plots of self-reported trust scores per condition on a 5-point scale ('1' indicating low trust, '5' indicating high trust). Aggregation of 19 items, values in the plot rounded to the next integer value.
Fig. 6 Likert plots of self-reported trust scores per condition on a 5-point scale ('1' indicating low trust, '5' indicating high trust). Aggregation of 19 items, values in the plot rounded to the next integer value.CC-BY-NC-SA · From the PDF
Image 7
CC-BY-NC-SA · From the PDF
Image 8
CC-BY-NC-SA · From the PDF
Image 9
CC-BY-NC-SA · From the PDF
Image 10
CC-BY-NC-SA · From the PDF
Fig. 9 Familiarity.
Fig. 9 Familiarity.CC-BY-NC-SA · From the PDF
Fig. 12 Trust in Automation.
Fig. 12 Trust in Automation.CC-BY-NC-SA · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 10

  1. P. S. Altares, A. R. I. Copo, Y. A. Gabuyo, A. T. Laddaran, L. D. P. Mejia, I. A. Policarpio, E. A. G. Sy, H. D. Tizon, and A. M. S. D. Yao. 2003. Elementary statistics: A modern approach. Rex Bookstore Inc. Manila, Philippines.
  2. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).
  3. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media. 512–515.
  4. 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
  5. Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In Proceedings of the 11th Australasian Conference on Information Systems. Citeseer, 6–8.
  6. Nazila Gol Mohammadi, Sachar Paulus, Mohamed Bishr, Andreas Metzger, Holger Könnecke, Sandro Hartenstein, Thorsten Weyer, and Klaus Pohl. 2013. Trustworthiness attributes and metrics for engineering trusted internet-based software systems. In Proceedings of the International Conference on Cloud Computing and Services Science. Springer, 19–35.
  7. Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 112–121.
  8. Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A proposal of user-centered explainable AI. In Proceedings of the IUI Workshops.
  9. Ariella Richardson and Avi Rosenfeld. 2018. A survey of interpretability and explainability in human-agent systems. In Proceedings of the XAI Workshop on Explainable Artificial Intelligence. 137–143.
  10. Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx. 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).

Cited by

See all 144

Metadata

Authors
Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C.
Affiliations
  • GESIS - Leibniz Institute for the Social Sciences (DE)
  • University of Twente (NL)
  • University of Duisburg-Essen (DE)
Venue
ACM Transactions on Computer-Human Interaction
Published
2022-03-31
Volume
29
Issue
4
Pages
1-33
ISSN
  • 1073-0516 (ISSN-L)
  • 1557-7325
OpenAlex
W4220693086
Cited by
144
References
46
Citations by year
2026: 43 · 2025: 42 · 2024: 33 · 2023: 18 · 2022: 8
FWCI
8.31
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Computer-Human Interaction (publishedVersion, cc-by-nc-sa)
  • University of Twente Research Information (publishedVersion, cc-by-nc-sa)
Funders
Cancer Research Center Cologne Essen
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Keywords
user trust, AI explanations, model accuracy, automated decision-making, black box problem, explainable AI, trust measurement, classification accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.