It's complicated: the relationship between user trust, model accuracy and explanations in AI
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Automated decision-making systems become increasingly powerful due to higher model complexity. While powerful in prediction accuracy, Deep Learning models are black boxes by nature, preventing users from making informed judgments about the correctness and fairness of such an automated system. Explanations have been proposed as a general remedy to the black box problem. However, it remains unclear if effects of explanations on user trust generalise over varying accuracy levels. In an online user study with 959 participants, we examined the practical consequences of adding explanations for user trust: We evaluated trust for three explanation types on three classifiers of varying accuracy. We find that the influence of our explanations on trust differs depending on the classifier’s accuracy. Thus, the interplay between trust and explanations is more complex than previously reported. Our findings also reveal discrepancies between self-reported and behavioural trust, showing that the choice of trust measure impacts the results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, research has shown that subjective trust can be a poor predictor of actual reliance (Dzindolet et al., 2003; Miller et al., 2016; Papenmeier et al., 2022).”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Papenmeier et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“The outcome of questionnaires is influenced by the participants' ability to reflect their attitude towards a system, which can be difficult for some participants (Papenmeier et al., 2022).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Papenmeier et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Across multiple studies in our overview a discrepancy between self-report and behavioural measures is observed (e.g., Papenmeier et al., 2022Papenmeier et al., , 2021;;Wang & Yin, 2023).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Papenmeier et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Rempel, J. K., Holmes, J. G., & Zanna, M. P. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.49.1.95 · Cited by 3,146 · cited 2×
“Moreover, trust builds up over time [42] and shows phases of development and refinement [54].”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“[15] therefore suggest to account for the limited human attention span and cognitive abilities when designing explanations, e.”
BackgroundStraightforward Statistics for the Behavioral Sciences.
Wuensch, K. L., & Evans, J. D. (1996) · Journal of the American Statistical Association · doi:10.2307/2291607 · Cited by 2,409
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“An example for such model-agnostic add-on systems are LIME and the Anchors, both developed by Ribeiro, Singh, and Guestrin [43, 44].”
MethodEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131
Foundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 6×
“[11] find that explaining the reasons for misclassification improves user trust, while not providing an explanation for misclassifications decreases trust.”
BackgroundInfluentialThe misunderstood limits of folk science: An illusion of explanatory depth
Rozenblit, L., & Keil, F. (2002) · Cognitive Science · doi:10.1207/s15516709cog2605_1 · Cited by 1,032
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 6×
“Inspired by [19], Poursabzi-Sangdeh et al. [39] suggest the “weight of advice” as a behavioural measure for trust in regression tasks.”
BackgroundMethodInfluentialManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Langer, E. J., Blank, A., & Chanowitz, B. (1978) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.36.6.635 · Cited by 694
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“, by visualising a decision tree) [18].”
Taking Advice: Accepting Help, Improving Judgment, and Sharing Responsibility
Harvey, N., & Fischer, I. (1997) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1997.2697 · Cited by 588
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576
Why and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 12×
“Participants were neither told their own classification performance nor the one of the system they interacted with, as [53] found that displayed accuracy information will be overwritten by experienced accuracy in the long run.”
BackgroundMethodInfluentialThe effects of transparency on trust in and acceptance of a content-based art recommender
Cramer, H., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., & Wielinga, B. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-008-9051-3 · Cited by 448
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Transparency and trust in artificial intelligence systems
Schmidt, P., Biessmann, F., & Teubner, T. (2020) · Journal of Decision Systems · doi:10.1080/12460125.2020.1819094 · Cited by 440
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · cited 3×
“[26] define a good explanation as truthful and complete, but not overwhelming.”
BackgroundExplanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388 · cited 4×
“Rader, Cotter, and Cho [41] further add that transparency concerns “non-obvious” information about the system that the user would not be able to acquire without additional explanatory mechanisms.”
BackgroundInfluentialTheoretical Considerations and Development of a Questionnaire to Measure Trust in Automation
Körber, M. (2019) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-319-96074-6_2 · Cited by 317 · cited 10×
“The user’s ability to trust and prior experiences with comparable systems influence trust as well [24].”
BackgroundMethodInfluentialProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 2×
“find a positive effect of specific explanation types on trust [31], whereas other research finds no impact of transparency on trust [8, 9].”
BackgroundResult"What is relevant in a text document?": An interpretable machine learning approach
Arras, L., Horn, F., Montavon, G., Müller, K.-R., & Samek, W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0181142 · Cited by 257
Analysing the Relationship between Risk and Trust
Jøsang, A., & Presti, S. L. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24747-0_11 · Cited by 248
Pathologies of Neural Models Make Interpretations Difficult
Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., & Boyd-Graber, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1407 · Cited by 245
Toward establishing trust in adaptive agents
Glass, A., McGuinness, D. L., & Wolverton, M. (2008) · Proceedings of the 13th international conference on Intelligent user interfaces · doi:10.1145/1378773.1378804 · Cited by 211
Jay, T., & Janschewitz, K. (2008) · Journal of Politeness Research. Language, Behaviour, Culture · doi:10.1515/jplr.2008.013 · Cited by 192
Asking ‘Why’ in AI: Explainability of intelligent systems – perspectives and challenges
Preece, A. (2018) · Intelligent Systems in Accounting, Finance and Management · doi:10.1002/isaf.1422 · Cited by 153
The Impact of Placebic Explanations on Trust in Intelligent Systems
Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312787 · Cited by 152 · cited 7×
“[12] found a first indication that the results from [27] also hold for human-machine interaction.”
BackgroundMethodResultInfluentialI can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119
Investigating intelligibility for uncertain context-aware applications
Lim, B. Y., & Dey, A. K. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030168 · Cited by 101
User Trust Dynamics: An Investigation Driven by Differences in System Performance
Yu, K., Berkovsky, S., Taib, R., Conway, D., Zhou, J., & Chen, F. (2017) · Proceedings of the 22nd International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025219 · Cited by 90
Illusory Causation and Illusory Correlation
Zebrowitz, L. A. (1980) · Personality and Social Psychology Bulletin · doi:10.1177/014616728064003 · Cited by 80
Socially Aware Systems Design: A Perspective Towards Technology-society Coupling
Baranauskas, M. C. C., Pereira, R., & Bonacin, R. (2017) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00101 · Cited by 49
Oduor, K. F., & Wiebe, E. (2008) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120805200422 · Cited by 39
Effects of Influence on User Trust in Predictive Decision Making
Zhou, J., Li, Z., Hu, H., Yu, K., Chen, F., Li, Z., & Wang, Y. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312962 · Cited by 29
Assessing Demand for Transparency in Intelligent Systems Using Machine Learning
Vorm, E. S. (2018) · 2018 Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista.2018.8466328 · Cited by 25 · cited 4×
“Vorm [51] reports “willingness to accept a computer-generated recommendation” as an observable sign of trust.”
BackgroundInfluentialProceedings of NIPS 2017 Symposium on Interpretable Machine LearningPreprint
Wilson, A. G., Yosinski, J., Simard, P., Caruana, R., & Herlands, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09889 · Cited by 5
Research works printed but not linked: 10
- P. S. Altares, A. R. I. Copo, Y. A. Gabuyo, A. T. Laddaran, L. D. P. Mejia, I. A. Policarpio, E. A. G. Sy, H. D. Tizon, and A. M. S. D. Yao. 2003. Elementary statistics: A modern approach. Rex Bookstore Inc. Manila, Philippines.
- Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).
- Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media. 512–515.
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
- Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In Proceedings of the 11th Australasian Conference on Information Systems. Citeseer, 6–8.
- Nazila Gol Mohammadi, Sachar Paulus, Mohamed Bishr, Andreas Metzger, Holger Könnecke, Sandro Hartenstein, Thorsten Weyer, and Klaus Pohl. 2013. Trustworthiness attributes and metrics for engineering trusted internet-based software systems. In Proceedings of the International Conference on Cloud Computing and Services Science. Springer, 19–35.
- Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 112–121.
- Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A proposal of user-centered explainable AI. In Proceedings of the IUI Workshops.
- Ariella Richardson and Avi Rosenfeld. 2018. A survey of interpretability and explainability in human-agent systems. In Proceedings of the XAI Workshop on Explainable Artificial Intelligence. 137–143.
- Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx. 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).
Cited by
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“For example, some studies estimate overall accuracy on the basis of the evaluation of as many as 500 distinct images 25 , whereas others estimate it on the basis of the evaluation of as few as 15 distinct ones 42 .”
The role of explainable AI in the context of the AI Act
Panigutti, C., Hamon, R., Hupont, I., Llorca, D. F., Yela, D. F., Junklewitz, H., Scalzo, S., Mazzini, G., Sánchez, I., Garrido, J. S., et al. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594069 · Cited by 273
“Many factors play a role in the relationship between trust and AI explanations [91, 119].”
BackgroundExplainable Artificial Intelligence-Based Decision Support Systems: A Recent Review
Kostopoulos, G., Davrazos, G., & Kotsiantis, S. B. (2024) · Electronics · doi:10.3390/electronics13142842 · Cited by 267
“This can be further analyzed through statistical tests to determine if the differences are significant [85].”
Westphal, M., Vössing, M., Satzger, G., Yom‐Tov, G. B., & Rafaeli, A. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107714 · Cited by 180
Artificial Intelligence and User Experience in reciprocity: Contributions and state of the art
Virvou, M. K. (2023) · Intelligent Decision Technologies · doi:10.3233/idt-230092 · Cited by 218
Metadata
- Authors
- Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C.
- Affiliations
- GESIS - Leibniz Institute for the Social Sciences (DE)
- University of Twente (NL)
- University of Duisburg-Essen (DE)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2022-03-31
- Volume
- 29
- Issue
- 4
- Pages
- 1-33
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4220693086
- Cited by
- 144
- References
- 46
- Citations by year
- 2026: 43 · 2025: 42 · 2024: 33 · 2023: 18 · 2022: 8
- FWCI
- 8.31
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion, cc-by-nc-sa)
- University of Twente Research Information (publishedVersion, cc-by-nc-sa)
- Funders
- Cancer Research Center Cologne Essen
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- user trust, AI explanations, model accuracy, automated decision-making, black box problem, explainable AI, trust measurement, classification accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.