It's complicated: the relationship between user trust, model accuracy and explanations in AI
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Automated decision-making systems become increasingly powerful due to higher model complexity. While powerful in prediction accuracy, Deep Learning models are black boxes by nature, preventing users from making informed judgments about the correctness and fairness of such an automated system. Explanations have been proposed as a general remedy to the black box problem. However, it remains unclear if effects of explanations on user trust generalise over varying accuracy levels. In an online user study with 959 participants, we examined the practical consequences of adding explanations for user trust: We evaluated trust for three explanation types on three classifiers of varying accuracy. We find that the influence of our explanations on trust differs depending on the classifier’s accuracy. Thus, the interplay between trust and explanations is more complex than previously reported. Our findings also reveal discrepancies between self-reported and behavioural trust, showing that the choice of trust measure impacts the results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, research has shown that subjective trust can be a poor predictor of actual reliance (Dzindolet et al., 2003; Miller et al., 2016; Papenmeier et al., 2022).”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Papenmeier et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“The outcome of questionnaires is influenced by the participants' ability to reflect their attitude towards a system, which can be difficult for some participants (Papenmeier et al., 2022).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Papenmeier et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Across multiple studies in our overview a discrepancy between self-report and behavioural measures is observed (e.g., Papenmeier et al., 2022Papenmeier et al., , 2021;;Wang & Yin, 2023).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Papenmeier et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Rempel, J. K., Holmes, J. G., & Zanna, M. P. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.49.1.95 · Cited by 3,146 · cited 2×
“Moreover, trust builds up over time [42] and shows phases of development and refinement [54].”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“[15] therefore suggest to account for the limited human attention span and cognitive abilities when designing explanations, e.”
BackgroundStraightforward Statistics for the Behavioral Sciences.
Wuensch, K. L., & Evans, J. D. (1996) · Journal of the American Statistical Association · doi:10.2307/2291607 · Cited by 2,409
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“An example for such model-agnostic add-on systems are LIME and the Anchors, both developed by Ribeiro, Singh, and Guestrin [43, 44].”
Method
Research works printed but not linked: 10
- P. S. Altares, A. R. I. Copo, Y. A. Gabuyo, A. T. Laddaran, L. D. P. Mejia, I. A. Policarpio, E. A. G. Sy, H. D. Tizon, and A. M. S. D. Yao. 2003. Elementary statistics: A modern approach. Rex Bookstore Inc. Manila, Philippines.
- Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).
- Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media. 512–515.
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
- Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In Proceedings of the 11th Australasian Conference on Information Systems. Citeseer, 6–8.
- Nazila Gol Mohammadi, Sachar Paulus, Mohamed Bishr, Andreas Metzger, Holger Könnecke, Sandro Hartenstein, Thorsten Weyer, and Klaus Pohl. 2013. Trustworthiness attributes and metrics for engineering trusted internet-based software systems. In Proceedings of the International Conference on Cloud Computing and Services Science. Springer, 19–35.
- Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 112–121.
- Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A proposal of user-centered explainable AI. In Proceedings of the IUI Workshops.
- Ariella Richardson and Avi Rosenfeld. 2018. A survey of interpretability and explainability in human-agent systems. In Proceedings of the XAI Workshop on Explainable Artificial Intelligence. 137–143.
- Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx. 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).
Cited by
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“For example, some studies estimate overall accuracy on the basis of the evaluation of as many as 500 distinct images 25 , whereas others estimate it on the basis of the evaluation of as few as 15 distinct ones 42 .”
The role of explainable AI in the context of the AI Act
Panigutti, C., Hamon, R., Hupont, I., Fernandez Llorca, D., Fano Yela, D., Junklewitz, H., Scalzo, S., Mazzini, G., Sanchez, I., Soler Garrido, J., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594069 · Cited by 137
“Many factors play a role in the relationship between trust and AI explanations [91, 119].”
BackgroundExplainable Artificial Intelligence-Based Decision Support Systems: A Recent Review
Kostopoulos, G., Davrazos, G., & Kotsiantis, S. B. (2024) · Electronics · doi:10.3390/electronics13142842 · Cited by 267
“This can be further analyzed through statistical tests to determine if the differences are significant [85].”
Westphal, M., Vössing, M., Satzger, G., Yom‐Tov, G. B., & Rafaeli, A. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107714 · Cited by 183
Artificial Intelligence and User Experience in reciprocity: Contributions and state of the art
Virvou, M. K. (2023) · Intelligent Decision Technologies · doi:10.3233/idt-230092 · Cited by 218
Metadata
- Authors
- Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C.
- Affiliations
- GESIS - Leibniz Institute for the Social Sciences (DE)
- University of Twente (NL)
- University of Duisburg-Essen (DE)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2022-03-31
- Volume
- 29
- Issue
- 4
- Pages
- 1-33
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4220693086
- Cited by
- 144
- References
- 46
- Citations by year
- 2026: 43 · 2025: 42 · 2024: 33 · 2023: 18 · 2022: 8
- FWCI
- 8.31
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion, cc-by-nc-sa)
- University of Twente Research Information (publishedVersion, cc-by-nc-sa)
- Funders
- Cancer Research Center Cologne Essen
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- user trust, AI explanations, model accuracy, automated decision-making, black box problem, explainable AI, trust measurement, classification accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: It's complicated: the relationship between user trust, model accuracy and explanations in AI. https://app.cochat.ai/papers/10.1145/3495013
- The paper
- Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C. (2022). It's complicated: the relationship between user trust, model accuracy and explanations in AI. ACM Transactions on Computer-Human Interaction. https://doi.org/10.1145/3495013
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.