It's complicated: the relationship between user trust, model accuracy and explanations in AI
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Automated decision-making systems become increasingly powerful due to higher model complexity. While powerful in prediction accuracy, Deep Learning models are black boxes by nature, preventing users from making informed judgments about the correctness and fairness of such an automated system. Explanations have been proposed as a general remedy to the black box problem. However, it remains unclear if effects of explanations on user trust generalise over varying accuracy levels. In an online user study with 959 participants, we examined the practical consequences of adding explanations for user trust: We evaluated trust for three explanation types on three classifiers of varying accuracy. We find that the influence of our explanations on trust differs depending on the classifier’s accuracy. Thus, the interplay between trust and explanations is more complex than previously reported. Our findings also reveal discrepancies between self-reported and behavioural trust, showing that the choice of trust measure impacts the results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, research has shown that subjective trust can be a poor predictor of actual reliance (Dzindolet et al., 2003; Miller et al., 2016; Papenmeier et al., 2022).”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Papenmeier et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“The outcome of questionnaires is influenced by the participants' ability to reflect their attitude towards a system, which can be difficult for some participants (Papenmeier et al., 2022).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Papenmeier et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Across multiple studies in our overview a discrepancy between self-report and behavioural measures is observed (e.g., Papenmeier et al., 2022Papenmeier et al., , 2021;;Wang & Yin, 2023).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Papenmeier et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Rempel, J. K., Holmes, J. G., & Zanna, M. P. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.49.1.95 · Cited by 3,146 · cited 2×
“Moreover, trust builds up over time [42] and shows phases of development and refinement [54].”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“[15] therefore suggest to account for the limited human attention span and cognitive abilities when designing explanations, e.”
BackgroundStraightforward Statistics for the Behavioral Sciences.
Wuensch, K. L., & Evans, J. D. (1996) · Journal of the American Statistical Association · doi:10.2307/2291607 · Cited by 2,409
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“An example for such model-agnostic add-on systems are LIME and the Anchors, both developed by Ribeiro, Singh, and Guestrin [43, 44].”
Method
Research works printed but not linked: 10
- P. S. Altares, A. R. I. Copo, Y. A. Gabuyo, A. T. Laddaran, L. D. P. Mejia, I. A. Policarpio, E. A. G. Sy, H. D. Tizon, and A. M. S. D. Yao. 2003. Elementary statistics: A modern approach. Rex Bookstore Inc. Manila, Philippines.
- Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).
- Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media. 512–515.
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
- Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In Proceedings of the 11th Australasian Conference on Information Systems. Citeseer, 6–8.
- Nazila Gol Mohammadi, Sachar Paulus, Mohamed Bishr, Andreas Metzger, Holger Könnecke, Sandro Hartenstein, Thorsten Weyer, and Klaus Pohl. 2013. Trustworthiness attributes and metrics for engineering trusted internet-based software systems. In Proceedings of the International Conference on Cloud Computing and Services Science. Springer, 19–35.
- Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 112–121.
- Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A proposal of user-centered explainable AI. In Proceedings of the IUI Workshops.
- Ariella Richardson and Avi Rosenfeld. 2018. A survey of interpretability and explainability in human-agent systems. In Proceedings of the XAI Workshop on Explainable Artificial Intelligence. 137–143.
- Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx. 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI).
Cited by
Items 101–144 of 144
How Do We Measure Over-Reliance? A Unified Probabilistic View
Mikhaylova, D., Turchi, T., & Malizia, A. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742414.3794719 · Cited by 2
BC-TRUST: Blockchain-Assisted Trust Assessment via Auditable End-to-End Latency Evidence in SIoT
Luo, S., Hu, T., Sun, C., & Guan, Z. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3707856 · Cited by 1
Yang, M. H., Jiang, J., Liu, X., & Kahai, S. S. (2026) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2026.103898 · Cited by 1
Task Allocation and Role Distribution in Humane-AI Ecosystems
Nowak, A., Ziembowicz, K., Pitt, J. V., Biesaga, M., Boros, A., Bartkowski, W., Lewandowska, W. M., Borkowski, W., & Mertzani, A. (2026) · Springer eBooks · doi:10.1007/978-3-031-61050-9_24-1 · Cited by 1
Tomlinson, M., Pinney, J., & Bentotahewa, V. (2024) · Springer eBooks · doi:10.1007/978-3-031-76473-8_5 · Cited by 2
The Impact of Usability and Reliability on ChatGPT Satisfaction among Gen Z and Gen Y
Bach, M. P., Palić, M., & Šimičević, V. (2025) · Organizacija · doi:10.2478/orga-2025-0013 · Cited by 3
Krämer, N. C., Wischnewski, M., & Müller, E. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/h32ze · Cited by 4
Assessing and Addressing Model Trustworthiness Trade-offs in Trauma Triage
Talbert, D. A., Phillips, K., Brown, K. E., & Talbert, S. (2024) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213024600078 · Cited by 2
Ethical AI in Banking: A Sectoral Literature Review with Case-Based Interventions
Smit, J. (2025) · Open Journal of Business and Management · doi:10.4236/ojbm.2025.134125 · Cited by 3
Al-Ansari, N., Al‐Thani, D., & Bahameish, M. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101083 · Cited by 2
Páez, A. (2025) · Wiley eBooks · doi:10.1002/9781394240821.ch11 · Cited by 2
“In the context of AI, user trust can be understood as ‘the extent to which the trustee believes that an automated system will behave as expected’ (Papenmeier et al. 2022).”
Taka, E., Bhattacharya, D., Garde‐Hansen, J., Sharma, S., & Guha, T. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750785 · Cited by 1
Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance
Imperial, J. M., Jones, M. D., & Madabushi, H. T. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i5.16189 · Cited by 16
Akalın, N., & Riveiro, M. (2025) · IEEE RO-MAN · doi:10.1109/ro-man63969.2025.11217728
“In our study, we adopted the definition for explanation from [8], which defines it as “the mechanisms by which a system communicates information about its inner workings to the user.””
User-centred design towards trust and engagement with agricultural climate services in Australia
Snow, S., & Wealands, K. (2024) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3726986.3727019
“The same situation is found throughout HCI studies of trust in autonomous systems, where users’ trust in (new) technical systems is unavoidably formed with imperfect knowledge of that system [64].”
InfluentialContext-dependent Effects of Explanations on Multiple Layers of Trust
Ismail-Tsaous, E., Uhl, M., Spannagl, C., Krügel, S., & Schmid, U. (2025) · doi:10.64136/dryx3176
Intelligent delignification: leveraging explainable AI for ozone transport modeling and optimization
Rizwan, M., Khan, M., Khan, S., Baig, K. S., Ali, A., Abdulnabi, M. S., & Khan, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-28638-7
“The results demonstrate that the Extra Trees Regressor achieved the highest predictive accuracy among the 19 evaluated ML models, as indicated by its superior R 2 (0.9874), lowest RMSE (0.907), and minimal MAPE (0.01).”
InfluentialEvidence-Driven Decision Support for AI Model Selection in Research Software EngineeringPreprint
Joonbakhsh, A., Rostami, A., Kamalinia, A., Nazeri, A., Khunjush, F., Tekinerdoğan, B., & Farshidi, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6105552 · Cited by 2
Blending Designer Insights With Predicted Crowd‐Based Color Compatibility in Interior Color Design
Jin, S., & Hyun, K. H. (2026) · Color Research & Application · doi:10.1002/col.70055
den Hengst, F., Alaka, S., & Kamphorst, B. A. (2026) · SoftwareX · doi:10.1016/j.softx.2026.102536 · Cited by 2
Kahr, P. K., & Bernstein, A. (2026) · doi:10.1145/3780045.3780056 · Cited by 3
Passing the Buck to AI: How Individuals’ Decision-Making Patterns Affect Reliance on AI
Mei, K. X., Pang, R. Y., Lyford, A., Wang, L. L., & Reinecke, K. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3786326 · Cited by 8
“Users’ trust and reliance on systems are often captured via objective measures such as the agreement of an individual’s final decision and an AI’s suggestion in decision-making tasks [13, 14, 48, 52, 56, 73] or switching from one’s decision to the suggestion of an AI [13, 52, 73].”
Uma, S., Ghosh, S., & Mustaqim, D. D. A. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789130
shuning, Z., Xu, Y., Wang, Y. M., Pang, Y., Chen, H., Bai, J., Li, S., Tian, Y., Yi, X., & Jiang, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6390776
Challenges and Successes in Implementing AI-Powered Professional Learning for University Staff
Kayyali, M. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-3638-1.ch003
Vereschak, O. (2022) · theses.fr (ABES) · doi:10.70675/7ecd53bfza7ccz4edfz8f1azdb8052ba1233 · Cited by 2
Once Upon AI Time: Combining Narrative and Games for Early AI Literacy
Pu, I. M., Yi, M., Bagiati, A., Evangelou, D., Alghowinem, S., & Breazeal, C. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790471 · Cited by 1
“While GenAI’s accessibility via natural language has lowered access barriers [50], this can also foster a misleading illusion of understanding: fluent interaction does not mean users grasp the mechanisms, limitations, or biases behind these systems [69, 87].”
Xu, G., Zhang, Y., & Zhang, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6589389
Streamlining risk assessment for predictive maintenance: Can AI be trusted as an assistant?
Shobanke, M., Roberts, C., & Shittu, E. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104803
Trust Me, I'm Probably Right: Investigating AI Explanations for Establishing Appropriate Trust
Hartvig, F. D., Egelund, A. L., & Elmqvist, N. (2026) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3811427.3811445 · Cited by 1
“Studies have found that explanations can have mixed effects, with one study finding that not giving an explanation leads to equal or even higher levels of trust compared to providing one [26].”
For Those About to Rely—A Taxonomy of Experimental Studies on AI Reliance: Completed Research Paper
Schaschek, M., Spatscheck, N., & Winkelmann, A. (2026) · Lecture notes in information systems and organisation · doi:10.1007/978-3-032-08480-4_2
Savchuk, D., Doroshenko, A., & Kynash, Y. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10060184
Nneke, N., Crockett, K. A., & Latham, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7011187
Auditing a Robotic System for the AI Act
Lucaj, L., Bok, F., & van der Smagt, P. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812223 · Cited by 2
“We looked at the best practices for accuracy, robustness and cybersecurity [Papenmeier et al. 2022; Zhang et al. 2020].”
Kivaste, C. K., Värno, P., & Sousa, S. C. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-30836-8_40
TrueRate: Operationalizing Sentiment Analysis for Accurate Smartphone App Rating Inference
Setu, J. H., Hossain, M. J., Islam, A., Rahman, A. K. M. M., & Amin, M. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30549-7_14
Explainable Brain Tumor Classification Using VGG16 and Grad-CAM on MRI Scans
Anuradha, K. D., Afnaan, K., Singh, T., & Naik, G. (2025) · Springer eBooks · doi:10.1007/978-981-95-3811-9_8
Wei, C., Su, E., Kalirai, M., Tan, A., Dushniku, P., Jones, B., Kim, Y., & Kuzminykh, A. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3830401
“For example, when a user over-estimates or over-trusts a system’s capabilities, their inflated expectation of system capabilities might result in their acceptance of AI recommendations without critical assessment of recommendation reliability in relation to a given decision task [70, 95, 98].”
Zhu, J., Chen, J., Covarrubias, M., Li, D., & Wei, W. (2026) · Sustainability · doi:10.3390/su18157986 · Cited by 1
Hein, I., Ullrich, D., Brunner, C., & Diefenbach, S. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1869586
Man, S. S., Ding, M., Zhang, Z., Qin, Z., Gong, X., & Chan, A. H. S. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2712809
xAARA: Explainable and Augmented Automated Rehabilitation Assessment
Ahmed, T., Rikakis, T., Kelliher, A., Khan, M. S. S., & Guo, Z. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3848511
“Developing explainable AI (xAI) in these contexts is therefore challenging [48, 68, 90].”
Park, G. H., & Kang, H. (2026) · Applied Sciences · doi:10.3390/app16199400
The influence of data-driven personalised fashion recommendations on consumer decision-making
Liu, C., Cui, R., & Jiang, Q. (2026) · Kybernetes · doi:10.1108/k-07-2025-1661
Metadata
- Authors
- Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C.
- Affiliations
- GESIS - Leibniz Institute for the Social Sciences (DE)
- University of Twente (NL)
- University of Duisburg-Essen (DE)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2022-03-31
- Volume
- 29
- Issue
- 4
- Pages
- 1-33
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4220693086
- Cited by
- 144
- References
- 46
- Citations by year
- 2026: 43 · 2025: 42 · 2024: 33 · 2023: 18 · 2022: 8
- FWCI
- 8.31
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion, cc-by-nc-sa)
- University of Twente Research Information (publishedVersion, cc-by-nc-sa)
- Funders
- Cancer Research Center Cologne Essen
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- user trust, AI explanations, model accuracy, automated decision-making, black box problem, explainable AI, trust measurement, classification accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.