CoChat
Sign Up
PreprintCited by 1Explainable Artificial Intelligence (XAI)Human-Automation Interaction and SafetyUsability and User Interface Design

The Effect of Explainable AI-based Decision Support on Human Task Performance: A Meta-Analysis

Open access
Haag, Felix·2025·arXiv
DOI 10.48550/arxiv.2504.13858arXiv 2504.13858
Open this paper in CoChat — free accountOpen PDF

Abstract

The desirable properties of explanations in information systems have fueled the demands for transparency in artificial intelligence (AI) outputs. To address these demands, the field of explainable AI (XAI) has put forth methods that can support human decision-making by explaining AI outputs. However, current empirical works present inconsistent findings on whether such explanations help to improve users' task performance in decision support systems (DSS). In this paper, we conduct a meta-analysis to explore how XAI affects human performance in classification tasks. Our results show an improvement in task performance through XAI-based decision support, though explanations themselves are not the decisive driver for this improvement. The analysis reveals that the studies' risk of bias moderates the effect of explanations in AI, while the explanation type appears to play only a negligible role. Our findings contribute to the human computer interaction field by enhancing the understanding of human-XAI collaboration in DSS.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.13858

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Haag (2025) demonstrated that the benefits of explainable AI-based decision support depend strongly on user expertise, with explanations improving performance primarily for less experienced users while having limited or even negative effects for experts.

    “Similarly, Haag (2025) demonstrated that the benefits of explainable AI-based decision support depend strongly on user expertise, with explanations improving performance primarily for less experienced users while having limited or even negative effects for experts, highlighting the role of expertise in influencing reliance behaviours.”

    From Investigating appropriate reliance on AI-Based decision support systems: the role of expertise, trust, and self-confidence (2025) · cites this paper as Haag, 2025

    The Introduction (p. 1) says the meta-analysis examines risk of bias and explanation type as moderators; the Results (pp. 3–4) report findings for those factors. Searches for user expertise, novice and expert users, and experience found no passage comparing their performance or reliance. The copy therefore does not establish the claimed expertise-dependent benefit. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 25

  1. [1]1. Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020). Evaluating saliency map explanations for convolutional neural networks: A user study. Proceedings of the 25th International Conference on Intelligent User Interfaces , 275 -285. https://doi.org/10.1145/3377325.3377519 · DOI · Paper page
  2. [2]2. Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 1 -16. https://doi.org/10.1145/3411764.3445717 · DOI · Paper page
  3. [3]3. Bauer, K., Von Zahn, M., & Hinz, O. (2023). Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users' Information Processing. Information Systems Research . https://doi.org/10.1287/isre.2023.1199 · DOI
  4. [4]4. Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020). Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. Proceedings of the 25th International Conference on Intelligent User Interfaces , 454 -464. https://doi.org/10.1145/3377325.3377498 · DOI · Paper page
  5. [5]5. Carton, S., Mei, Q., & Resnick, P. (2020). Featurebased explanations don't help people detect misclassifications of online toxicity. Proceedings of the 14th International AAAI Conference on Web and Social Media , 95 -106.
  6. [6]6. Gregor, S., & Benbasat, I. (1999). Explanations from intelligent systems: Theoretical foundations and implications for practice. MIS Quarterly , 497 -530.
  7. [7]7. Haag, F., Stingl, C., Zerfass, K., Hopf, K., & Staake, T. (2023). Overcoming Anchoring Bias: The Potential of AI and XAI-based Decision Support. ICIS 2023 Proceedings .
  8. [8]8. Hedges, L. V. (1981). Distribution theor y for Glass's estimator of effect size and related estimators. Journal of Educational Statistics , 6 (2), 107 -128.
  9. [9]9. Hemmer, P., Schemmer, M., Riefle, L., Rosellen, N., Vössing, M., & Kühl, N. (2022). Factors that influence the adoption of human-AI collaboration in clinical decision-making. ECIS 2022 Proceedings .
  10. [12]12. Jakubik, J., Schöffer, J., Hoge, V., Vössing, M., & Kühl, N. (2022). An empirical evaluation of predicted outcomes as explanations in human-AI decisionmaking. Joint European Conference on Machine Learning and Knowledge Discovery in Databases , 353 -368.
  11. [13]13. Lai, V., Liu, H., & Tan, C. (2020). 'Why is 'Chicago' deceptive?' Towards Building Model -Driven Tutorials for Humans. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , 1 -13. https://doi.org/10.1145/3313831.3376873 · DOI · Paper page
  12. [14]14. Leichtmann, B., Hinterreiter, A., Humer, C., Streit, M., & Mara, M. (2023a). Explainable Artificial Intelligence Improves Human Decision-Making: Results from a Mushroom Picking Experiment at a Public Art Festival. International Journal of HumanComputer Interaction , 1 -18. https://doi.org/10.1080/10447318.2023.2221605 · DOI
  13. [15]15. Leichtmann, B., Humer, C., Hinterreiter, A., Streit, M., & Mara, M. (2023b). Effects of Explainable Artificial Intelligence on trust and human behavior in a high-risk decision task. Computer in Human Behavior , 139 . https://doi.org/10.1016/j.chb.2022.107539 · DOI · Paper page
  14. [16]16. Liu, H., Lai, V., & Tan, C. (2021). Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making. Proceedings of the ACM on Human-Computer Interaction , 5 (CSCW2), 1 -45. https://doi.org/10.1145/3479552 · DOI · Paper page
  15. [17]17. Meske, C., Bunde, E., Schneider, J., & Gersch, M. (2020). Explainable Artificial Intelligence: Objectives, Stakeholders, and Future Research Opportunities. Information Systems Management , 1 -11. https://doi.org/10.1080/10580530.2020.1849465 · DOI
  16. [18]18. Mohseni, S., Zarei, N., & Ragan, E. D. (2021). A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems. ACM Transactions on Interactive Intelligent Systems , 11 (3 -4), 1 -45. https://doi.org/10.1145/3387166 · DOI · Paper page
  17. [20]20. Schauer, A. (2024). In the Eyes of the User: A Systematic Literature Review of User Perceptions in Human-XAI Interaction. ICIS 2024 Proceedings .
  18. [21]21. Schemmer, M., Hemmer, P., Nitsche, M., Kühl, N., & Vössing, M. (2022). A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , 617 -626. https://doi.org/10.1145/3514094.3534128 · DOI · Paper page
  19. [22]22. Silva, A., Schrum, M., Hedlund-Botti, E., Gopalan, N., & Gombolay, M. (2023). Explainable artificial intelligence: Evaluating the objective and subjective impacts of xai on human-agent interaction. International Journal of Human -Computer Interaction , 39 (7), 1390 -1404.
  20. [23]23. Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.Y., Corbett, M. S., Eldridge, S. M., Emberson, J. R., Hernán, M. A., Hopewell, S., Hróbjartsson, A., Junqueira, D. R., Jüni, P., Kirkham, J. J., Lasserson, T., Li, T., … Higgins, J. P. T. (2019). RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ , l4898. https://doi.org/10.1136/bmj.l4898 · DOI · Paper page
  21. [24]24. van der Waa, J., Nieuwburg, E., Cremers, A., & Neerincx, M. (2021). Evaluating XAI: A comparison of rule-based and example-based explanations. Artificial Intelligence , 291 . https://doi.org/10.1016/j.artint.2020.103404 · DOI · Paper page
  22. [25]25. vom Brocke, J., Simons, A., Niehaves, B., Riemer, K., Plattfaut, R., & Cleven, A. (2009). Reconstructing the giant: On the importance of rigour in documenting the literature search process. ECIS 2009 Proceedings .
  23. [26]26. Wang, X., & Yin, M. (2021). Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making. 26th International Conference on Intelligent User Interfaces , 318 -328. https://doi.org/10.1145/3397481.3450650 · DOI · Paper page
  24. [27]27. Wilson, H. J., & Daugherty, P. R. (2018). Collaborative intelligence: Humans and AI are joining forces. Harvard Business Review , 96 (4), 114 -123.
  25. [28]28. Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , 295 -305. https://doi.org/10.1145/3351095.3372852 · DOI · Paper page

Cited by

Sorted by citations

Authors
Haag, F.
Corresponding author
Haag, Felix
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.13858
arXiv
2504.13858
OpenAlex
W6948300545
Cited by
1
References
0
Citations by year
2025: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
explainable AI, decision support system, AI explanations, classification tasks, human-AI collaboration, risk of bias, explanation types, AI transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review