CoChat

Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics

Open access

The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.

Abstract

The most successful Machine Learning (ML) systems remain complex black boxes to end-users, and even experts are often unable to understand the rationale behind their decisions. The lack of transparency of such systems can have severe consequences or poor uses of limited valuable resources in medical diagnosis, financial decision-making, and in other high-stake domains. Therefore, the issue of ML explanation has experienced a surge in interest from the research community to application domains. While numerous explanation methods have been explored, there is a need for evaluations to quantify the quality of explanation methods to determine whether and to what extent the offered explainability achieves the defined objective, and compare available explanation methods and suggest the best explanation from the comparison for a specific task. This survey paper presents a comprehensive overview of methods proposed in the current literature for the evaluation of ML explanations. We identify properties of explainability from the review of definitions of explainability. The identified properties of explainability are used as objectives that evaluation metrics should achieve. The survey found that the quantitative metrics for both model-based and example-based explanations are primarily used to evaluate the parsimony/simplicity of interpretability, while the quantitative metrics for attribution-based explanations are primarily used to evaluate the soundness of fidelity of explainability. The survey also demonstrated that subjective measures, such as trust and confidence, have been embraced as the focal point for the human-centered evaluation of explainable systems. The paper concludes that the evaluation of ML explanations is a multidisciplinary research topic. It is also not possible to define an implementation of evaluation metrics, which can be applied to all explanation methods.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/electronics10050593

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 5

  1. Zhou, J., and Chen, F. (2018). Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent, Springer.
  2. Lage, I., and Doshi-Velez, F. (2020, January 17). Human-in-the-Loop Learning of Interpretable and Intuitive Representations. Proceedings of the ICML Workshop on Human Interpretability in Machine Learning, Vienna, Austria.
  3. Becker, B., Kohavi, R., and Sommerfield, D. (2001). Visualizing the Simple Bayesian Classifier. Information Visualization in Data Mining and Knowledge Discovery, Morgan Kaufmann Publishers Inc.
  4. Poursabzi-Sangdeh, F., Goldstein, D.G., Hofman, J.M., Vaughan, J.W., and Wallach, H. (2019). Manipulating and Measuring Model Interpretability. arXiv.
  5. Yang, M., and Kim, B. (2019). Benchmarking Attribution Methods with Relative Feature Importance. arXiv, 1907.

Cited by

See all 676

Metadata

Authors
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A.
Affiliations
  • University of Technology Sydney (AU)
  • Medical University of Graz (AT)
Corresponding author
  • Zhou, Jianlong
  • Gandomi, Amir H.
Venue
Electronics
Publisher
MDPI AG
Published
2021-03-04
Volume
10
Issue
5
Pages
593
ISSN
2079-9292 (ISSN-L)
OpenAlex
W3133543405
Cited by
606
References
80
Citations by year
2026: 75 · 2025: 138 · 2024: 179 · 2023: 109 · 2022: 84 · 2021: 20
FWCI
34.65
Type
journal article
Language
en
Open access
Locations
  • Electronics (publishedVersion, cc-by)
  • OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • University of Technology Sydney
  • Austrian Science Fund (P 32554)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
explainable machine learning, explanation quality evaluation, evaluation metrics, explainability, model-based explanation, example-based explanations, interpretability, fidelity, human-centered evaluation, black-box models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.