CoChat

A Survey of Methods for Explaining Black Box Models

Open access
Guidotti, Riccardo, Monreale, Anna, Ruggieri, Salvatore, Turini, Franco, Giannotti, Fosca, Pedreschi, Dino · 2019 · ACM Computing Surveys

The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index; 4 of the 59 sources its claims cite were checked for support.

Abstract

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.

Overview

CoChat Summary

TL;DR

This survey classifies methods for explaining black-box machine learning models into four problem types: black box model explanation, black box outcome explanation, black box inspection, and transparent box design. It organizes the literature by problem, explanator, black-box type, and data type, and highlights open challenges around defining and measuring explanation and comprehensibility.

Key findings

  • The paper identifies and formally defines four explanation problems: black box model explanation (global interpretable mimic), black box outcome explanation (local explanation for a prediction), black box inspection (visual/textual representation of behavior), and transparent box design (interpretable model by design).
  • It classifies explanators into decision trees, decision rules, feature importance, saliency masks, sensitivity analysis, partial dependence plots, prototype selection, and neuron activation; black boxes into neural networks, tree ensembles, SVMs, deep neural networks, and agnostic black boxes; and data into tabular, image, and text.
  • Many methods are specific to a problem or black-box type, while agnostic/generalizable approaches use reverse engineering by querying the black box as an oracle; examples include Trepan, inTrees, LIME, and PALM.
  • Motivating cases cited include COMPAS ethnic bias (black non-reoffenders classified high risk twice as much as whites who did not re-offend; white repeat offenders classified low risk twice as much as black repeat offenders), credit scoring discordance (in 500,000 records, 29% of consumers received scores differing by at least fifty points among three major US banks), and the St. George’s Hospital screening program that discriminated via inferred ethnicity from surnames and place of birth.

Methods

  • Literature survey/review; no primary empirical study. The paper formalizes the general classification problem and defines the four explanation problems as specific cases.
  • The classification dimensions are: type of problem faced, type of explanator adopted, type of black box model opened, and type of data used as input by the black box. Table 1 summarizes methods with features including General, Random, Examples, Code, and Dataset.
  • The common reverse-engineering approach queries the black box with a test dataset to create an oracle dataset, then trains a comprehensible predictor; approaches are distinguished as generalizable/pedagogical versus not generalizable/decompositional.

Limitations

  • The paper states that there is no agreement on what an explanation is, no general/common formalism for defining explanations or properties such as soundness, completeness, compactness, and comprehensibility, and no serious work quantifying human comprehensibility; it also lists open questions on latent features, recommender systems, and learning predictors directly from explanations.
  • The paper states it does not address whether the black box can be queried at will versus only a fixed dataset being available, or whether the complete feature set is known, because there is insufficient literature.
  • Evident from the text: the survey scope is limited to tabular, image, and text data, with no work addressing interpretability for sequence, spatio-temporal, or complex network data; many reviewed methods are ad hoc or specific, only a few consider user expertise, and no work provides real experiments on the time required to understand an explanation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3236009

Cites

132 of 132 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 13 Not valid · 38 Not checked · 49 Own results and interpretations

23 more claims

Create a free account to see all 104 claims and to verify the 46 references still open.

See all claims — free account
  • It may state two separate claims as one. 37
  • Probably not a substantial claim. 65
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

16 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 132 references

Research works printed but not linked: 31

  1. 10.5555/645920.672836 · doi:10.5555/645920.672836
  2. Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, and Karol Zieba. 2016. VisualBackProp: Visualizing CNNs for autonomous driving. CoRR, Vol. abs/1611.05418 (2016). · arXiv
  3. Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press.
  4. Carolyn Carter, Elizabeth Renuart, Margot Saunders, and Chi Chi Wu. 2006. The credit card market and regulation: In need of repair. NC Bank. Inst. 10 (2006), 23.
  5. H. A. Chipman, E. I. George, and R. E. McCulloh. 1998. Making sense of a forest of trees. In Proceedings of the 30th Symposium on the Interface, S. Weisberg (Ed.). Fairfax Station, VA: Interface Foundation of North America, 84--92.
  6. Paulo Cortez and Mark J. Embrechts. 2011. Opening black box data mining models using sensitivity analysis. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’11). IEEE, 341--348.
  7. 10.5555/3091574.3091580 · doi:10.5555/3091574.3091580
  8. 10.5555/2998828.2998832 · doi:10.5555/2998828.2998832
  9. 10.5555/3000292.3000299 · doi:10.5555/3000292.3000299
  10. Bryce Goodman and Seth Flaxman. 2016. EU regulations on algorithmic decision-making and a “right to explanation.” In Proceedings of the ICML Workshop on Human Interpretability in Machine Learning (WHI’16). Retrieved from http://arxiv. org/abs/1606.08813 v1.
  11. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018). · arXiv
  12. U. Johansson, R. König, and L. Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In Proceedings of the 13th International Conference on Artificial Neural Networks. 13--16.
  13. 10.5555/3157096.3157352 · doi:10.5555/3157096.3157352
  14. 10.5555/2969033.2969045 · doi:10.5555/2969033.2969045
  15. 10.5555/2969442.2969492 · doi:10.5555/2969442.2969492
  16. Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable algorithms. U. Penn. Law Rev. 165 (2017), 633--705.
  17. Will Landecker, Michael D. Thomure, Luís M. A. Bettencourt, Melanie Mitchell, Garrett T. Kenyon, and Steven P. Brumby. 2013. Interpreting individual classifications of hierarchical networks. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’13). IEEE, 32--38.
  18. Dmitry M. Malioutov, Kush R. Varshney, Amin Emad, and Sanjeeb Dash. 2017. Learning interpretable classification rules with boolean compressed sensing. In Transparent Data Mining for Big and Small Data. Springer, 95--121.
  19. 10.5555/3091825.3091863 · doi:10.5555/3091825.3091863
  20. 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
  21. 10.5555/2717112 · doi:10.5555/2717112
  22. Michael J. Pazzani, S. Mani, William R. Shankle et al. 2001. Acceptance of rules generated by machine learning among medical experts. Methods Info. Med. 40, 5 (2001), 380--385.
  23. 10.5555/1597122.1597143 · doi:10.5555/1597122.1597143
  24. 10.5555/1625015.1625078 · doi:10.5555/1625015.1625078
  25. 10.5555/152181 · doi:10.5555/152181
  26. Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. 2016. Programs as black-box explanations. arXiv preprint arXiv:1611.07579 (2016). · arXiv
  27. Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 353--382.
  28. Marina M.-C. Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567 (2016). · arXiv
  29. Tong Wang. 2017. Multi-value rule sets. arXiv preprint arXiv:1710.05257 (2017). · arXiv
  30. 10.5555/273530.273548 · doi:10.5555/273530.273548
  31. 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336

Cited by

Metadata

Authors
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D.
Affiliations
  • University of Pisa (IT)
  • Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
Venue
ACM Computing Surveys
Published
2018-08-22
Volume
51
Issue
5
Pages
1-42
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W2786715987
Cited by
629
References
121
Citations by year
2026: 3 · 2025: 62 · 2024: 82 · 2023: 86 · 2022: 92 · 2021: 145 · 2020: 105 · 2019: 40 · 2018: 13
FWCI
42.69
Type
journal article
Language
en
Open access
Locations
  • ACM Computing Surveys (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
  • Scuola Normale Superiore di Pisa (submittedVersion, public-domain)
  • arXiv (Cornell University)
Funders
Horizon 2020 Framework Programme (-2014-2015, 654024)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
Keywords
explainable AI, black-box models, interpretability, decision support system, explanation methods, model transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.