A Survey of Methods for Explaining Black Box Models
Open accessThe DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index; 4 of the 59 sources its claims cite were checked for support.
Abstract
In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Overview
CoChat Summary
TL;DR
This survey classifies methods for explaining black-box machine learning models into four problem types: black box model explanation, black box outcome explanation, black box inspection, and transparent box design. It organizes the literature by problem, explanator, black-box type, and data type, and highlights open challenges around defining and measuring explanation and comprehensibility.
Key findings
- The paper identifies and formally defines four explanation problems: black box model explanation (global interpretable mimic), black box outcome explanation (local explanation for a prediction), black box inspection (visual/textual representation of behavior), and transparent box design (interpretable model by design).
- It classifies explanators into decision trees, decision rules, feature importance, saliency masks, sensitivity analysis, partial dependence plots, prototype selection, and neuron activation; black boxes into neural networks, tree ensembles, SVMs, deep neural networks, and agnostic black boxes; and data into tabular, image, and text.
- Many methods are specific to a problem or black-box type, while agnostic/generalizable approaches use reverse engineering by querying the black box as an oracle; examples include Trepan, inTrees, LIME, and PALM.
- Motivating cases cited include COMPAS ethnic bias (black non-reoffenders classified high risk twice as much as whites who did not re-offend; white repeat offenders classified low risk twice as much as black repeat offenders), credit scoring discordance (in 500,000 records, 29% of consumers received scores differing by at least fifty points among three major US banks), and the St. George’s Hospital screening program that discriminated via inferred ethnicity from surnames and place of birth.
Methods
- Literature survey/review; no primary empirical study. The paper formalizes the general classification problem and defines the four explanation problems as specific cases.
- The classification dimensions are: type of problem faced, type of explanator adopted, type of black box model opened, and type of data used as input by the black box. Table 1 summarizes methods with features including General, Random, Examples, Code, and Dataset.
- The common reverse-engineering approach queries the black box with a test dataset to create an oracle dataset, then trains a comprehensible predictor; approaches are distinguished as generalizable/pedagogical versus not generalizable/decompositional.
Limitations
- The paper states that there is no agreement on what an explanation is, no general/common formalism for defining explanations or properties such as soundness, completeness, compactness, and comprehensibility, and no serious work quantifying human comprehensibility; it also lists open questions on latent features, recommender systems, and learning predictors directly from explanations.
- The paper states it does not address whether the black box can be queried at will versus only a fixed dataset being available, or whether the complete feature set is known, because there is insufficient literature.
- Evident from the text: the survey scope is limited to tabular, image, and text data, with no work addressing interpretability for sequence, spatio-temporal, or complex network data; many reviewed methods are ad hoc or specific, only a few consider user expertise, and no work provides real experiments on the time required to understand an explanation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 13 Not valid · 38 Not checked · 49 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and no one provides real experiments about the time required to understand an explanation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The works reviewed in the literature only implicitly specify if their proposal is global or local.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Just a few of them take into account the nature of user expertise [29,84,87]”
What the sources say
- Not checked yetcites [29] 29. R. D. Gibbons, G. Hooker, M. D. Finkelman, D. J. Weiss, P. A. Pilkonis, E. Frank, T. Moore, and D. J. Kupfer. The cad-mdd: a computerized adaptive diagnostic screening tool for depression. The Journal of clinical psychiatry , 74(7):669, 2013. source
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
- Not checked yetcites [87] 87. V. Schetinin, J. E. Fieldsend, D. Partridge, T. J. Coats, W. J. Krzanowski, R. M. Everson, T. C. Bailey, and A. Hernandez. Confident interpretation of bayesian decision tree ensembles for clinical applications. IEEE Transactions on Information Technology in Biomedicine , 11(3):312-319, 2007. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In the following, we list all the black boxes opened in the reviewed papers. These black boxes are all supervised learning algorithm designed to solve a classification problem [95].”
What the sources say
- Not checked yetcites [95] 95. P.-N. Tan et al. Introduction to data mining . Pearson Education India, 2006.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, this is not always the case in practice [84].”
What the sources say
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 104 claims and to verify the 46 references still open.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 65
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While “black‐box” algorithms raise complex opaqueness and interpretability challenges, simpler interpretable ML algorithms do not, but have generally also been regarded as less powerful (Guidotti et al. 2019; see however, Rudin 2019).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Guidotti et al. 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since we could not find a complete and recent overview of explanation types, we reviewed recent XAI surveys [14,58,167,280] and adapted and extended the categories identified by Guidotti et al. [86] with these insights.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, there is no consensus on what a good XAD technique should be.”
From A Survey on Explainable Anomaly Detection (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Among the main impediments prohibiting banking institutions from implementing their AI strategy is the ''black-box'' paradox [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The models are black boxes due to their significant non-linearity and intricacy, making it unfeasible for individuals to fathom their fundamental operational processes and judgment mechanisms [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Problem Formulation · p. 13
“for the comprehensible predictor c we get the set of predictions ¯ Y = ⋃ x ∈ X c ( x )”
§ 4.1 Problem Formulation · p. 13
“where as target value is used the prediction ¯ Y of the black box b instead of the real values Y”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a single representat…”
BackgroundLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 5×
“al. 2017 Model Expl. DT AGN ANY X X X - [97] Tolomei et al. 2017 Model Expl. FI TE TAB X - [108] Xu et al. 2015 Outcome Expl. SM DNN IMG X X X - [25] Fong et al. 2017 Outcome Expl. SM DNN IMG X X CAM [113] Zhou et al. 2016 Outcome Expl. SM DNN IMG X X X Grad-CAM [89] Selvaraju et al. 2016 Outcome Expl. SM DNN IMG X X…”
BackgroundMethodInfluentialShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Reference [134] introduces an attention-based model f , which automatically identifies the contents of an image.”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“In Reference [116] it is shown the alteration of an image (e.”
Background(2008) · Data Mining and Market Intelligence for Optimal Marketing Returns · doi:10.4324/9780080878096-12 · Cited by 6,954
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundThe Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375 · cited 1×
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
Background
Research works printed but not linked: 31
- 10.5555/645920.672836 · doi:10.5555/645920.672836
- Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, and Karol Zieba. 2016. VisualBackProp: Visualizing CNNs for autonomous driving. CoRR, Vol. abs/1611.05418 (2016). · arXiv
- Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press.
- Carolyn Carter, Elizabeth Renuart, Margot Saunders, and Chi Chi Wu. 2006. The credit card market and regulation: In need of repair. NC Bank. Inst. 10 (2006), 23.
- H. A. Chipman, E. I. George, and R. E. McCulloh. 1998. Making sense of a forest of trees. In Proceedings of the 30th Symposium on the Interface, S. Weisberg (Ed.). Fairfax Station, VA: Interface Foundation of North America, 84--92.
- Paulo Cortez and Mark J. Embrechts. 2011. Opening black box data mining models using sensitivity analysis. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’11). IEEE, 341--348.
- 10.5555/3091574.3091580 · doi:10.5555/3091574.3091580
- 10.5555/2998828.2998832 · doi:10.5555/2998828.2998832
- 10.5555/3000292.3000299 · doi:10.5555/3000292.3000299
- Bryce Goodman and Seth Flaxman. 2016. EU regulations on algorithmic decision-making and a “right to explanation.” In Proceedings of the ICML Workshop on Human Interpretability in Machine Learning (WHI’16). Retrieved from http://arxiv. org/abs/1606.08813 v1.
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018). · arXiv
- U. Johansson, R. König, and L. Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In Proceedings of the 13th International Conference on Artificial Neural Networks. 13--16.
- 10.5555/3157096.3157352 · doi:10.5555/3157096.3157352
- 10.5555/2969033.2969045 · doi:10.5555/2969033.2969045
- 10.5555/2969442.2969492 · doi:10.5555/2969442.2969492
- Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable algorithms. U. Penn. Law Rev. 165 (2017), 633--705.
- Will Landecker, Michael D. Thomure, Luís M. A. Bettencourt, Melanie Mitchell, Garrett T. Kenyon, and Steven P. Brumby. 2013. Interpreting individual classifications of hierarchical networks. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’13). IEEE, 32--38.
- Dmitry M. Malioutov, Kush R. Varshney, Amin Emad, and Sanjeeb Dash. 2017. Learning interpretable classification rules with boolean compressed sensing. In Transparent Data Mining for Big and Small Data. Springer, 95--121.
- 10.5555/3091825.3091863 · doi:10.5555/3091825.3091863
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- 10.5555/2717112 · doi:10.5555/2717112
- Michael J. Pazzani, S. Mani, William R. Shankle et al. 2001. Acceptance of rules generated by machine learning among medical experts. Methods Info. Med. 40, 5 (2001), 380--385.
- 10.5555/1597122.1597143 · doi:10.5555/1597122.1597143
- 10.5555/1625015.1625078 · doi:10.5555/1625015.1625078
- 10.5555/152181 · doi:10.5555/152181
- Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. 2016. Programs as black-box explanations. arXiv preprint arXiv:1611.07579 (2016). · arXiv
- Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 353--382.
- Marina M.-C. Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567 (2016). · arXiv
- Tong Wang. 2017. Multi-value rule sets. arXiv preprint arXiv:1710.05257 (2017). · arXiv
- 10.5555/273530.273548 · doi:10.5555/273530.273548
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
“Although some recent surveys [8,10,13–17] summarize the upsurge of activity in XAI across sectors and disciplines, this overview aims to cover the creation of a complete unified framework of categories and concepts that allow for scrutiny and understanding of the field of XAI methods.”
InfluentialBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
“[63] reviewed methods for explaining black-box models at a large scale including data mining as well as machine learning.”
MethodA review on the attention mechanism of deep learning
Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
“Another survey that attempted to categorise the existing explainability methods is this of Guidotti et al. [19].”
Metadata
- Authors
- Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D.
- Affiliations
- University of Pisa (IT)
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- Venue
- ACM Computing Surveys
- Published
- 2018-08-22
- Volume
- 51
- Issue
- 5
- Pages
- 1-42
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2786715987
- Cited by
- 629
- References
- 121
- Citations by year
- 2026: 3 · 2025: 62 · 2024: 82 · 2023: 86 · 2022: 92 · 2021: 145 · 2020: 105 · 2019: 40 · 2018: 13
- FWCI
- 42.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- Scuola Normale Superiore di Pisa (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Horizon 2020 Framework Programme (-2014-2015, 654024)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- explainable AI, black-box models, interpretability, decision support system, explanation methods, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Survey of Methods for Explaining Black Box Models. https://app.cochat.ai/papers/10.1145/3236009
- The paper
- Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019). A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys. https://doi.org/10.1145/3236009
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.