A Survey of Methods for Explaining Black Box Models
Open accessThe DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index; 4 of the 59 sources its claims cite were checked for support.
Abstract
In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Overview
CoChat Summary
TL;DR
This survey classifies methods for explaining black-box machine learning models into four problem types: black box model explanation, black box outcome explanation, black box inspection, and transparent box design. It organizes the literature by problem, explanator, black-box type, and data type, and highlights open challenges around defining and measuring explanation and comprehensibility.
Key findings
- The paper identifies and formally defines four explanation problems: black box model explanation (global interpretable mimic), black box outcome explanation (local explanation for a prediction), black box inspection (visual/textual representation of behavior), and transparent box design (interpretable model by design).
- It classifies explanators into decision trees, decision rules, feature importance, saliency masks, sensitivity analysis, partial dependence plots, prototype selection, and neuron activation; black boxes into neural networks, tree ensembles, SVMs, deep neural networks, and agnostic black boxes; and data into tabular, image, and text.
- Many methods are specific to a problem or black-box type, while agnostic/generalizable approaches use reverse engineering by querying the black box as an oracle; examples include Trepan, inTrees, LIME, and PALM.
- Motivating cases cited include COMPAS ethnic bias (black non-reoffenders classified high risk twice as much as whites who did not re-offend; white repeat offenders classified low risk twice as much as black repeat offenders), credit scoring discordance (in 500,000 records, 29% of consumers received scores differing by at least fifty points among three major US banks), and the St. George’s Hospital screening program that discriminated via inferred ethnicity from surnames and place of birth.
Methods
- Literature survey/review; no primary empirical study. The paper formalizes the general classification problem and defines the four explanation problems as specific cases.
- The classification dimensions are: type of problem faced, type of explanator adopted, type of black box model opened, and type of data used as input by the black box. Table 1 summarizes methods with features including General, Random, Examples, Code, and Dataset.
- The common reverse-engineering approach queries the black box with a test dataset to create an oracle dataset, then trains a comprehensible predictor; approaches are distinguished as generalizable/pedagogical versus not generalizable/decompositional.
Limitations
- The paper states that there is no agreement on what an explanation is, no general/common formalism for defining explanations or properties such as soundness, completeness, compactness, and comprehensibility, and no serious work quantifying human comprehensibility; it also lists open questions on latent features, recommender systems, and learning predictors directly from explanations.
- The paper states it does not address whether the black box can be queried at will versus only a fixed dataset being available, or whether the complete feature set is known, because there is insufficient literature.
- Evident from the text: the survey scope is limited to tabular, image, and text data, with no work addressing interpretability for sequence, spatio-temporal, or complex network data; many reviewed methods are ad hoc or specific, only a few consider user expertise, and no work provides real experiments on the time required to understand an explanation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 13 Not valid · 38 Not checked · 49 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and no one provides real experiments about the time required to understand an explanation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The works reviewed in the literature only implicitly specify if their proposal is global or local.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Just a few of them take into account the nature of user expertise [29,84,87]”
What the sources say
- Not checked yetcites [29] 29. R. D. Gibbons, G. Hooker, M. D. Finkelman, D. J. Weiss, P. A. Pilkonis, E. Frank, T. Moore, and D. J. Kupfer. The cad-mdd: a computerized adaptive diagnostic screening tool for depression. The Journal of clinical psychiatry , 74(7):669, 2013. source
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
- Not checked yetcites [87] 87. V. Schetinin, J. E. Fieldsend, D. Partridge, T. J. Coats, W. J. Krzanowski, R. M. Everson, T. C. Bailey, and A. Hernandez. Confident interpretation of bayesian decision tree ensembles for clinical applications. IEEE Transactions on Information Technology in Biomedicine , 11(3):312-319, 2007. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In the following, we list all the black boxes opened in the reviewed papers. These black boxes are all supervised learning algorithm designed to solve a classification problem [95].”
What the sources say
- Not checked yetcites [95] 95. P.-N. Tan et al. Introduction to data mining . Pearson Education India, 2006.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, this is not always the case in practice [84].”
What the sources say
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 104 claims and to verify the 46 references still open.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 65
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While “black‐box” algorithms raise complex opaqueness and interpretability challenges, simpler interpretable ML algorithms do not, but have generally also been regarded as less powerful (Guidotti et al. 2019; see however, Rudin 2019).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Guidotti et al. 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since we could not find a complete and recent overview of explanation types, we reviewed recent XAI surveys [14,58,167,280] and adapted and extended the categories identified by Guidotti et al. [86] with these insights.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, there is no consensus on what a good XAD technique should be.”
From A Survey on Explainable Anomaly Detection (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Among the main impediments prohibiting banking institutions from implementing their AI strategy is the ''black-box'' paradox [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The models are black boxes due to their significant non-linearity and intricacy, making it unfeasible for individuals to fathom their fundamental operational processes and judgment mechanisms [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Problem Formulation · p. 13
“for the comprehensible predictor c we get the set of predictions ¯ Y = ⋃ x ∈ X c ( x )”
§ 4.1 Problem Formulation · p. 13
“where as target value is used the prediction ¯ Y of the black box b instead of the real values Y”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a single representat…”
BackgroundLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 5×
“al. 2017 Model Expl. DT AGN ANY X X X - [97] Tolomei et al. 2017 Model Expl. FI TE TAB X - [108] Xu et al. 2015 Outcome Expl. SM DNN IMG X X X - [25] Fong et al. 2017 Outcome Expl. SM DNN IMG X X CAM [113] Zhou et al. 2016 Outcome Expl. SM DNN IMG X X X Grad-CAM [89] Selvaraju et al. 2016 Outcome Expl. SM DNN IMG X X…”
BackgroundMethodInfluentialShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Reference [134] introduces an attention-based model f , which automatically identifies the contents of an image.”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“In Reference [116] it is shown the alteration of an image (e.”
Background(2008) · Data Mining and Market Intelligence for Optimal Marketing Returns · doi:10.4324/9780080878096-12 · Cited by 6,954
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundThe Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375 · cited 1×
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
BackgroundDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 3×
“However, in Reference [109] is proposed a method f for visualizing saliency mask εl specialized for CNN black boxes b.”
BackgroundMethodOn Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 2×
“to overcome a functional approach, where the explanation results form the local analysis of the prediction function [109], and where the explanation is obtained by running a backward pass in that graph [7, 137].”
BackgroundMethodAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 1×
“Reference [115] studies the problem of attributing the prediction of a DNN (the black boxb) to its input features.”
BackgroundStriving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 1×
“Explaining the outcome of black boxes is not the main topic of Reference [112], as the paper is more focused in showing that homogeneous and not complicated CNN can reach the state of art performance.”
BackgroundDeep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Nguyen, A., Yosinski, J., & Clune, J. (2014) · arXiv · doi:10.48550/arxiv.1412.1897 · Cited by 3,258
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Thus, to realize an interpretable model, it is necessary to take into account the following list of desiderata, which are mentioned by a set of papers in the state of art [5, 28, 32, 45]:”
BackgroundMethodInfluentialUnderstanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 1×
“In Reference [138] it is shown how conventional regularization and small generalization error fail to explain why DNNs generalize well in practice.”
BackgroundLearning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
Quinlan, J. R. (1987) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(87)80053-6 · Cited by 2,553
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“An extension of LIME using decision rules as local interpretable classifier cl is presented in Reference [99].”
MethodGoldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861 · cited 1×
“provide in Reference [35] a technique f that extends classical PDP named Individual Conditional Expectation (ICE) to visualize the model approximated by a black box b that helps in visualizing the average partial relationship between the outcome and some features.”
MethodUnderstanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827 · cited 5×
“113], [89], [56], [83], [98] Black Box Inspection [71], [7], [92], [16], [35], [30], [48], [2], [1], [110], [96], [90], [81] Transparent Design [109], [102], [57], [91], [53], [104], [64], [8], [44], [63], [47], [103] Open The Black Box Survey 39 A Supplementary Materials Acknowledgement This work is partially support…”
BackgroundMethodInfluentialIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 1×
“A case study on health care showing the application of the GAM the refinement is presented in Reference [16].”
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
On the interpretation of weight vectors of linear models in multivariate neuroimaging
Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., & Bießmann, F. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.067 · Cited by 1,389 · cited 6×
“In particular, the f described in Reference [39] enables the neurophysiological interpretation of the parameters of nonlinear models.”
BackgroundMethodInfluentialUnderstanding Neural Networks Through Deep Visualization
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,358 · cited 1×
“In Reference [136] are proposed two tools for visualizing and interpreting DNNs and for understanding what computations DNNs perform at intermediate layers and which neurons are activated.”
MethodExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
The Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308 · cited 1×
“Despite divergent opinions among legal scholars regarding the real scope of these clauses [36, 74, 126], there is a general agreement on the need for the implementation of such a principle is urgent and that it represents today a huge open scientific challenge.”
BackgroundSensitivity Analysis for Importance Assessment
Saltelli, A. (2002) · Risk Analysis · doi:10.1111/0272-4332.00040 · Cited by 1,302 · cited 1×
“Sensitivity analysis studies the correlation between the uncertainty in the output of a predictor and that one in its inputs [102].”
BackgroundGenerating Accurate Rule Sets Without Global Optimization
Frank, E., & Witten, I. H. (1998) · Research Commons (University of Waikato) · doi:10.5555/645527.657305 · Cited by 1,269
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
Olden, J. D., & Jackson, D. A. (2002) · Ecological Modelling · doi:10.1016/s0304-3800(02)00064-9 · Cited by 1,223
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164 · cited 1×
“In [103] it is presented a comparison between explanations provided with LRP and more traditional deconvolution and sensitivity analysis.”
MethodInterpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
CPAR: Classification based on Predictive Association Rules
Yin, X., & Han, J. (2003) · Proceedings of the 2003 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972733.40 · Cited by 833
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · cited 1×
“Finally, with respect to texts, in Reference [65] the authors develop an approach f that incorporates rationales as part of the learning process of b.”
BackgroundLetham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728
Wettschereck, D., Aha, D. W., & Mohri, T. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006593614256 · Cited by 717 · cited 2×
“Indeed, rule-based classifiers have the disadvantage of requiring an additional approach for resolving such situations of conflicting outcome [133].”
BackgroundModel-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687
Discrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676 · cited 1×
“Training a classifier on historical datasets, reporting human decisions, could lead to the discovery of endemic preconceptions [88].”
BackgroundSelf-citationComprehensible classification models: a position paper
Freitas, A. A. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2594473.2594475 · Cited by 572
Generating Production Rules From Decision Trees
Quinlan, J. R. (1987) · Cited by 565
Accurate intelligible models with pairwise interactions
Lou, Y., Caruana, R., Gehrke, J., & Hooker, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487579 · Cited by 564 · cited 1×
“A refinement of the GAM approach is proposed by the same authors in Reference [70].”
Method(2016) · neural information processing systems · Cited by 556
Intelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547 · cited 1×
“Probably the first attempt of an agnostic solution was proposed in Reference [69].”
BackgroundQuinlan, J. R., & Cameron-Jones, R. M. (1993) · Lecture Notes in Computer Science · doi:10.1007/3-540-56602-3_124 · Cited by 528 · cited 1×
“Indeed, following the basic idea of FOIL [94], CPAR does not generate a large set of candidates, as in associative classification, and applies a greedy approach for generating rules cд directly from training data.”
Visualizing Deep Convolutional Neural Networks Using Natural Pre-images
Mahendran, A., & Vedaldi, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0911-8 · Cited by 496 · cited 1×
“Moreover, in Reference [73] this work is extended by comparing various family of methods analyzed for understanding “representations.”
BackgroundVisualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487 · cited 1×
“In Reference [143] the authors present a probabilistic methodology f for explaining classification decisions made by DNN.”
MethodComprehensible credit scoring models using rule extraction from support vector machines
Martens, D., Baesens, B., Van Gestel, T., & Vanthienen, J. (2007) · European Journal of Operational Research · doi:10.1016/j.ejor.2006.04.051 · Cited by 448 · cited 3×
“In Reference [76] the authors propose a qualitative comparison of the explanations returned by techniques for extraction of rules from SVM black boxes (e.”
BackgroundMethodInfluentialUsing sensitivity analysis and visualization techniques to open black box data mining models
Cortez, P., & Embrechts, M. J. (2013) · Information Sciences · doi:10.1016/j.ins.2012.10.039 · Cited by 443 · cited 6×
“References [97] and [96] propose some extensions of LIME with an analysis of particular aspects and cases not overcoming the aforementioned limitations.”
BackgroundMethodInfluentialPrediction and explanation in social systems
Hofman, J. M., Sharma, A., & Watts, D. J. (2017) · Science · doi:10.1126/science.aal3856 · Cited by 429 · cited 1×
“results, and a systematic organization and classication of these methodologies is missing. Many questions feed the papers in the literature proposing methodologies for interpreting black box systems [106,34]: What Open The Black Box Survey 3 does it mean that a model is interpretable or transparent? What is an explana…”
BackgroundHuysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425 · cited 4×
“However, other works focus on particular aspects like the impact of representation formats on comprehensibility [43] or the interpretability issues in term of advantages and disadvantages of selected predictive models [32].”
BackgroundInfluentialInteracting with Predictions: Visual Inspection of Black-box Machine Learning Models
Krause, J., Perer, A., & Ng, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858529 · Cited by 390
An Efficient Explanation of Individual Classifications using Game Theory
StrumbeljErik, & KononenkoIgor (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1756007 · Cited by 380 · cited 2×
“Then, in Reference [29], Strumbelj et al.”
MethodInterpretable deep neural networks for single-trial EEG classification
Sturm, I., Lapuschkin, S., Samek, W., & Müller, K.-R. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.008 · Cited by 374 · cited 3×
“The only exceptions are [39, 113] that using EEG data provide heatmaps with data point’s relevance for the decision’s outcome.”
BackgroundMethodDeep Text Classification Can be Fooled
Liang, B., Li, H., Su, M., Bian, P., Li, X., & Shi, W. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/585 · Cited by 370 · cited 2×
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
BackgroundMethodBuilding comprehensible customer churn prediction models with advanced rule induction techniques
Verbeke, W., Martens, D., Mues, C., & Baesens, B. (2011) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.08.023 · Cited by 363 · cited 1×
“The degree of trust on a model increases if the model is built by respecting constraints of monotonicity given by the users [77, 87, 123].”
BackgroundThe Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification
Kim, B., Rudin, C. D., & Shah, J. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 340
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 2×
“Finally, it is worth mentioning that [81] presents the discovery that a single neuron unit of a DNN can perform alone a sentiment analysis task after the training of the network reaching the same level of performance of strong baselines.”
BackgroundA multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288 · cited 1×
“The first principle requires that the model guarantees the protection of groups against (direct or indirect) discrimination [100]; while the second one requires that the model does not reveal sensitive information about people [4].”
BackgroundSelf-citationHOGgles: Visualizing Object Detection Features
Vondrick, C., Khosla, A., Malisiewicz, T., & Torralba, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.8 · Cited by 282 · cited 1×
“elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a single representat…”
BackgroundAuditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280
Auditing Black-Box Models for Indirect Influence
Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0011 · Cited by 280
Malgieri, G., & Comandé, G. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx019 · Cited by 271 · cited 1×
“Despite divergent opinions among legal scholars regarding the real scope of these clauses [36, 74, 126], there is a general agreement on the need for the implementation of such a principle is urgent and that it represents today a huge open scientific challenge.”
BackgroundQUINLAN, J. R. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1987.0321 · Cited by 268 · cited 1×
“Indeed, a decision tree can be linearized into a set of decision rules with the if-then form [77,78,26]:”
BackgroundPrototype selection for interpretable classification
Bien, J., & Tibshirani, R. (2011) · The Annals of Applied Statistics · doi:10.1214/11-aoas495 · Cited by 245 · cited 1×
“Design DR - TAB X X PS [8] Bien et al.”
BackgroundExplanation in Recommender Systems
Mcsherry, D. (2005) · Artificial Intelligence Review · doi:10.1007/s10462-005-4612-x · Cited by 182 · cited 1×
“arch question is related to providing explanations in the eld of recommender systems. When a suggestion is provided to a customer, it should come together with the reasons for this recommendation. In [67] the authors dene a case-based reasoning approach to generate recommendations with the opportunity of obtaining bot…”
BackgroundReverse Engineering the Neural Networks for Rule Extraction in Classification Problems
Augasta, M. G., & Kathirvalavakumar, T. (2012) · Neural Processing Letters · doi:10.1007/s11063-011-9207-8 · Cited by 173
The Feature Importance Ranking Measure
Zien, A., Krämer, N., Sonnenburg, S., & Rätsch, G. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04174-7_45 · Cited by 171 · cited 2×
“In Reference [142] is introduced FIRM (Feature Importance Ranking Measure) for implementing function f as an extension of the POIMs [111].”
BackgroundMethodTrading accuracy for simplicity in decision trees
Bohanec, M., & Bratko, I. (1994) · Machine Learning · doi:10.1007/bf00993345 · Cited by 170 · cited 1×
“In the decision tree literature, the problems of over-fitting the training data and of “trading accuracy for simplicity” [10] have been addressed by a class of post-processing algorithms called”
MethodRule Extraction from Support Vector Machines
(2006) · Advanced Information and Knowledge Processing · doi:10.1007/3-540-28803-1_10 · Cited by 167
(2015) · International Conference on Artificial Intelligence and Statistics · Cited by 162
Reconstructing an image from its local descriptors
Weinzaepfel, P., Jegou, H., & Perez, P. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995616 · Cited by 159 · cited 1×
“elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a single representat…”
BackgroundRule extraction from linear support vector machines
Fung, G., Sandilya, S., & Rao, R. B. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081878 · Cited by 154 · cited 2×
“, in Reference [33], describe as function f an algorithm based on constraint programming for converting linear SVM b (and other hyperplane-based linear classifiers) into a set of non”
MethodPerformance of classification models from a user perspective
Martens, D., Vanthienen, J., Verbeke, W., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2011.01.013 · Cited by 153
Knowledge discovery via multiple models
DOMINGOS, P. (1998) · Intelligent Data Analysis · doi:10.1016/s1088-467x(98)00023-7 · Cited by 140
The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098066 · Cited by 138
Extracting decision trees from trained neural networks
Krishnan, R., Sivakumar, G., & Bhattacharya, P. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00181-2 · Cited by 134 · cited 4×
“g explanations rather than the logs of previous plays. Table 3. Summary of methods for opening and explaining black boxes with respect to the problem faced. Problem References Model Explanation [20], [50], [9], [39], [14], [29], [114], [87], [32], [94], [5], [19], [37], [115], [6], [70], [28], [65], [22], [61], [33],…”
BackgroundMethodInfluentialA comprehensive review on privacy preserving data mining
Aldeen, Y. A. A. S., Salleh, M., & Razzaque, M. A. (2015) · SpringerPlus · doi:10.1186/s40064-015-1481-x · Cited by 129 · cited 1×
“The first principle requires that the model guarantees the protection of groups against (direct or indirect) discrimination [100]; while the second one requires that the model does not reveal sensitive information about people [4].”
BackgroundInterpretable Predictions of Tree-based Ensembles via Actionable Feature Tweaking
Tolomei, G., Silvestri, F., Haines, A., & Lalmas, M. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098039 · Cited by 129 · cited 1×
“In Reference [121], a solution for the model explanation problem is presented.”
Discovering additive structure in black box functions
Hooker, G. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014122 · Cited by 124 · cited 1×
“In Reference [42], the authors present an approach f aimed at evaluating the importance of non-additive interactions between any set of features.”
BackgroundMind the Gap: a generative approach to interpretable feature selection and extraction
Kim, B., Shah, J., & Doshi‐Velez, F. (2015) · neural information processing systems · Cited by 122
Transparency: Motivations and ChallengesPreprint
Weller, A. (2017) · arXiv (Cornell University) · doi:10.17863/cam.101505 · Cited by 110
Artificial Intelligence and Legal Liability
Kingston, J. K. C. (2016) · Research and Development in Intelligent Systems XXXIII · doi:10.1007/978-3-319-47175-4_20 · Cited by 103 · cited 1×
“This impacts not only information on ethics but also on accountability [59], on safety [23], and on industrial liability [53].”
BackgroundGibbons, R. D., Hooker, G., Finkelman, M. D., Weiss, D. J., Pilkonis, P. A., Frank, E., Moore, T., & Kupfer, D. J. (2013) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.12m08338 · Cited by 101
A peek into the black box: exploring classifiers by randomization
Henelius, A., Puolamäki, K., Boström, H., Asker, L., & Papapetrou, P. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0368-8 · Cited by 98
ID2-of-3: Constructive Induction of M-of-N Concepts for Discriminators in Decision Trees
Murphy, P. M., & Pazzani, M. J. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50040-4 · Cited by 98 · cited 1×
“ke into consideration only rules with conjunctions. Other types of rules are: m-of-n rules where given a set of nconditions if mof them are veried then the consequence of the rule is considered true [68]; list of rules where given an ordered set of rules is considered true the consequent of the rst rule which is verie…”
BackgroundVisualizations of Deep Neural Networks in Computer Vision: A Survey
Seifert, C., Aamir, A., Balagopalan, A., Jain, D., Sharma, A., Grottel, S., & Gumhold, S. (2017) · Studies in big data · doi:10.1007/978-3-319-54024-5_6 · Cited by 91
Learning From Explanations Using Sentiment and Advice in RL
Krening, S., Harrison, B., Feigh, K. M., Isbell, C. L., Riedl, M., & Thomaz, A. (2017) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2016.2628365 · Cited by 84
PALM: Machine Learning Explanations For Iterative Debugging
Krishnan, S., & Wu, E. (2017) · Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/3077257.3077271 · Cited by 81
Image Reconstruction from Bag-of-Visual-Words
Kato, H., & Harada, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.127 · Cited by 81 · cited 1×
“elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a single representat…”
BackgroundTurner, R. (2016) · 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP) · doi:10.1109/mlsp.2016.7738872 · Cited by 75 · cited 4×
“N IMG X X X Grad-CAM [89] Selvaraju et al. 2016 Outcome Expl. SM DNN IMG X X X - [56] Lei et al. 2016 Outcome Expl. SM DNN TXT X X LIME [83] Ribeiro et al. 2016 Outcome Expl. FI AGN ANY X X X X X MES [98] Turner et al. 2016 Outcome Expl. DR AGN ANY X X X NID [71] Olden et al. 2002 Inspection SA NN TAB X GDP [7] Baehre…”
BackgroundMethodInfluentialExtracting decision trees from trained neural networks
Boz, O. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775113 · Cited by 71 · cited 3×
“DT NN TAB X X X DecText [9] Boz 2002 Model Expl.”
MethodInfluentialSonnenburg, S., Zien, A., Philips, P., & Rätsch, G. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn170 · Cited by 64 · cited 3×
“The problem faced in Reference [111] is specific for the classification of DNA sequences and k-mers.”
BackgroundMethodConfident Interpretation of Bayesian Decision Tree Ensembles for Clinical Applications
Schetinin, V., Fieldsend, J. E., Partridge, D., Coats, T. J., Krzanowski, W. J., Everson, R. M., Bailey, T. C., & Hernandez, A. (2007) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2006.880553 · Cited by 64
Evolving decision trees using oracle guides
Johansson, U., & Niklasson, L. (2009) · 2009 IEEE Symposium on Computational Intelligence and Data Mining · doi:10.1109/cidm.2009.4938655 · Cited by 63 · cited 4×
“ons rather than the logs of previous plays. Table 3. Summary of methods for opening and explaining black boxes with respect to the problem faced. Problem References Model Explanation [20], [50], [9], [39], [14], [29], [114], [87], [32], [94], [5], [19], [37], [115], [6], [70], [28], [65], [22], [61], [33], [51], [97]…”
BackgroundInfluentialThe Truth is in There : Rule Extraction from Opaque Models Using Genetic Programming
Johansson, U., König, R., & Niklasson, L. (2004) · Cited by 62
Regulating Autonomous Systems: Beyond Standards
Danks, D., & London, A. J. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.1 · Cited by 61 · cited 1×
“This impacts not only information on ethics but also on accountability [59], on safety [23], and on industrial liability [53].”
BackgroundMaking Tree Ensembles Interpretable
Hara, S., & Hayashi, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05390 · Cited by 60 · cited 1×
“In Reference [38] the complexity is identified by the number of regions, i.”
BackgroundUsing Data Mining for Wine Quality Assessment
Cortez, P., Teixeira, J., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04747-3_8 · Cited by 60 · cited 1×
“Then, the visualizations r realized are barplots for the features importance, and Variable Effect Characteristic curve (VEC) [20] plotting the input values (x-axis) versus the (average) outcome responses (see Figure 14(left)).”
Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance
Ribeiro, M. T., Singh, S. K., & Guestrin, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05817 · Cited by 59 · cited 7×
“The main intuition of LIME is that the explanation may be derived locally from the records generated randomly in the neighborhood of the record to be explained, and weighted according to their proximity to it.”
BackgroundInfluentialLowry, S., & Macpherson, G. (1988) · BMJ · doi:10.1136/bmj.296.6623.657 · Cited by 56 · cited 1×
“However, the program was found to unfairly discriminate against ethnic minorities and women by inferring this information from surnames and place of birth, and lowering their chances of being selected for interview [71].”
BackgroundHow to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54 · cited 1×
“In Reference [8] the authors propose a procedure based on Gaussian Process Classification (GDP), which allows explaining the decisions of any classification method through an explanation vector.”
MethodAccuracy vs. comprehensibility in data mining models
Johansson, U., Niklasson, L., & König, R. (2004) · Cited by 49
Interpreting Tree Ensembles with inTreesPreprint
Deng, H. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5456 · Cited by 47 · cited 2×
“In Reference [25] the complexity of a rule (and thus of an explanation) is measured by the length of the rule condition, defined as the number of attribute-value pairs in the condition.”
BackgroundMethodBayesian Rule Sets for Interpretable Classification
Wang, T., Rudin, C., Velez-Doshi, F., Liu, Y., Klampfl, E., & MacNeille, P. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0171 · Cited by 45
TreeView: Peeking into Deep Neural Networks Via Feature-Space Partitioning
Thiagarajan, J. J., Kailkhura, B., Sattigeri, P., & Ramamurthy, K. N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07429 · Cited by 44 · cited 1×
“Reference [119] shows the extraction of a visual interpretation r of a DNN using a decision tree.”
MethodTrading Interpretability for Accuracy: Oblique Treed Sparse Additive Models
Wang, J., Fujimaki, R., & Motohashi, Y. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783407 · Cited by 42
Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models
Adebayo, J., & Kagal, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04967 · Cited by 42 · cited 3×
“2016 Inspection PDP AGN TAB X X X X OPIA [1] Adebayo et al.”
BackgroundMethodInfluentialInterpreting Models via Single Tree Approximation
Zhou, Y., & Hooker, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09036 · Cited by 39
Tree Space Prototypes: Another Look at Making Tree Ensembles Interpretable
Tan, S., Soloviev, M., Hooker, G., & Wells, M. T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07115 · Cited by 36
(2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 34
Improving the interpretability of classification rules discovered by an ant colony algorithm
Otero, F. E. B., & Freitas, A. A. (2013) · Proceedings of the 15th annual conference on Genetic and evolutionary computation · doi:10.1145/2463372.2463382 · Cited by 34 · cited 1×
“In this kind of model, a more honest measure could be the average number of conditions evaluated to classify a set of test records [84].”
BackgroundInterpretable Two-level Boolean Rule Learning for Classification
Su, G., Wei, D., Varshney, K. R., & Malioutov, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05798 · Cited by 24
Interpretable & Explorable Approximations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01154 · Cited by 23 · cited 1×
“In [54] the authors propose a method f exploiting a two-level boolean rule predictor to”
MethodPredicting post-synaptic activity in proteins with data mining
Pappa, G. L., Baines, A. J., & Freitas, A. A. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti1102 · Cited by 20 · cited 1×
“Even though this sort of exception analysis is hardly performed, it can be more informative than the direct explanation, and it can also provide clues about the application domain [37, 85].”
BackgroundInterpretable Two-level Boolean Rule Learning for Classification
Su, G., Wei, D., Varshney, K. R., & Malioutov, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07361 · Cited by 14
Extracting symbolic rules from trained neural network ensembles
ZhouZhi-Hua, JiangYuan, & ChenShi-Fu (2003) · AI Communications · doi:10.5555/859130.859132 · Cited by 5
Subtree Replacement in Decision Tree Simplification
Ruggieri, S. (2012) · Proceedings of the 2012 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972825.33 · Cited by 5 · cited 1×
“The most well-known simplification method is decision tree pruning, which relies on an error estimation function to compare the errors of the two alternatives: to prune a subtree or not [101].”
MethodSelf-citationKnowledge Discovery Via Multiple Models
DomingosPedro (1998) · doi:10.5555/2639331.2639334 · Cited by 4
Opening the Black Box of Deep Neural Networks via Information
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv · doi:10.48550/arxiv.1703.00810 · cited 1×
“in Reference [108] proposes f that shows the effectiveness of the Information Plane visualization of DNNs r , by highlighting that the empirical error minimization of each stochastic gradient descent phase epoch is always followed by a slow representation compression.”
BackgroundRationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · arXiv · doi:10.48550/arxiv.1606.04155 · cited 1×
“Finally, with respect to texts, in Reference [65] the authors develop an approach f that incorporates rationales as part of the learning process of b.”
BackgroundTransparency: Motivations and Challenges
Weller, A. (2017) · arXiv · doi:10.48550/arxiv.1708.01870 · cited 1×
“Many questions feed the papers in the literature proposing methodologies for interpreting black box systems [106,34]: What”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
Selvaraju, R. R., Cogswell, M., Das, A., et al. (2016) · arXiv · doi:10.48550/arxiv.1610.02391 · cited 2×
“Their solution is not black box agnostic limited to NN, but it requires specific architectural modifications [139] or access to intermediate layers [106].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1607.02533 · cited 1×
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
BackgroundSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2016) · arXiv · doi:10.48550/arxiv.1608.07187
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R., & Vedaldi, A. (2017) · arXiv · doi:10.48550/arxiv.1704.03296 · cited 2×
“(Right) From Reference [30] the mask and the level of accuracy on the image considering and”
BackgroundDeep Text Classification Can be Fooled
Liang, B., Li, H., Su, M., Bian, P., Li, X., & Shi, W. (2017) · arXiv · doi:10.48550/arxiv.1704.08006 · cited 2×
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
BackgroundMethod
Research works printed but not linked: 31
- 10.5555/645920.672836 · doi:10.5555/645920.672836
- Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, and Karol Zieba. 2016. VisualBackProp: Visualizing CNNs for autonomous driving. CoRR, Vol. abs/1611.05418 (2016). · arXiv
- Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press.
- Carolyn Carter, Elizabeth Renuart, Margot Saunders, and Chi Chi Wu. 2006. The credit card market and regulation: In need of repair. NC Bank. Inst. 10 (2006), 23.
- H. A. Chipman, E. I. George, and R. E. McCulloh. 1998. Making sense of a forest of trees. In Proceedings of the 30th Symposium on the Interface, S. Weisberg (Ed.). Fairfax Station, VA: Interface Foundation of North America, 84--92.
- Paulo Cortez and Mark J. Embrechts. 2011. Opening black box data mining models using sensitivity analysis. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’11). IEEE, 341--348.
- 10.5555/3091574.3091580 · doi:10.5555/3091574.3091580
- 10.5555/2998828.2998832 · doi:10.5555/2998828.2998832
- 10.5555/3000292.3000299 · doi:10.5555/3000292.3000299
- Bryce Goodman and Seth Flaxman. 2016. EU regulations on algorithmic decision-making and a “right to explanation.” In Proceedings of the ICML Workshop on Human Interpretability in Machine Learning (WHI’16). Retrieved from http://arxiv. org/abs/1606.08813 v1.
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018). · arXiv
- U. Johansson, R. König, and L. Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In Proceedings of the 13th International Conference on Artificial Neural Networks. 13--16.
- 10.5555/3157096.3157352 · doi:10.5555/3157096.3157352
- 10.5555/2969033.2969045 · doi:10.5555/2969033.2969045
- 10.5555/2969442.2969492 · doi:10.5555/2969442.2969492
- Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable algorithms. U. Penn. Law Rev. 165 (2017), 633--705.
- Will Landecker, Michael D. Thomure, Luís M. A. Bettencourt, Melanie Mitchell, Garrett T. Kenyon, and Steven P. Brumby. 2013. Interpreting individual classifications of hierarchical networks. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’13). IEEE, 32--38.
- Dmitry M. Malioutov, Kush R. Varshney, Amin Emad, and Sanjeeb Dash. 2017. Learning interpretable classification rules with boolean compressed sensing. In Transparent Data Mining for Big and Small Data. Springer, 95--121.
- 10.5555/3091825.3091863 · doi:10.5555/3091825.3091863
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- 10.5555/2717112 · doi:10.5555/2717112
- Michael J. Pazzani, S. Mani, William R. Shankle et al. 2001. Acceptance of rules generated by machine learning among medical experts. Methods Info. Med. 40, 5 (2001), 380--385.
- 10.5555/1597122.1597143 · doi:10.5555/1597122.1597143
- 10.5555/1625015.1625078 · doi:10.5555/1625015.1625078
- 10.5555/152181 · doi:10.5555/152181
- Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. 2016. Programs as black-box explanations. arXiv preprint arXiv:1611.07579 (2016). · arXiv
- Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 353--382.
- Marina M.-C. Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567 (2016). · arXiv
- Tong Wang. 2017. Multi-value rule sets. arXiv preprint arXiv:1710.05257 (2017). · arXiv
- 10.5555/273530.273548 · doi:10.5555/273530.273548
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
“Although some recent surveys [8,10,13–17] summarize the upsurge of activity in XAI across sectors and disciplines, this overview aims to cover the creation of a complete unified framework of categories and concepts that allow for scrutiny and understanding of the field of XAI methods.”
InfluentialBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
“[63] reviewed methods for explaining black-box models at a large scale including data mining as well as machine learning.”
MethodA review on the attention mechanism of deep learning
Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
“Another survey that attempted to categorise the existing explainability methods is this of Guidotti et al. [19].”
Metadata
- Authors
- Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D.
- Affiliations
- University of Pisa (IT)
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- Venue
- ACM Computing Surveys
- Published
- 2018-08-22
- Volume
- 51
- Issue
- 5
- Pages
- 1-42
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2786715987
- Cited by
- 629
- References
- 121
- Citations by year
- 2026: 3 · 2025: 62 · 2024: 82 · 2023: 86 · 2022: 92 · 2021: 145 · 2020: 105 · 2019: 40 · 2018: 13
- FWCI
- 42.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- Scuola Normale Superiore di Pisa (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Horizon 2020 Framework Programme (-2014-2015, 654024)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- explainable AI, black-box models, interpretability, decision support system, explanation methods, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.