A Survey of Methods for Explaining Black Box Models
Open accessThe DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index; 4 of the 59 sources its claims cite were checked for support.
Abstract
In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Overview
CoChat Summary
TL;DR
This survey classifies methods for explaining black-box machine learning models into four problem types: black box model explanation, black box outcome explanation, black box inspection, and transparent box design. It organizes the literature by problem, explanator, black-box type, and data type, and highlights open challenges around defining and measuring explanation and comprehensibility.
Key findings
- The paper identifies and formally defines four explanation problems: black box model explanation (global interpretable mimic), black box outcome explanation (local explanation for a prediction), black box inspection (visual/textual representation of behavior), and transparent box design (interpretable model by design).
- It classifies explanators into decision trees, decision rules, feature importance, saliency masks, sensitivity analysis, partial dependence plots, prototype selection, and neuron activation; black boxes into neural networks, tree ensembles, SVMs, deep neural networks, and agnostic black boxes; and data into tabular, image, and text.
- Many methods are specific to a problem or black-box type, while agnostic/generalizable approaches use reverse engineering by querying the black box as an oracle; examples include Trepan, inTrees, LIME, and PALM.
- Motivating cases cited include COMPAS ethnic bias (black non-reoffenders classified high risk twice as much as whites who did not re-offend; white repeat offenders classified low risk twice as much as black repeat offenders), credit scoring discordance (in 500,000 records, 29% of consumers received scores differing by at least fifty points among three major US banks), and the St. George’s Hospital screening program that discriminated via inferred ethnicity from surnames and place of birth.
Methods
- Literature survey/review; no primary empirical study. The paper formalizes the general classification problem and defines the four explanation problems as specific cases.
- The classification dimensions are: type of problem faced, type of explanator adopted, type of black box model opened, and type of data used as input by the black box. Table 1 summarizes methods with features including General, Random, Examples, Code, and Dataset.
- The common reverse-engineering approach queries the black box with a test dataset to create an oracle dataset, then trains a comprehensible predictor; approaches are distinguished as generalizable/pedagogical versus not generalizable/decompositional.
Limitations
- The paper states that there is no agreement on what an explanation is, no general/common formalism for defining explanations or properties such as soundness, completeness, compactness, and comprehensibility, and no serious work quantifying human comprehensibility; it also lists open questions on latent features, recommender systems, and learning predictors directly from explanations.
- The paper states it does not address whether the black box can be queried at will versus only a fixed dataset being available, or whether the complete feature set is known, because there is insufficient literature.
- Evident from the text: the survey scope is limited to tabular, image, and text data, with no work addressing interpretability for sequence, spatio-temporal, or complex network data; many reviewed methods are ad hoc or specific, only a few consider user expertise, and no work provides real experiments on the time required to understand an explanation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 13 Not valid · 38 Not checked · 49 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and no one provides real experiments about the time required to understand an explanation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The works reviewed in the literature only implicitly specify if their proposal is global or local.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Just a few of them take into account the nature of user expertise [29,84,87]”
What the sources say
- Not checked yetcites [29] 29. R. D. Gibbons, G. Hooker, M. D. Finkelman, D. J. Weiss, P. A. Pilkonis, E. Frank, T. Moore, and D. J. Kupfer. The cad-mdd: a computerized adaptive diagnostic screening tool for depression. The Journal of clinical psychiatry , 74(7):669, 2013. source
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
- Not checked yetcites [87] 87. V. Schetinin, J. E. Fieldsend, D. Partridge, T. J. Coats, W. J. Krzanowski, R. M. Everson, T. C. Bailey, and A. Hernandez. Confident interpretation of bayesian decision tree ensembles for clinical applications. IEEE Transactions on Information Technology in Biomedicine , 11(3):312-319, 2007. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In the following, we list all the black boxes opened in the reviewed papers. These black boxes are all supervised learning algorithm designed to solve a classification problem [95].”
What the sources say
- Not checked yetcites [95] 95. P.-N. Tan et al. Introduction to data mining . Pearson Education India, 2006.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, this is not always the case in practice [84].”
What the sources say
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 104 claims and to verify the 46 references still open.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 65
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While “black‐box” algorithms raise complex opaqueness and interpretability challenges, simpler interpretable ML algorithms do not, but have generally also been regarded as less powerful (Guidotti et al. 2019; see however, Rudin 2019).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Guidotti et al. 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since we could not find a complete and recent overview of explanation types, we reviewed recent XAI surveys [14,58,167,280] and adapted and extended the categories identified by Guidotti et al. [86] with these insights.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, there is no consensus on what a good XAD technique should be.”
From A Survey on Explainable Anomaly Detection (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Among the main impediments prohibiting banking institutions from implementing their AI strategy is the ''black-box'' paradox [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The models are black boxes due to their significant non-linearity and intricacy, making it unfeasible for individuals to fathom their fundamental operational processes and judgment mechanisms [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Problem Formulation · p. 13
“for the comprehensible predictor c we get the set of predictions ¯ Y = ⋃ x ∈ X c ( x )”
§ 4.1 Problem Formulation · p. 13
“where as target value is used the prediction ¯ Y of the black box b instead of the real values Y”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a single representat…”
BackgroundLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 5×
“al. 2017 Model Expl. DT AGN ANY X X X - [97] Tolomei et al. 2017 Model Expl. FI TE TAB X - [108] Xu et al. 2015 Outcome Expl. SM DNN IMG X X X - [25] Fong et al. 2017 Outcome Expl. SM DNN IMG X X CAM [113] Zhou et al. 2016 Outcome Expl. SM DNN IMG X X X Grad-CAM [89] Selvaraju et al. 2016 Outcome Expl. SM DNN IMG X X…”
BackgroundMethodInfluentialShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Reference [134] introduces an attention-based model f , which automatically identifies the contents of an image.”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“In Reference [116] it is shown the alteration of an image (e.”
Background(2008) · Data Mining and Market Intelligence for Optimal Marketing Returns · doi:10.4324/9780080878096-12 · Cited by 6,954
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundThe Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375 · cited 1×
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
Background
Research works printed but not linked: 31
- 10.5555/645920.672836 · doi:10.5555/645920.672836
- Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, and Karol Zieba. 2016. VisualBackProp: Visualizing CNNs for autonomous driving. CoRR, Vol. abs/1611.05418 (2016). · arXiv
- Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press.
- Carolyn Carter, Elizabeth Renuart, Margot Saunders, and Chi Chi Wu. 2006. The credit card market and regulation: In need of repair. NC Bank. Inst. 10 (2006), 23.
- H. A. Chipman, E. I. George, and R. E. McCulloh. 1998. Making sense of a forest of trees. In Proceedings of the 30th Symposium on the Interface, S. Weisberg (Ed.). Fairfax Station, VA: Interface Foundation of North America, 84--92.
- Paulo Cortez and Mark J. Embrechts. 2011. Opening black box data mining models using sensitivity analysis. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’11). IEEE, 341--348.
- 10.5555/3091574.3091580 · doi:10.5555/3091574.3091580
- 10.5555/2998828.2998832 · doi:10.5555/2998828.2998832
- 10.5555/3000292.3000299 · doi:10.5555/3000292.3000299
- Bryce Goodman and Seth Flaxman. 2016. EU regulations on algorithmic decision-making and a “right to explanation.” In Proceedings of the ICML Workshop on Human Interpretability in Machine Learning (WHI’16). Retrieved from http://arxiv. org/abs/1606.08813 v1.
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018). · arXiv
- U. Johansson, R. König, and L. Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In Proceedings of the 13th International Conference on Artificial Neural Networks. 13--16.
- 10.5555/3157096.3157352 · doi:10.5555/3157096.3157352
- 10.5555/2969033.2969045 · doi:10.5555/2969033.2969045
- 10.5555/2969442.2969492 · doi:10.5555/2969442.2969492
- Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable algorithms. U. Penn. Law Rev. 165 (2017), 633--705.
- Will Landecker, Michael D. Thomure, Luís M. A. Bettencourt, Melanie Mitchell, Garrett T. Kenyon, and Steven P. Brumby. 2013. Interpreting individual classifications of hierarchical networks. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’13). IEEE, 32--38.
- Dmitry M. Malioutov, Kush R. Varshney, Amin Emad, and Sanjeeb Dash. 2017. Learning interpretable classification rules with boolean compressed sensing. In Transparent Data Mining for Big and Small Data. Springer, 95--121.
- 10.5555/3091825.3091863 · doi:10.5555/3091825.3091863
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- 10.5555/2717112 · doi:10.5555/2717112
- Michael J. Pazzani, S. Mani, William R. Shankle et al. 2001. Acceptance of rules generated by machine learning among medical experts. Methods Info. Med. 40, 5 (2001), 380--385.
- 10.5555/1597122.1597143 · doi:10.5555/1597122.1597143
- 10.5555/1625015.1625078 · doi:10.5555/1625015.1625078
- 10.5555/152181 · doi:10.5555/152181
- Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. 2016. Programs as black-box explanations. arXiv preprint arXiv:1611.07579 (2016). · arXiv
- Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 353--382.
- Marina M.-C. Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567 (2016). · arXiv
- Tong Wang. 2017. Multi-value rule sets. arXiv preprint arXiv:1710.05257 (2017). · arXiv
- 10.5555/273530.273548 · doi:10.5555/273530.273548
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 101–200 of 400
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283
“There is no agreement on what is meant by ‘explanation ’ [262].”
BackgroundCarbon dioxide separation and capture by adsorption: a review
Karimi, M., Shirzad, M., Silva, J. A. C., & Rodrigues, A. E. (2023) · Environmental Chemistry Letters · doi:10.1007/s10311-023-01589-z · Cited by 275
On the interpretability of machine learning-based model for predicting hypertension
Elshawi, R., Al-Mallah, M. H., & Sakr, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0874-0 · Cited by 271
“Decision rules have used intensively to mimic the behavior of a black-box model globally or locally given that the training data is available when providing local explanations [32].”
BackgroundInterpretable machine learning with an ensemble of gradient boosting machines
Konstantinov, A. V., & Utkin, L. V. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106993 · Cited by 271
Neural Prototype Trees for Interpretable Fine-grained Image Recognition
Nauta, M., van Bree, R., & Seifert, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01469 · Cited by 269
“fraction of test images for which the soft and hard strategy make the same classification [19]) is practically 1.”
BackgroundCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267
“Later, it was found that the photos of friendly tanks were taken on sunny days, while for enemy tanks, photos clicked only on overcast days were available [58].”
BackgroundLow-Cost Outdoor Air Quality Monitoring and Sensor Calibration: A Survey and Critical Analysis
Concas, F., Mineraud, J., Lagerspetz, E., Varjonen, S., Liu, X., Puolamäki, K., Nurmi, P., & Tarkoma, S. (2021) · ACM Transactions on Sensor Networks · doi:10.1145/3446005 · Cited by 265
“tes, which may lead to unexpected and counter-intuitive calibration behavior in real-life applications. For a thorough discussion on black-box models and issues with them, we refer to Guidotti et al. [115]. The alternative to black boxes is to use white boxes, models of which the performance can be easily understood a…”
BackgroundBenchmarking and survey of explanation methods for black box models
Bodria, F., Giannotti, F., Guidotti, R., Naretto, F., Pedreschi, D., & Rinzivillo, S. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00933-9 · Cited by 264
“In this section, we synthetically recall the existing taxonomy and classification of XAI methods present in the literature [59,3,50,13,105,26] to allow the reader to complete the proposed explanation-based categorization of explanation methods.”
BackgroundMethodInfluentialSelf-citationFour principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
“There are surveys [29, 37, 121, 153, 163, 227] summarizing explanation approaches in machine learning.”
BackgroundA Deep Gravity model for mobility flows generation
Simini, F., Barlacchi, G., Luca, M., & Pappalardo, L. (2021) · Nature Communications · doi:10.1038/s41467-021-26752-4 · Cited by 261
“Understanding why a model makes a certain prediction is crucial to interpret results, explain differences between models, and assess to what extent we understand the phenomenon under analysis 67 – 69 .”
BackgroundOne Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268
“Again, we do not claim that this is a complete listing, although most of the methods reviewed by Guidotti et al. [25] are included.”
BackgroundResultSoldani, J., & Brogi, A. (2023) · ACM Computing Surveys · doi:10.1145/3501297 · Cited by 256
“This is tightly coupled with the problem of explainable AI [26], given that anomaly detection is mainly enacted with machine learning-based techniques.”
BackgroundCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K., Dickerson, J., & Shah, C. (2024) · ACM Computing Surveys · doi:10.1145/3677119 · Cited by 255
“The explainability problem can be divided into model explanation and outcome explanation problems [127].”
Enchanted Determinism: Power without Responsibility in Artificial Intelligence
Campolo, A., & Crawford, K. (2020) · Engaging Science, Technology, and Society · doi:10.17351/ests2020.277 · Cited by 254
explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
Spinner, T., Schlegel, U., Schafer, H., & El-Assady, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934629 · Cited by 254
“[25] present an extensive overview of XAI, especially current methods for the explanation of models, which our framework incorporates to tackle the latest challenges black-box models impose.”
BackgroundMethodInterpreting tree ensembles with inTrees
Deng, H. (2019) · International Journal of Data Science and Analytics · doi:10.1007/s41060-018-0144-8 · Cited by 253
“[19] provided a thorough review on that.”
Explaining the black-box model: A survey of local interpretation methods for deep neural networks
Liang, Y., Li, S., Yan, C., Li, M., & Jiang, C. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.08.011 · Cited by 253
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251
Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247
Grimmelikhuijsen, S. (2023) · Public Administration Review · doi:10.1111/puar.13483 · Cited by 259
Abbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
“See [1, 2, 7, 27, 29, 41, 46, 48, 49, 84, 107, 109, 110, 121] for in-depth surveys, and the following for examples of XAI research done in diferent”
BackgroundExplainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227
Explainability of deep vision-based autonomous driving systems: Review and challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2021) · arXiv · doi:10.48550/arxiv.2101.05307 · Cited by 227
A Survey of Data-driven and Knowledge-aware eXplainable AI
Li, X.-H., Cao, C. C., Shi, Y., Bai, W., Gao, H., Qiu, L., Wang, C., Gao, Y., Zhang, S., Xue, X., et al. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2983930 · Cited by 224
Passi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225
“The fact that models do not always furnish explanations has sparked efforts to produce humanly understandable explanations of data science workings and results [38,82,92].”
BackgroundCausal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219
“Model interpretability techniques are often categorized into two classes: global and local [15, 21, 55].”
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219
“A full review of XAI techniques is beyond the scope of this paper and can be found in many recent survey papers [1, 29, 55, 56].”
BackgroundMienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212
Kenny, E. M., Ford, C., Quinn, M., & Keane, M. T. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103459 · Cited by 211
Human-centered XAI: Developing design patterns for explanations of clinical decision support systems
Schoonderwoerd, T. A. J., Jorritsma, W., Neerincx, M. A., & van den Bosch, K. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102684 · Cited by 210
Nasarian, E., Alizadehsani, R., Acharya, U. R., & Tsui, K.-L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102412 · Cited by 207
Schramowski, P., Stammer, W., Teso, S., Brugger, A., Herbert, F., Shao, X., Luigs, H.-G., Mahlein, A.-K., & Kersting, K. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0212-3 · Cited by 207
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“Deep neural nets lack interpretability as it is difficult to analyze which modalities or features are driving the predictions [28].”
BackgroundGuidelines and evaluation of clinical explainable AI in medical image analysis
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102684 · Cited by 199
“The notion of XAI and its corresponding techniques were originally proposed in the machine learning community (Barredo Arrieta et al., 2020; Guidotti et al., 2018; Zhang and Zhu, 2018), and were then applied and developed in the medical image analysis (MIA) community (Yang et al.”
MethodLeichtmann, B., Humer, C., Hinterreiter, A., Streit, M., & Mara, M. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107539 · Cited by 200
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations
Laugel, T., Lesot, M.-J., Marsala, C., Renard, X., & Detyniecki, M. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/388 · Cited by 194
Explainable decision forest: Transforming a decision forest into an interpretable tree
Sagi, O., & Rokach, L. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.03.013 · Cited by 189
Local Rule-Based Explanations of Black Box Decision Systems
Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., & Giannotti, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10820 · Cited by 179
“Recently, the research of methods for explaining black box decision systems has caught much attention [17].”
BackgroundInfluentialSelf-citationExplainable and interpretable artificial intelligence in medicine: a systematic bibliometric review
Frasca, M., La Torre, D., Pravettoni, G., & Cutica, I. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00114-7 · Cited by 190
“Guidotti et al. [21] provides a comprehensive overview of the methods proposed in the literature to explain decision-making systems based on opaque and dark machine learning models.”
An artificial intelligence life cycle: from conception to production
De Silva, D., & Alahakoon, D. (2022) · Patterns · doi:10.1016/j.patter.2022.100489 · Cited by 189
Perez-Cerrolaza, J., Abella, J., Borg, M., Donzella, C., Cerquides, J., Cazorla, F. J., Englund, C., Tauber, M., Nikolakopoulos, G., & Flores, J. L. (2024) · ACM Computing Surveys · doi:10.1145/3626314 · Cited by 185
“• Explainability/Interpretability : łExtent to which a ML system can provide an explanation about a decision in a form understandable by a human” (e.g., see surveys [Adadi and Berrada 2018; Barredo Arrieta et al. 2020; Guidotti et al. 2018]).”
BackgroundInfluentialHuman-Centered Artificial Intelligence for Designing Accessible Cultural Heritage
Pisoni, G., Díaz-Rodríguez, N., Gijlers, H., & Tonolli, L. (2021) · Applied Sciences · doi:10.3390/app11020870 · Cited by 183
Grimmelikhuijsen, S., & Meijer, A. (2022) · Perspectives on Public Management and Governance · doi:10.1093/ppmgov/gvac008 · Cited by 189
A literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180
“Continuing research in XAI is focused on explaining AI capabilities, encouraging ethical and responsible use of technology, and empowering a knowledgeable society to interact with advancing AI systems [9], [10], [11].”
From explainable to interactive AI: A literature review on current trends in human-AI interaction
Raees, M., Meijerink, I., Lykourentzou, I., Khan, V.-J., & Papangelis, K. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103301 · Cited by 180
Jacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445385 · Cited by 177
“(3) Stability and dropout feature importance (figure 1B): Awidely used approach for explaining ML predictions that shows relevant features and their contributions to the prediction [21].”
BackgroundMachine Learning for Perovskite Solar Cells and Component Materials: Key Technologies and Prospects
Liu, Y., Tan, X., Liang, J., Han, H., Xiang, P., & Yan, W. (2023) · Advanced Functional Materials · doi:10.1002/adfm.202214271 · Cited by 174
Degas, A., Islam, M. R., Hurter, C., Barua, S., Rahman, H., Poudel, M., Ruscio, D., Ahmed, M. U., Begum, S., Rahman, M. A., et al. (2022) · Applied Sciences · doi:10.3390/app12031295 · Cited by 168
“Above all, the term “Explainability” affiliates the interface between humans and decision-makers, which is concurrently comprehensible to humans and an accurate representation of the decision-maker [263].”
BackgroundQuantifying Climate and Catchment Control on Hydrological Drought in the Continental United States
Konapala, G., & Mishra, A. (2020) · Water Resources Research · doi:10.1029/2018wr024620 · Cited by 168
Explainable Artificial Intelligence: Concepts, Applications, Research Challenges and Visions
Longo, L., Goebel, R., Lecue, F., Kieseberg, P., & Holzinger, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_1 · Cited by 162
“…for any attempt to articulate the current concepts for XAI is that there is a very high volume of current activity, both on the research side [68, 74, 22], and in aggressive industrial developments, where any XAI functions can provide a market advantage to all for profit applications of AI [23, 62].”
Legal requirements on explainability in machine learning
Bibal, A., Lognoul, M., de Streel, A., & Frénay, B. (2021) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09270-4 · Cited by 165
“Nowadays, the two terms are beginning to have different meanings (Guidotti et al. 2018), with interpretability describing the fact that the model is understandable by its nature (e.g. decision trees) and explainability corresponding to the capacity of a black-box model to be explained using…”
BackgroundDoctor XAI: an ontology-based approach to black-box sequential data classification explanations
Panigutti, C., Perotti, A., & Pedreschi, D. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372855 · Cited by 164
“In response to the increasing demand for interpretability, a vast literature on this matter has been produced [18].”
BackgroundSelf-citationFuture applications of generative large language models: a data-driven case study on ChatGPT
Chiarello, F., Giordano, V., Spada, I., Barandoni, S., & Fantoni, G. (2024) · Technovation · doi:10.1016/j.technovation.2024.103002 · Cited by 164
Evaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163
RTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter
Mazza, M., Cresci, S., Avvenuti, M., Quattrociocchi, W., & Tesconi, M. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326015 · Cited by 163
“Model and decision explainability has now become one of the major practical and ethical concerns around AI [23].”
BackgroundA survey on aspect-based sentiment classification
Brauwers, G., & Frasincar, F. (2023) · ACM Computing Surveys · doi:10.1145/3503044 · Cited by 165
Discrimination, artificial intelligence, and algorithmic decision-making
Borgesius, F. Z. (2025) · arXiv · doi:10.48550/arxiv.2510.13465 · Cited by 152
Eling, M., Nuessle, D., & Staubli, J. (2022) · The Geneva Papers on Risk and Insurance Issues and Practice · doi:10.1057/s41288-020-00201-7 · Cited by 156
Pichler, M., Boreux, V., Klein, A., Schleuning, M., & Hartig, F. (2020) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13329 · Cited by 149
Ethical machines: The human-centric use of artificial intelligence
Lepri, B., Oliver, N., & Pentland, A. (2021) · iScience · doi:10.1016/j.isci.2021.102249 · Cited by 149
“Detailed reviews and perspectives on these topics can also be found in several recent publications (Pasquale, 2015; Mittelstadt et al., 2016; Veale and Binns, 2017; Barocas et al., 2018; Cath et al., 2018; Guidotti et al., 2018; Lipton, 2018; Jobin et al., 2019; Brundage et al., 2020; Kearns and Roth, 2020).”
BackgroundInfluentialThe Rhetoric and Reality of Anthropomorphism in Artificial Intelligence
Watson, D. (2019) · Minds and Machines · doi:10.1007/s11023-019-09506-6 · Cited by 148
An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148
“According to the literature survey of model-agnostic techniques [30], we selected and discussed two state-of-the-art model-agnostic techniques (i.e., LIME and BreakDown).”
BackgroundMethodEthical Framework for Harnessing the Power of AI in Healthcare and Beyond
Nasir, S., Khan, R. A., & Bai, S. (2024) · IEEE Access · doi:10.1109/access.2024.3369912 · Cited by 152
“In other words, interpretability is a desirable component for explainability, but not all inter-pretable models are explainable [17].”
BackgroundIt's complicated: the relationship between user trust, model accuracy and explanations in AI
Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3495013 · Cited by 144
“, by visualising a decision tree) [18].”
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
“A diverse set of XAI methods have been proposed that surface technical details of ML models, as surveyed by several authors [1, 2, 18, 35, 38].”
BackgroundExplaining the Explainer: A First Theoretical Analysis of LIME
Garreau, D., & von Luxburg, U. (2020) · arXiv · doi:10.48550/arxiv.2001.03447 · Cited by 143
“We refer to Guidotti et al. (2019) for a thorough review.”
Bogina, V., Hartman, A., Kuflik, T., & Shulner-Tal, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00248-0 · Cited by 142
A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images
Messina, P., Pino, P., Parra, D., Soto, A., Besa, C., Uribe, S., Andía, M., Tejos, C., Prieto, C., & Capurro, D. (2022) · ACM Computing Surveys · doi:10.1145/3522747 · Cited by 142
GLocalX—From Local to Global Explanations of Black Box AI Models
Setzu, M., Guidotti, R., Monreale, A., Turini, F., Pedreschi, D., & Giannotti, F. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103457 · Cited by 140
“Examples include deep neural networks, SVMs, and ensemble classifiers like Random Forest and AdaBoost classifiers [33, 2, 4].”
BackgroundMethodInfluentialSelf-citationKliegr, T., Bahník, Š., & Fürnkranz, J. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103458 · Cited by 139
“, to [68]).”
Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150
“Surveys of the research landscape show a vast number of mechanisms for developing ‘interpretable’ or ‘explainable’ ML systems have been proposed, seeking to provide explanations and other interpretable accounts of model behaviour to a range of forums, many of which have come in the last few years [4, 32].”
BackgroundI Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144
“XAI research distinguishes two types of explanations - local and global [2, 21].”
BackgroundFan, H., Zhang, X., Xu, Y., Fang, J., Zhang, S., Zhao, X., & Yu, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102161 · Cited by 137
Su, S., Li, W., Mou, J., Garg, A., Gao, L., & Liu, J. (2023) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2022.3204843 · Cited by 136
“The black-box model is the data-driving one that reflects the mapping relation between input and output variables [25].”
BackgroundThe explainability paradox: Challenges for xAI in digital pathology
Evans, T., Retzlaff, C. O., Geißler, C., Kargl, M., Plass, M., Müller, H., Kiehl, T.-R., Zerbe, N., & Holzinger, A. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.03.009 · Cited by 135
Keane, M. T., & Smyth, B. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58342-2_11 · Cited by 128
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135
“According to Guidotti et al. [20], black-box explanation problems can be categorized into the following: • Model explanation problems , which requires explaining the underlying logic behind a black-box model.”
BackgroundOswald, M. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0359 · Cited by 132
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
A Survey on Explainable Anomaly Detection
Li, Z., Zhu, Y., & Van Leeuwen, M. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3609333 · Cited by 139
“For instance, references [20, 76] analyze the XAI literature and propose important properties that should be considered when designing an XAI technique.”
BackgroundA survey on XAI and natural language explanations
Cambria, E., Malandri, L., Mercorio, F., Mezzanzanica, M., & Nobani, N. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2022.103111 · Cited by 131
“In their survey, Guidotti et al. (2018b) divide XAI problems into two broad categories of black-box explanation problems and transparent box design, with the first one further has been categorized into the following three sub-categories: Model Explanation : Explanations are made through the…”
BackgroundCo-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127
“This topic has recently witnessed an increased interest that generated vast literature on AI trans-parency and explainability [19, 57 ].”
BackgroundSelf-citationA scoping review of artificial intelligence-based methods for diabetes risk prediction
Mohsen, F., Al-Absi, H. R. H., Yousri, N. A., El Hajj, N., & Shah, Z. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00933-5 · Cited by 125
“Hence, there has been a shift from focusing solely on prediction performance to placing greater emphasis on understanding algorithm dynamics, a notable trend in recent research 73,74 .”
BackgroundEXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124
“Other interpretable strategies include logistic regression and rule-based learning [26], which have significant limitations.”
BackgroundMethodInfluentialVale, D., El-Sharif, A., & Ali, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00142-y · Cited by 124
“They include methods supporting models based on different data types including tabular data [32], computer vision [45], and natural language text [52].”
BackgroundMethodInfluentialPredicting breast cancer 5-year survival using machine learning: A systematic review
Li, J., Zhou, Z., Dong, J., Fu, Y., Li, Y., Luan, Z., & Peng, X. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250370 · Cited by 139
Bayesian networks for interpretable machine learning and optimization
Mihaljević, B., Bielza, C., & Larrañaga, P. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.138 · Cited by 123
“For example, neural networks are often converted into decision trees or logical rules while deep neural networks for text and images are explained with saliency masks that highlight the determining aspects of a text or image [13].”
BackgroundDeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
Han, D., Wang, Z., Chen, W., Zhong, Y., Wang, S., Zhang, H., Yang, J., Shi, X., & Yin, X. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484589 · Cited by 121
“Although DL interpretability has been extensively studied [19, 37], we argue that existing interpretations are unadaptable for unsupervised DL-based security systems for the following two reasons.”
BackgroundHuman-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“[44] categorize post-hoc XAI techniques into global explanation on the overall logic of the model, local explanation on a particular prediction, and counterfactual inspection that supports understanding how the model would behave with an alternative input.”
BackgroundMethodInterpretable Anomaly Detection with DIFFI: Depth-based feature importance of Isolation Forest
Carletti, M., Terzi, M., & Susto, G. A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105730 · Cited by 118
Human-XAI Interaction: A Review and Design Principles for Explanation User Interfaces
Chromik, M., & Butz, A. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85616-8_36 · Cited by 207
“They formalize and ground the concept of XAI [1, 3], relate it to adjacent concepts and disciplines [1, 62], categorize methods [36, 57], analyze the user perspective [33], review evaluation practices [65], or outline future research directions [1, 3].”
BackgroundRethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112
“In contrast, it is well-known that modern deep neural networks suffer from interpretability issues, and addressing them is an active line of research (e.g. [Sundararajan et al., 2017]; see [Guidotti et al., 2018] for a survey).”
BackgroundHu, C.-A., Chen, C.-M., Fang, Y.-C., Liang, S.-J., Wang, H.-C., Fang, W.-F., Sheu, C.-C., Perng, W.-C., Yang, K.-Y., Kao, K.-C., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-033898 · Cited by 106
Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
“, example or rule based explanations [35]), given the former approach’s popularity and intuitiveness– it reflects how the model weighs different features and gives people direct access to the inner working of the model.”
BackgroundInfluentialBell, A., Solano-Kamaiko, I., Nov, O., & Stoyanovich, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533090 · Cited by 103
“[31] point out that the distinction between interpretable and black-box models lacks an attention to model complexity.”
BackgroundPedreschi, D., Pappalardo, L., Ferragina, E., Baeza-Yates, R., Barabási, A.-L., Dignum, F., Dignum, V., Eliassi-Rad, T., Giannotti, F., Kertész, J., et al. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2024.104244 · Cited by 102
“AI has achieved human-like performance in many challenging tasks [33–38] and is becoming increasingly explainable and human-centric [39–45].”
Self-citationTowards Trustable Explainable AI
Ignatiev, A. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/726 · Cited by 184
Metadata
- Authors
- Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D.
- Affiliations
- University of Pisa (IT)
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- Venue
- ACM Computing Surveys
- Published
- 2018-08-22
- Volume
- 51
- Issue
- 5
- Pages
- 1-42
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2786715987
- Cited by
- 629
- References
- 121
- Citations by year
- 2026: 3 · 2025: 62 · 2024: 82 · 2023: 86 · 2022: 92 · 2021: 145 · 2020: 105 · 2019: 40 · 2018: 13
- FWCI
- 42.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- Scuola Normale Superiore di Pisa (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Horizon 2020 Framework Programme (-2014-2015, 654024)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- explainable AI, black-box models, interpretability, decision support system, explanation methods, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.