A Survey of Methods for Explaining Black Box Models
Open accessThe DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index; 4 of the 59 sources its claims cite were checked for support.
Abstract
In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Overview
CoChat Summary
TL;DR
This survey classifies methods for explaining black-box machine learning models into four problem types: black box model explanation, black box outcome explanation, black box inspection, and transparent box design. It organizes the literature by problem, explanator, black-box type, and data type, and highlights open challenges around defining and measuring explanation and comprehensibility.
Key findings
- The paper identifies and formally defines four explanation problems: black box model explanation (global interpretable mimic), black box outcome explanation (local explanation for a prediction), black box inspection (visual/textual representation of behavior), and transparent box design (interpretable model by design).
- It classifies explanators into decision trees, decision rules, feature importance, saliency masks, sensitivity analysis, partial dependence plots, prototype selection, and neuron activation; black boxes into neural networks, tree ensembles, SVMs, deep neural networks, and agnostic black boxes; and data into tabular, image, and text.
- Many methods are specific to a problem or black-box type, while agnostic/generalizable approaches use reverse engineering by querying the black box as an oracle; examples include Trepan, inTrees, LIME, and PALM.
- Motivating cases cited include COMPAS ethnic bias (black non-reoffenders classified high risk twice as much as whites who did not re-offend; white repeat offenders classified low risk twice as much as black repeat offenders), credit scoring discordance (in 500,000 records, 29% of consumers received scores differing by at least fifty points among three major US banks), and the St. George’s Hospital screening program that discriminated via inferred ethnicity from surnames and place of birth.
Methods
- Literature survey/review; no primary empirical study. The paper formalizes the general classification problem and defines the four explanation problems as specific cases.
- The classification dimensions are: type of problem faced, type of explanator adopted, type of black box model opened, and type of data used as input by the black box. Table 1 summarizes methods with features including General, Random, Examples, Code, and Dataset.
- The common reverse-engineering approach queries the black box with a test dataset to create an oracle dataset, then trains a comprehensible predictor; approaches are distinguished as generalizable/pedagogical versus not generalizable/decompositional.
Limitations
- The paper states that there is no agreement on what an explanation is, no general/common formalism for defining explanations or properties such as soundness, completeness, compactness, and comprehensibility, and no serious work quantifying human comprehensibility; it also lists open questions on latent features, recommender systems, and learning predictors directly from explanations.
- The paper states it does not address whether the black box can be queried at will versus only a fixed dataset being available, or whether the complete feature set is known, because there is insufficient literature.
- Evident from the text: the survey scope is limited to tabular, image, and text data, with no work addressing interpretability for sequence, spatio-temporal, or complex network data; many reviewed methods are ad hoc or specific, only a few consider user expertise, and no work provides real experiments on the time required to understand an explanation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 13 Not valid · 38 Not checked · 49 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and no one provides real experiments about the time required to understand an explanation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The works reviewed in the literature only implicitly specify if their proposal is global or local.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Just a few of them take into account the nature of user expertise [29,84,87]”
What the sources say
- Not checked yetcites [29] 29. R. D. Gibbons, G. Hooker, M. D. Finkelman, D. J. Weiss, P. A. Pilkonis, E. Frank, T. Moore, and D. J. Kupfer. The cad-mdd: a computerized adaptive diagnostic screening tool for depression. The Journal of clinical psychiatry , 74(7):669, 2013. source
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
- Not checked yetcites [87] 87. V. Schetinin, J. E. Fieldsend, D. Partridge, T. J. Coats, W. J. Krzanowski, R. M. Everson, T. C. Bailey, and A. Hernandez. Confident interpretation of bayesian decision tree ensembles for clinical applications. IEEE Transactions on Information Technology in Biomedicine , 11(3):312-319, 2007. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In the following, we list all the black boxes opened in the reviewed papers. These black boxes are all supervised learning algorithm designed to solve a classification problem [95].”
What the sources say
- Not checked yetcites [95] 95. P.-N. Tan et al. Introduction to data mining . Pearson Education India, 2006.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, this is not always the case in practice [84].”
What the sources say
- Not checked yetcites [84] 84. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 104 claims and to verify the 46 references still open.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 65
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While “black‐box” algorithms raise complex opaqueness and interpretability challenges, simpler interpretable ML algorithms do not, but have generally also been regarded as less powerful (Guidotti et al. 2019; see however, Rudin 2019).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Guidotti et al. 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since we could not find a complete and recent overview of explanation types, we reviewed recent XAI surveys [14,58,167,280] and adapted and extended the categories identified by Guidotti et al. [86] with these insights.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, there is no consensus on what a good XAD technique should be.”
From A Survey on Explainable Anomaly Detection (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Among the main impediments prohibiting banking institutions from implementing their AI strategy is the ''black-box'' paradox [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The models are black boxes due to their significant non-linearity and intricacy, making it unfeasible for individuals to fathom their fundamental operational processes and judgment mechanisms [28].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [28]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Problem Formulation · p. 13
“for the comprehensible predictor c we get the set of predictions ¯ Y = ⋃ x ∈ X c ( x )”
§ 4.1 Problem Formulation · p. 13
“where as target value is used the prediction ¯ Y of the black box b instead of the real values Y”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“elated to producing a certain prediction by realizing a sort of articial image prototype. There is a signicant amount of work in understanding the representation of DNN by means of artifact images, [41,100,105,111]. We conclude this section presenting how [25] deals with artifacts in DNNs. Finding a single representat…”
BackgroundLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 5×
“al. 2017 Model Expl. DT AGN ANY X X X - [97] Tolomei et al. 2017 Model Expl. FI TE TAB X - [108] Xu et al. 2015 Outcome Expl. SM DNN IMG X X X - [25] Fong et al. 2017 Outcome Expl. SM DNN IMG X X CAM [113] Zhou et al. 2016 Outcome Expl. SM DNN IMG X X X Grad-CAM [89] Selvaraju et al. 2016 Outcome Expl. SM DNN IMG X X…”
BackgroundMethodInfluentialShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Reference [134] introduces an attention-based model f , which automatically identifies the contents of an image.”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“In Reference [116] it is shown the alteration of an image (e.”
Background(2008) · Data Mining and Market Intelligence for Optimal Marketing Returns · doi:10.4324/9780080878096-12 · Cited by 6,954
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundThe Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 1×
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375 · cited 1×
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
Background
Research works printed but not linked: 31
- 10.5555/645920.672836 · doi:10.5555/645920.672836
- Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, and Karol Zieba. 2016. VisualBackProp: Visualizing CNNs for autonomous driving. CoRR, Vol. abs/1611.05418 (2016). · arXiv
- Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press.
- Carolyn Carter, Elizabeth Renuart, Margot Saunders, and Chi Chi Wu. 2006. The credit card market and regulation: In need of repair. NC Bank. Inst. 10 (2006), 23.
- H. A. Chipman, E. I. George, and R. E. McCulloh. 1998. Making sense of a forest of trees. In Proceedings of the 30th Symposium on the Interface, S. Weisberg (Ed.). Fairfax Station, VA: Interface Foundation of North America, 84--92.
- Paulo Cortez and Mark J. Embrechts. 2011. Opening black box data mining models using sensitivity analysis. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’11). IEEE, 341--348.
- 10.5555/3091574.3091580 · doi:10.5555/3091574.3091580
- 10.5555/2998828.2998832 · doi:10.5555/2998828.2998832
- 10.5555/3000292.3000299 · doi:10.5555/3000292.3000299
- Bryce Goodman and Seth Flaxman. 2016. EU regulations on algorithmic decision-making and a “right to explanation.” In Proceedings of the ICML Workshop on Human Interpretability in Machine Learning (WHI’16). Retrieved from http://arxiv. org/abs/1606.08813 v1.
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018). · arXiv
- U. Johansson, R. König, and L. Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In Proceedings of the 13th International Conference on Artificial Neural Networks. 13--16.
- 10.5555/3157096.3157352 · doi:10.5555/3157096.3157352
- 10.5555/2969033.2969045 · doi:10.5555/2969033.2969045
- 10.5555/2969442.2969492 · doi:10.5555/2969442.2969492
- Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable algorithms. U. Penn. Law Rev. 165 (2017), 633--705.
- Will Landecker, Michael D. Thomure, Luís M. A. Bettencourt, Melanie Mitchell, Garrett T. Kenyon, and Steven P. Brumby. 2013. Interpreting individual classifications of hierarchical networks. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM’13). IEEE, 32--38.
- Dmitry M. Malioutov, Kush R. Varshney, Amin Emad, and Sanjeeb Dash. 2017. Learning interpretable classification rules with boolean compressed sensing. In Transparent Data Mining for Big and Small Data. Springer, 95--121.
- 10.5555/3091825.3091863 · doi:10.5555/3091825.3091863
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- 10.5555/2717112 · doi:10.5555/2717112
- Michael J. Pazzani, S. Mani, William R. Shankle et al. 2001. Acceptance of rules generated by machine learning among medical experts. Methods Info. Med. 40, 5 (2001), 380--385.
- 10.5555/1597122.1597143 · doi:10.5555/1597122.1597143
- 10.5555/1625015.1625078 · doi:10.5555/1625015.1625078
- 10.5555/152181 · doi:10.5555/152181
- Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. 2016. Programs as black-box explanations. arXiv preprint arXiv:1611.07579 (2016). · arXiv
- Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 353--382.
- Marina M.-C. Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567 (2016). · arXiv
- Tong Wang. 2017. Multi-value rule sets. arXiv preprint arXiv:1710.05257 (2017). · arXiv
- 10.5555/273530.273548 · doi:10.5555/273530.273548
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 201–300 of 400
Condran, S., Bewong, M., Islam, M. Z., Maphosa, L., & Zheng, L. (2022) · IEEE Access · doi:10.1109/access.2022.3188649 · Cited by 99
Survey on Explainable AI: Techniques, challenges and open issues
Abusitta, A., Li, M. Q., & Fung, B. C. M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124710 · Cited by 98
Krafft, T. D., Zweig, K. A., & König, P. D. (2022) · Regulation & Governance · doi:10.1111/rego.12369 · Cited by 95
AI, Opacity, and Personal Autonomy
Vaassen, B. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00577-5 · Cited by 94
Fintech for the Poor: Financial Intermediation Without Discrimination
Tantri, P. (2021) · Review of Finance · doi:10.1093/rof/rfaa039 · Cited by 94
Kalakoti, R., Bahsi, H., & Nõmm, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3360626 · Cited by 93
“The idea of explaining ML models [21], [22] has become increasingly important and has been explored by researchers in various domains [23].”
BackgroundJha, A., K. Aicher, J., R. Gazzara, M., Singh, D., & Barash, Y. (2020) · Genome Biology · doi:10.1186/s13059-020-02055-7 · Cited by 93
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269
“Only a few scholars attempted to make a more comprehensive survey and organization of the methods for explainability as a whole [1, 21].”
BackgroundMethodInfluentialCabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89
Straw, I. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101965 · Cited by 86
FairLens: Auditing black-box clinical decision support systems
Panigutti, C., Perotti, A., Panisson, A., Bajardi, P., & Pedreschi, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102657 · Cited by 86
Self-citationAbedin, B. (2021) · Internet Research · doi:10.1108/intr-05-2020-0300 · Cited by 132
“Guidotti et al. (2018) review black box models and create a taxonomy of interpretability practices and explainability methods in the literature.”
BackgroundMethodVisualization of Convolutional Neural Networks for Monocular Depth Estimation
Hu, J., Zhang, Y., & Okatani, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00397 · Cited by 111
“However, there are only a few methods that have been recognized to be practically useful in the community [11, 20, 21].”
BackgroundKeane, M. T., & Kenny, E. M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-29249-2_11 · Cited by 84
““deep neural networks” (DNNs; following [5]).”
BackgroundInfluentialInterpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 80
“Many groups have developed techniques intended to improve machine learning systems’ interpretability (Zhang & Zhu, 2018; Guidotti et al., 2018; Chakraborty et al., 2017; Rudin, 2014), though the definitions of and motivations for interpretability (if specified) vary wildly between methods (Lipton,…”
MethodFarrow, R. (2023) · Learning, Media and Technology · doi:10.1080/17439884.2023.2185630 · Cited by 81
“Guidotti et al. (2018) propose a universal typology for understanding issues around ‘ black box ’ computation: the model explanation problem; the outcome explanation problem; the inspection problem; and the transparent box design problem.”
Explainable goal-driven agents and robots-a comprehensive review
Sado, F., Loo, C. K., Liew, W. S., Kerzel, M., & Wermter, S. (2023) · ACM Computing Surveys · doi:10.1145/3564240 · Cited by 79
“Thus, data-driven XAI implies explaining the decision made by a machine learning “blackbox” system, given input data [81].”
MethodMehdiyev, N., & Fettke, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-64949-4_1 · Cited by 78
SmileyCluster: supporting accessible machine learning in K-12 scientific discovery
Wan, X., Zhou, X., Ye, Z., Mortensen, C. K., & Bai, Z. (2020) · Proceedings of the Interaction Design and Children Conference · doi:10.1145/3392063.3394440 · Cited by 77
“The latter is critical to prepare young learners to be more sensible with respect to decision-making and ethical aspects of AI such as trust and fairness [24].”
BackgroundEnnab, M., & Mcheick, H. (2022) · Diagnostics · doi:10.3390/diagnostics12071557 · Cited by 76
“In essence, interpretability is the interface between humans and the decision model, which is both an accurate proxy for the decision model and understandable by humans [18].”
BackgroundOrtega-Bolaños, R., Bernal-Salcedo, J., Germán Ortiz, M., Galeano Sarmiento, J., Ruz, G. A., & Tabares-Soto, R. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10740-3 · Cited by 76
“Fairness, Explainability, and Interpretability remain principles with many technical resources (Guidotti et al. 2018; Namatēvs et al. 2022; Ras et al. 2022).”
AI Systems Under Criminal Law: a Legal Analysis and a Regulatory Perspective
Lagioia, F., & Sartor, G. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00362-x · Cited by 75
“…of different levels of complexity, can be used to this purpose, from enabling the system to provide logical implications of its belief and goal sets, to the much more difficult task of providing explanations for the formation of beliefs and choices (Doshi-Velez et al 2017, Guidotti et al 2018, 93).”
BackgroundExplainable machine learning for public policy: Use cases, gaps, and research directions
Amarasinghe, K., Rodolfa, K. T., Lamba, H., & Ghani, R. (2023) · Data & Policy · doi:10.1017/dap.2023.2 · Cited by 73
“Hase and Bansal [2020] describe an experiment where users were asked to perform the task of ‘forward simulation’ and subjectively evaluate the quality of explanations (measuring a concept of “simulatability”).”
BackgroundExplainability pitfalls: beyond dark patterns in explainable AI
Ehsan, U., & Riedl, M. O. (2024) · Patterns · doi:10.1016/j.patter.2024.100971 · Cited by 72
Beyond explainability: justifiability and contestability of algorithmic decision systems
Henin, C., & Le Métayer, D. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01251-8 · Cited by 71
“On the technical side, a lot of effort has been put on bias reduction, accuracy improvement and the generation of different types of explanations with or without access to the code of the system (“white box” versus “black box” methods) (Adadi and Berrada 2018; Guidotti et al. 2018; Arrieta et al. 2019; Henin and Le Mé…”
MethodMetrics, Explainability and the European AI Act Proposal
Sovrano, F., Sapienza, S., Palmirani, M., & Vitali, F. (2022) · J · doi:10.3390/j5010010 · Cited by 71
“Similar to [6], ref.”
MethodInfluentialBuilding Ontology-Driven Tutoring Models for Intelligent Tutoring Systems Using Data Mining
Chang, M., D'Aniello, G., Gaeta, M., Orciuoli, F., Sampson, D., & Simonelli, C. (2020) · IEEE Access · doi:10.1109/access.2020.2979281 · Cited by 71
“The drawback of the second class is that the resulting models are typically represented as a black box, so they cannot be directly processed by humans [27]–[29].”
BackgroundNeural-symbolic integration and the Semantic Web
Hitzler, P., Bianchi, F., Ebrahimi, M., & Sarker, M. K. (2020) · Semantic Web · doi:10.3233/sw-190368 · Cited by 70
“Consequently, researchers are investigating how to produce explanations for the behavior of deep learning systems [25].”
BackgroundAI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71
“Its broad relevance can be seen in the interdisciplinary nature of the scientific discussion that is currently taking place on such terms as interpretation, explanation and refined versions such as causability and causality in connection with AI methods [64,72–74].”
BackgroundInfluentialExplaining sentiment analysis results on social media texts through visualization
Jain, R., Kumar, A., Nayyar, A., Dewan, K., Garg, R., Raman, S., & Ganguly, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14432-y · Cited by 68
“On the downside, these neural networks are also categorized as black-box models [29].”
BackgroundThe Role of HCI in the Age of AI
Harper, R. H. R. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2019.1631527 · Cited by 68
A survey on outlier explanations
Panjei, E., Gruenwald, L., Leal, E., Nguyen, C., & Silvia, S. (2022) · The VLDB Journal · doi:10.1007/s00778-021-00721-1 · Cited by 67
“CNF is considered as an interpretable machine learning output [38].”
BackgroundTowards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67
“To address this, much research has been conducted on explainable AI (XAI) to develop new XAI algorithms and techniques [3, 7, 30, 36],”
BackgroundInterpretable Machine Learning for Discovery: Statistical Challenges and Opportunities
Allen, G. I., Gan, L., & Zheng, L. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040120-030919 · Cited by 115
Mohamed, E., Sirlantzis, K., & Howells, G. (2022) · Displays · doi:10.1016/j.displa.2022.102239 · Cited by 67
“Unlike Seifert et al. [14], Guidotti et al. [15], and Zhang et al. [16], whose surveys are extended to other analysis techniques such as confusion matrices, histograms, explanatory graphs [17], and decision trees for model analysis [18], this review is mainly focused on visualisation methods,…”
MethodSchoeffer, J., Kuehl, N., & Machowski, Y. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533218 · Cited by 67
“, [2, 4, 44, 82]), we confine ourselves to textual explanations (esp.”
BackgroundArtificial intelligence bias in medical system designs: a systematic review
Kumar, A., Aelgani, V., Vohra, R., Gupta, S. K., Bhagawati, M., Paul, S., Saba, L., Suri, N., Khanna, N. N., Laird, J. R., et al. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16029-x · Cited by 65
BRKAN, M., & BONNET, G. (2020) · European Journal of Risk Regulation · doi:10.1017/err.2020.10 · Cited by 63
A review of medical artificial intelligence
Liu, R., Rong, Y., & Peng, Z. (2020) · Global Health Journal · doi:10.1016/j.glohj.2020.04.002 · Cited by 63
Dissecting scientific explanation in AI (sXAI): A case for medicine and healthcare
Durán, J. M. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103498 · Cited by 62
Equity Beyond Bias in Language Technologies for Education
Mayfield, E., Madaio, M., Prabhumoye, S., Gerritsen, D., McLaughlin, B., Dixon-Román, E., & Black, A. W. (2019) · Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w19-4446 · Cited by 62
“NLP researchers have aggressively studied interpretability, offering explanation of results rather than predictions alone (Guidotti et al., 2018) - linguistic information is captured by newer neural language models of text (Conneau et al., 2018; Sommerauer and Fokkens, 2018) and speech (El-loumi et…”
BackgroundFlexible and Context-Specific AI Explainability: A Multidisciplinary Approach
Beaudouin, V., Bloch, I., Bounie, D., Clémençon, S., d'Alché-Buc, F., Eagan, J., Maxwell, W., Mozharovskyi, P., & Parekh, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3559477 · Cited by 61
Interpretability of Machine Learning Solutions in Public Healthcare: The CRISP-ML Approach
Kolyshkina, I., & Simoff, S. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.660206 · Cited by 60
On the Relation of Trust and Explainability: Why to Engineer for Trustworthiness
Kastner, L., Langer, M., Lazar, V., Schomacker, A., Speith, T., & Sterz, S. (2021) · 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00031 · Cited by 61
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69
Visser, R., Peters, T. M., Scharlau, I., & Hammer, B. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101357 · Cited by 58
“This includes concerns about the AI’s transparency, interpretability, accountability, and fairness (Arrieta et al., 2020; Guidotti et al., 2019; Mohseni et al., 2021).”
Machine learning applications in tobacco research: a scoping review
Fu, R., Kundu, A., Mitsakakis, N., Elton‐Marshall, T., Wang, W., Hill, S., Bondy, S. J., Hamilton, H. A., Selby, P. L., Schwartz, R. M., et al. (2021) · Tobacco Control · doi:10.1136/tobaccocontrol-2020-056438 · Cited by 57
Fairwashing: the risk of rationalization
Aïvodji, U., Arai, H., Fortineau, O., Gambs, S., Hara, S., & Tapp, A. (2019) · arXiv · doi:10.48550/arxiv.1901.09749 · Cited by 54
“In this paper, we focus on two black-box explanation tasks, namely model explanation and outcome explanation (Guidotti et al., 2018).”
BackgroundMethodInfluentialSharing Biomedical Data: Strengthening AI Development in Healthcare
Pereira, T., Morgado, J., Silva, F., Pelter, M. M., Dias, V. R., Barros, R., Freitas, C., Negrão, E., Flor de Lima, B., Correia da Silva, M., et al. (2021) · Healthcare · doi:10.3390/healthcare9070827 · Cited by 54
“An ideal AI-based model should be robust, reliable, and understandable [8].”
BackgroundGranlund, T., Stirbu, V., & Mikkonen, T. (2021) · SN Computer Science · doi:10.1007/s42979-021-00726-1 · Cited by 53
Rakhshan, K., Morel, J.-C., & Daneshkhah, A. (2021) · Journal of Building Engineering · doi:10.1016/j.jobe.2021.102791 · Cited by 53
“It is because this study intends to provide an easy-to-understand model that can be used by various stakeholders in the building sector who necessarily might not be able to use complex predictive models [47,48].”
BackgroundManaging Bias in Machine Learning Projects
Fahse, T., Huber, V., & van Giffen, B. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86797-3_7 · Cited by 66
Unveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future Directions
Noor, A. A., Manzoor, A., Mazhar Qureshi, M. D., Qureshi, M. A., & Rashwan, W. (2025) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.70018 · Cited by 52
Yepmo, V., Smits, G., & Pivert, O. (2022) · Data & Knowledge Engineering · doi:10.1016/j.datak.2021.101946 · Cited by 52
“In the case of explanations, with the outbreak of XAI, there has been a surge of general reviews these last years ([8, 9]); but to the best of our knowledge there is none devoted to the anomaly explanation methods.”
Explainable Sentiment Analysis with Applications in Medicine
Zucco, C., Liang, H., Fatta, G. D., & Cannataro, M. (2018) · 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2018.8621359 · Cited by 59
“[43], in which an explanation is thought as an”
BackgroundLeveraging rationales to improve human task performance
Das, D., & Chernova, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377512 · Cited by 52
“In [15] an ad-hoc genetic algorithm is used to generate neighbor nodes for a specific local instance, which is then trained by a decision tree classifier to generate a logic rule.”
MethodSolanke, A. A. (2022) · Forensic Science International: Digital Investigation · doi:10.1016/j.fsidi.2022.301403 · Cited by 52
ExplainExplore: Visual Exploration of Machine Learning Explanations
Collaris, D., & van Wijk, J. J. (2020) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis48177.2020.7090 · Cited by 57
“There are two main approaches in this field [21, 57]: either a model is used that is inherently interpretable, or an explanation is generated by means of a surrogate model.”
BackgroundLebovitz, S., Levina, N., & Lifshitz-Assaf, H. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16564 · Cited by 49
Explainable Deep Neural Network for Design of Electric Motors
Sasaki, H., Hidaka, Y., & Igarashi, H. (2021) · IEEE Transactions on Magnetics · doi:10.1109/tmag.2021.3063141 · Cited by 48
“The lack of interpretability and transparency is a common artificial intelligence problem [9].”
BackgroundZheng, C., Wu, Y., Shi, C., Ma, S., Luo, J., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581131 · Cited by 48
Robots Are Judging Me: Perceived Fairness of Algorithmic Recruitment Tools
Hilliard, A., Guenole, N., & Leutner, F. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.940456 · Cited by 47
AI ethics: Algorithmic determinism or self-determination? The GPDR approach
Milossi, M., Alexandropoulou-Egyptiadou, E., & Psannis, K. E. (2021) · IEEE Access · doi:10.1109/access.2021.3072782 · Cited by 46
“This can help to provide an overview of the types of processing that could have the highest impact on the fundamental rights and freedoms of data subjects in relation to the protection of their personal data [46].”
BackgroundAgent-Based Explanations in AI: Towards an Abstract Framework
Ciatto, G., Schumacher, M. I., Omicini, A., & Calvaresi, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-51924-7_1 · Cited by 45
“The survey by Guidotti et al. [13] is a nice entry point to explainable ML.”
BackgroundMethodExplainable Multi-Agent Systems Through Blockchain Technology
Calvaresi, D., Mualla, Y., Najjar, A., Galland, S., & Schumacher, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30391-4_3 · Cited by 48
Lin, M.-Y., Li, C.-C., Lin, P.-H., Wang, J.-L., Chan, M.-C., Wu, C.-L., & Chao, W.-C. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.663739 · Cited by 45
“Although AI technologies have achieved extraordinary advancement in a number of fields, the adoption of AI algorithms with the black-box issue in health care remains uncommon mainly due to physicians tending to take action only after realizing the rationale behind the results (30, 31).”
BackgroundSanti, L. A. D., Pasini, E., Santarelli, M. F., Genovesi, D., & Positano, V. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00719-3 · Cited by 53
Defining Locality for Surrogates in Post-hoc Interpretablity
Laugel, T., Renard, X., Lesot, M., Marsala, C., & Detyniecki, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07498 · Cited by 44
“We rely on admitted interpretability of the surrogate models as presented in the literature (Guidotti et al., 2018).”
MethodBridging the Transparency Gap: What Can Explainable AI Learn from the AI Act?
Gyevnár, B., Ferguson, N., & Schäfer, B. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230367 · Cited by 79
“, in [64, 3, 25, 18]), but they should remember that existence and scope of a new algorithm-centric “right to explanation” under the GDPR remain contested and so far no case law exists on its interpretation in this aspect.”
BackgroundMethodData science: a game changer for science and innovation
Grossi, V., Giannotti, F., Pedreschi, D., Manghi, P., Pagano, P., & Assante, M. (2021) · International Journal of Data Science and Analytics · doi:10.1007/s41060-020-00240-2 · Cited by 57
Self-citationDeep Weighted Averaging Classifiers
Card, D., Zhang, M., & Smith, N. A. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287595 · Cited by 37
“ways to provide explanations for predictions made by machine learning systems. Although there is currently an active debate in the field about how best to conceptualize and operationalize these terms [11, 13], recent research has broadly fallen into two camps. Some work has focused on models that are inherently interp…”
BackgroundExplaining Any Time Series Classifier
Guidotti, R., Monreale, A., Spinnato, F., Pedreschi, D., & Giannotti, F. (2020) · 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi50398.2020.00029 · Cited by 46
“We address the black box outcome explanation problem [22] in the domain of time series classification.”
BackgroundMethodSelf-citationTrust and Trustworthiness in AI
Durán, J. M., & Pozzi, G. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00843-2 · Cited by 47
“To answer this question, we will refer to the classification provided by Guidotti et al. (Guidotti et al., 2019, 93:15), which outlines four methods for opening the black box: (i) explaining the model, (ii) explaining the outcome, (iii) inspecting the black box internally, and (iv) providing a transparent solution.”
InfluentialAn assessment framework for explainable AI with applications to cybersecurity
Calzarossa, M. C., Giudici, P., & Zieni, R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11141-w · Cited by 40
“Nevertheless, the extant literature relies on the regulatory defi - nition, such as that reported in Bracke et al. (2019), for which explainability requires an explicit understanding of the factors which determine a given machine learning output (see also Guidotti et al. (2018)).”
Fifty Years of Prolog and Beyond
KÖRNER, P., LEUSCHEL, M., BARBOSA, J., COSTA, V. S., DAHL, V., HERMENEGILDO, M. V., MORALES, J. F., WIELEMAKER, J., DIAZ, D., ABREU, S., et al. (2022) · Theory and Practice of Logic Programming · doi:10.1017/s1471068422000102 · Cited by 40
Janik, A., Dodd, J. D., Ifrim, G., Sankaran, K., & Curran, K. M. (2021) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2582227 · Cited by 51
Understandable and trustworthy explainable robots: A sensemaking perspective
Papagni, G., & Koeszegi, S. T. (2020) · Paladyn Journal of Behavioral Robotics · doi:10.1515/pjbr-2021-0002 · Cited by 64
“” The research area of explain - able arti fi cial intelligence ( XAI ) , particularly in its “ goal - driven ” form ( rather than “ data - driven ” ) , stands as the most structured attempt in making AI systems ’ decisions understandable also by non - expert users [ 8,9,17,18 ] .”
BackgroundModel agnostic interpretability of rankers via intent modelling
Singh, J., & Anand, A. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3375234 · Cited by 54
“For other notions of interpretability and a more comprehensive description of the approaches we point the readers to [16].”
BackgroundRelation-Based Counterfactual Explanations for Bayesian Network Classifiers
Albini, E., Rago, A., Baroni, P., & Toni, F. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/63 · Cited by 39
“…explanations for AI systems (e.g. see [Ribeiro et al. , 2016; Lundberg and Lee, 2017; Ribeiro et al. , 2018] for some popular proposals and [Guidotti et al. , 2019] for a recent survey), as well as explanation meth-ods tailored to specific reasoning or machine learning meth-ods (e.g. see…”
BackgroundExploring Interpretable Predictive Models for Business Processes
Sindhgatta, R., Moreira, C., Ouyang, C., & Barros, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58666-9_15 · Cited by 53
“In parallel, there has been an increasing interest in the research community on interpretable or explainable machine learning [5].”
BackgroundResponsible AI by Design in Practice
Benjamins, R., Barbado, A., & Sierra, D. (2019) · arXiv · doi:10.48550/arxiv.1909.12838 · Cited by 38
Kabir, S., Hossain, M. S., & Andersson, K. (2025) · Algorithms · doi:10.3390/a18090556 · Cited by 44
“Global explanation describes the overall behavior of a model across the entire input space, allowing insights into feature importance and model structure [53].”
Predicting online shopping cart abandonment with machine learning approaches
Rausch, T. M., Derra, N. D., & Wolf, L. (2022) · International Journal of Market Research · doi:10.1177/1470785320972526 · Cited by 36
“…models’ complexity in terms of computation time and computation effort (e.g., numbers of hyperparameters to be optimized) is of similar importance regarding the application in practice and should therefore be considered as well (Doshi-Velez & Kim, 2017; Guidotti et al., 2019; Tambe et al., 2019).”
BackgroundPersonalization of Deep Learning
Schneider, J., & Vlachos, M. (2021) · Springer eBooks · doi:10.1007/978-3-658-32182-6_14 · Cited by 63
INTRUSION DETECTION IN COMPUTER NETWORKS USING LATENT SPACE REPRESENTATION AND MACHINE LEARNING
Hamolia, V., Melnyk, V. A., Zhezhnych, P., & Shilinh, A. (2020) · International Journal of Computing · doi:10.47839/ijc.19.3.1893 · Cited by 38
“Some of the ML methods under consideration have non-interceptable issues that differ in nature and complexity illustrated in [8] and [9].”
MethodILIME: Local and Global Interpretable Model-Agnostic Explainer of Black-Box Decision
Elshawi, R., Sherif, Y., Al-Mallah, M. H., & Sakr, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28730-6_4 · Cited by 56
Open the Black Box Data-Driven Explanation of Black Box Decision Systems
Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Pappalardo, L., Ruggieri, S., & Turini, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09936 · Cited by 29
“A different stream of approaches, still in the XbD setting, focuses on the local behavior of a black box [21], searching for an explanation of the decision made for a specific instance.”
BackgroundMethodInfluentialSelf-citationSalimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025) · Informatics · doi:10.3390/informatics12040119 · Cited by 39
“We adapt the categorization used by Adadi and Berrada [11] and Guidotti et al. [12] (refer to Figure 1).”
Degradation-Resistant Unfolding Network for Heterogeneous Image Fusion
He, C., Li, K., Xu, G., Zhang, Y., Hu, R., Guo, Z., & Li, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01159 · Cited by 37
“…it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. based methods are criticized as black boxes [6] for lacking interpretability, which inevitably suppresses the fusion performance, especially in complex degradation scenarios.”
BackgroundGlobal Aggregations of Local Explanations for Black Box models
van der Linden, I., Haned, H., & Kanoulas, E. (2019) · arXiv · doi:10.48550/arxiv.1907.03039 · Cited by 35
“The difficulty lies in questions regarding what makes for a valid, reliable and useful explanation; what are the desiderata? Prior work on explaining black box models has focused on either global or local explanations, where global explanations aim to explain a model’s decision making process in general, while local e…”
Towards a Taxonomy for the Opacity of AI Systems
Facchini, A., & Termine, A. (2022) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-031-09153-7_7 · Cited by 34
Towards Explainable Neural-Symbolic Visual Reasoning
Bennetot, A., Laurent, J.-L., Chatila, R., & Díaz-Rodríguez, N. (2019) · arXiv · doi:10.48550/arxiv.1909.09065 · Cited by 32
“One of the goals of having interpretability in a model is to explain its reasoning by expressing it in a way that is understandable and readable by human beings, while highlighting the biases learned by the model, in order to validate or invalidate its decision rationale [Guidotti et al., 2018].”
MethodMultimodal explainability via latent shift applied to COVID-19 stratification
Guarrasi, V., Tronchin, L., Albano, D., Faiella, E., Fazzini, D., Santucci, D., & Soda, P. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110825 · Cited by 31
“In particular, many authors agree that explanations are strongly recommended in medical applications [22], because this would help mapping explainability with causability that, in turns, would allow practitioners to understand why a model came up with a result.”
(2020) · doi:10.1007/978-3-030-51924-7_4 · Cited by 31
Interpretable Deep Models for Cardiac Resynchronisation Therapy Response Prediction
Puyol-Antón, E., Chen, C., Clough, J. R., Ruijsink, B., Sidhu, B. S., Gould, J., Porter, B., Elliott, M., Mehta, V., Rueckert, D., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59710-8_28 · Cited by 30
On the Art and Science of Machine Learning Explanations
Hall, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02909 · Cited by 30
“Others have defined key terms and put forward general motivations for better interpretability of machine learning models (and advocated for stronger scientific rigor in certain cases) [9], [11], [13], [18].”
BackgroundUnveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future Directions
Aziz, N. A., Manzoor, A., Mazhar Qureshi, M. D., Qureshi, M. A., & Rashwan, W. (2024) · doi:10.1101/2024.08.10.24311735 · Cited by 30
Lee, J., Lee, J. Y., Lee, M. H., Lee, M., Kim, Y. W., Hyung, J., Kim, K., Cha, Y., & Koo, J.-Y. (2022) · Water Science & Technology Water Supply · doi:10.2166/ws.2022.038 · Cited by 32
Bot-Detective: An explainable Twitter bot detection service with crowdsourcing functionalities
Kouvela, M., Dimitriadis, I., & Vakali, A. I. (2020) · International Conference on Management of Emergent Digital EcoSystems (MEDES) · doi:10.1145/3415958.3433075 · Cited by 45
“Our module is based on a state-of-the-art method[26] called LIME[39].”
BackgroundMethodEIGER: automated IOC generation for accurate and interpretable endpoint malware detection
Kurogome, Y., Otsuki, Y., Kawakoya, Y., Iwamura, M., Hayashi, S., Mori, T., & Sen, K. (2019) · Annual Computer Security Applications Conference · doi:10.1145/3359789.3359808 · Cited by 40
“We shall brie y discuss the two research direction to address this issue, instead of the comprehensive survey [23].”
Background
Metadata
- Authors
- Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D.
- Affiliations
- University of Pisa (IT)
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- Venue
- ACM Computing Surveys
- Published
- 2018-08-22
- Volume
- 51
- Issue
- 5
- Pages
- 1-42
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2786715987
- Cited by
- 629
- References
- 121
- Citations by year
- 2026: 3 · 2025: 62 · 2024: 82 · 2023: 86 · 2022: 92 · 2021: 145 · 2020: 105 · 2019: 40 · 2018: 13
- FWCI
- 42.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- Scuola Normale Superiore di Pisa (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Horizon 2020 Framework Programme (-2014-2015, 654024)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- explainable AI, black-box models, interpretability, decision support system, explanation methods, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.