The DOI matches its Crossref record; 170 of the 170 references it cites were found in a registry or an index.
Explainable AI: A Review of Machine Learning Interpretability Methods
Open accessAbstract
Recent advances in artificial intelligence (AI) have led to its widespread industrial adoption, with machine learning systems demonstrating superhuman performance in a significant number of tasks. However, this surge in performance, has often been achieved through increased model complexity, turning such systems into “black box” approaches and causing uncertainty regarding the way they operate and, ultimately, the way that they come to decisions. This ambiguity has made it problematic for machine learning systems to be adopted in sensitive yet critical domains, where their value could be immense, such as healthcare. As a result, scientific interest in the field of Explainable Artificial Intelligence (XAI), a field that is concerned with the development of new methods that explain and interpret machine learning models, has been tremendously reignited over recent years. This study focuses on machine learning interpretability methods; more specifically, a literature review and taxonomy of these methods are presented, as well as links to their programming implementations, in the hope that this survey would serve as a reference point for both theorists and practitioners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While valuable, these techniques can sometimes be inconsistent or fail to capture the true decision-making process of highly complex models, potentially leading to a false sense of interpretability66.”
From Bias recognition and mitigation strategies in artificial intelligence healthcare applications (2025) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These methods facilitate a deeper understanding of the model’s internal logic, support debugging and optimization processes, and help identify key variables that most strongly influence students’ behavioral intentions toward GenAI adoption (Linardatos et al., 2020).”
From A two-step structural equation modeling and explainable machine learning framework for understanding university students’ adoption of generative AI: balancing intrinsic motivations and perceived risks (2026) · cites this paper as Linardatos et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Yet, there is no common taxonomy neither within one field (such as data science) nor between different fields (law and data science).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Linardatos et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite the substantial amount of research devoted to solving the black-box issue of AI, many scholars in the field agree that a clear relevant taxonomy is missing (Doshi-Velez and Kim, 2017; Adadi and Berrada, 2018; Linardatos et al., 2021).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Linardatos et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior research on XAI across industries, has also found that the perceived quality of explanations is closely tied to their actionability and practical relevance for end users [8].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [8]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“An improvement over the initial method was proposed in [24], where the well-known Krizhevsky network [25] was utilised in order to outperform state-of-the-art saliency models by a large margin, increasing the amount of explained information by 67% when compared to state-of-the art.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · cited 1×
“On the one hand, there are the so called black-box models, which include deep learning [2] and ensembles [8–10].”
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 3×
“[59] PDPbox InterpretML Skater PH G Agnostic tab 589.”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“The breakDown method, as proposed in [62], is similar to SHAP [48], as both, based on the conditioned responses of a black-box model, attempt to attribute them proportionally to the input features.”
Classification and Regression by randomForest
Liaw, A. I., & Wiener, M. C. (2007) · Cited by 18,594
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 8×
“[44] showed that, while the Deconvolution [32], Guided BackPropagation [31], and LRP [37] methods help in interpreting deep neural networks, they do not produce the theoretically correct interpretation, even in the simplest neural network setting; a linear model developed while using data that were produced by a linea…”
InfluentialLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 2×
“In many cases, this is achieved through exploiting the gradient information flowing through the layers of the network, Grad-CAM [35], a direct extension of [34], being a prime and most influential example in terms of citations per year.”
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 1×
“A more recent study carried out by Arrieta et al. [21] introduced a different type of arrangement that initially distinguishes transparent and post-hoc methods and subsequently created sub-categories.”
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Koch, C., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,271 · cited 1×
“A substantial portion of attention regarding python tools is focused on deep learning for images more specifically on the concept of saliency in images, as initially proposed in [22].”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 1×
“A more recent study carried out by Arrieta et al. [21] introduced a different type of arrangement that initially distinguishes transparent and post-hoc methods and subsequently created sub-categories.”
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · cited 3×
“To this end, among the other methods used, a new method that improves upon the FGSM [116] algorithm was introduced, whereby the FGSM was repeated many times with small step size, truncating the intermediate results after each step in the process, so that the produced adversarial examples (pixels in…”
InfluentialMachine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“This has radically changed over the past decade as a combination of more powerful machines, improved learning algorithms, as well as easier access to vast amounts of data enabled advances in Machine Learning (ML) and led to its widespread industrial adoption[1].”
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859 · cited 1×
“Carlini and Wagner [123] introduced three novel adversarial attack algorithms, based on the L 0 , L 2 , and L ∞ norms, respectively, which were very effective against neural networks, even those where defensive distillation technique [124] had been applied.”
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 3×
“The proposed attacks aim to address the same minimal perturbation problem as Szegedy et al [115], but they formulate it using the margin loss instead of cross-entropy loss, thus minimising the distance between adversarial and benign examples in a more direct way.”
InfluentialGlobal Sensitivity Analysis. The Primer
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., & Tarantola, S. (2007) · doi:10.1002/9780470725184 · Cited by 6,430
Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates
Sobol′, I. M. (2001) · Mathematics and Computers in Simulation · doi:10.1016/s0378-4754(00)00270-6 · Cited by 6,389 · cited 1×
“…was formalised by Sobol, who proposed the approximation of sensitivity indices of first and higher order while using Monte-Carlo methods [101], while Saltelli [102] and Saltelli et al. [103] improved upon Sobol’s approach while using more efficient sampling techniques for first, higher,…”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 4×
“Although intuitive, these definitions lack mathematical formality and rigorousness [17].”
InfluentialLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“The concepts of interpretability and explainability are hard to rigorously define; however, multiple attempts have been made towards that goal, the most emblematic works being [14,15].”
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Gradients: first proposed in [23], the gradients explanation technique, as its name suggests, is gradient-based attribution method, according to which each gradient quantifies how much a change in each input dimension would a change the predictions in a small neighborhood around the input.”
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×
“In order to test the sensitivity of deep learning models, Moosavi-Dezfooli et al. proposed DeepFool [117], a method that generates minimum-perturbation adversarial examples that are optimised for the L 2 norm.”
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Factorial Sampling Plans for Preliminary Computational Experiments
Morris, M. D. (1991) · Technometrics · doi:10.1080/00401706.1991.10484804 · Cited by 4,943
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 2×
“Kindermans et al. [44] showed that, while the Deconvolution [32], Guided BackPropa-gation [31], and LRP [37] methods help in interpreting deep neural networks, they do not produce the theoretically correct interpretation, even in the simplest neural network setting; a linear model developed while…”
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 8×
“Gradients: first proposed in [23], the gradients explanation technique, as its name suggests, is gradient-based attribution method, according to which each gradient quantifies how much a change in each input dimension would a change the predictions in a small neighborhood around the input.”
InfluentialThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
A survey of decision tree classifier methodology
Safavian, S. R., & Landgrebe, D. (1991) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.97458 · Cited by 3,808 · cited 1×
“On the other hand, there are the so called white-box or glass-box models, which easily produce explainable results—with common examples, including linear [11] and decision-tree based [12] models.”
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00097 · Cited by 3,639 · cited 2×
“[36] Grad-CAM++ PH L Specific img 81 2018”
Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., & Tarantola, S. (2010) · Computer Physics Communications · doi:10.1016/j.cpc.2009.09.018 · Cited by 3,623 · cited 1×
“…Sobol, who proposed the approximation of sensitivity indices of first and higher order while using Monte-Carlo methods [101], while Saltelli [102] and Saltelli et al. [103] improved upon Sobol’s approach while using more efficient sampling techniques for first, higher, as well as total-effect indices.”
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“In [88], a framework for fair classification, which consisted of two parts, was presented.”
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495 · cited 1×
“Papernot et al. [134] proposed a novel method for generating adversarial examples by examining the inputs that were provided to a deep neural network and the corresponding the labels that were assigned by the network.”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 6×
“[44] showed that, while the Deconvolution [32], Guided BackPropagation [31], and LRP [37] methods help in interpreting deep neural networks, they do not produce the theoretically correct interpretation, even in the simplest neural network setting; a linear model developed while using data that were produced by a linea…”
InfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 6×
“One of the most popular definitions of interpretability is the one of Doshi-Velez and Kim, who, in their work [15], define it as “the ability to explain or to present in understandable terms to a human”.”
InfluentialDistillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.41 · Cited by 2,982 · cited 1×
“Carlini and Wagner [123] introduced three novel adversarial attack algorithms, based on the L 0 , L 2 , and L ∞ norms, respectively, which were very effective against neural networks, even those where defensive distillation technique [124] had been applied.”
Boosting Adversarial Attacks with Momentum
Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., & Li, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00957 · Cited by 2,956 · cited 1×
“Dong et al. [121] promoted the use of momentum in oder to enhance the process of creating adversarial instances while using iterative algorithms, thus introducing the a broad class of adversarial momentum-based iterative algorithms.”
Permutation importance: a corrected feature importance measure
Altmann, A., Toloşi, L., Sander, O., & Lengauer, T. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq134 · Cited by 2,843 · cited 1×
“Permutation importance (PIMP) [57] is a heuristic approach that attempts to correct the feature importance bias through the normalisation of feature importance measures.”
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · cited 8×
“Firstly, the authors define the class of additive feature at-tribution methods, which unifies six current methods, including LIME [45], DeepLIFT [29], and Layer-Wise Relevance Propagation [49], which all use the same explanation model.”
InfluentialUniversal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 4×
“[97], building on from [92], studied the problem of producing calibrated probability scores, the end goal of many machine learning applications, while, at the same time, ensuring fair decisions across different demographic segments.”
InfluentialExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 3×
“There is not a concrete mathematical definition for interpretability or explainability, nor have they been measured by some metric; however, a number of attempts have been made [14–16] in order to clarify not only these two terms, but also related concepts such as comprehensibility.”
InfluentialOne Pixel Attack for Fooling Deep Neural Networks
Su, J., Vargas, D. V., & Sakurai, K. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2890858 · Cited by 2,406 · cited 1×
“One-pixel Attack: Su et al. [127] showed how neural networks can be fooled by altering the value of just a single input pixel.”
A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output
Saltelli, A., Tarantola, S., & Chan, K. P.-S. (1999) · Technometrics · doi:10.1080/00401706.1999.10485594 · Cited by 2,275 · cited 1×
“Saltelli et al. [105] improved upon FAST to compute the total-effect indices, while Tarantola et al. [106] extended random balance designs, applied by Satterthwait in regression problems, to sensitivity analysis for non-linear, non-additive models by combining them with FAST (RBD-FAST).”
Making best use of model evaluations to compute sensitivity indices
Saltelli, A. (2002) · Computer Physics Communications · doi:10.1016/s0010-4655(02)00280-1 · Cited by 2,228 · cited 1×
“…by Sobol, who proposed the approximation of sensitivity indices of first and higher order while using Monte-Carlo methods [101], while Saltelli [102] and Saltelli et al. [103] improved upon Sobol’s approach while using more efficient sampling techniques for first, higher, as well as total-effect…”
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 1×
“[20] published a survey in 2019, in which they created an interpretability framework in the hope that it would help to bridge the aforementioned gap in the field.”
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2×
“Ancors: in [50], another model-agnostic interpretability approach that works for any black-box model with a high probability guarantee was proposed.”
An effective screening design for sensitivity analysis of large models
Campolongo, F., Cariboni, J., & Saltelli, A. (2007) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2006.10.004 · Cited by 2,011 · cited 1×
“By devising a more effective sampling strategy as well as other improvements, Campolongo et al. [111] proposed an improved version of Morris’s method.”
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861 · cited 1×
“Originally proposed in [60], ICE plots is a model agnostic interpretability method, which builds on the concept of PDPs and improves it.”
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 2×
“Disparate impact testing [72] is a model agnostic method that is able to assess the fairness of a model, but is not able to provide any insight or detail regarding the causes of any discovered bias.”
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 2×
“In [65], Microsoft presented two case studies on real medical data, where naturally interpretable generalized additive models with pairwise interactions (GA 2 Ms), as originally proposed in [66], achieved state-of-the-art accuracy, showing that GA 2 Ms are the first step towards deploying…”
Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J., & Hsieh, C.-J. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140448 · Cited by 1,656 · cited 1×
“…one has access to the prediction probability scores (rather than just the predicted labels) of a classifier and the respective inputs, Chen et al. [128] proposed an algorithm to infer the gradient information by observing the changes in the prediction scores, thus eliminating the need for a…”
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 1×
“The fairness of word embeddings, a vectorised representation of text data, used in many real world machine learning application, was studied in [85] and it was revealed that word embeddings, even those that were trained on Google News articles, carry strong gender bias.”
Visualizing the effects of predictor variables in black box supervised learning models
Apley, D. W., & Zhu, J. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12377 · Cited by 1,535 · cited 1×
“Another method that is closely-related to PDPs is the Accumulated Local Effect (ALE) plots [61].”
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978392 · Cited by 1,530 · cited 1×
“Sharif et al. [141] proposed a method for fooling face recognition neural networks by modifying the original images through the insertion of 3D printed sunglasses in the original face images.”
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 2×
“[38] iNNvestigate tf-explain PH L Specific img 113.”
Weisberg, S. (2005) · Wiley Series in Probability and Statistics · doi:10.1002/0471704091 · Cited by 1,446
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×
“In their study, Jia and Liang [152] investigated generating examples for reading comprehension tasks: given a paragraph and a related question, the model has to generate an answer.”
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · cited 1×
“Adversarial debiasing [77] is a framework for mitigating biases concerning demographic segments in machine learning systems by selecting a feature regarding the segment of interest and, subsequently, training a main model and an adversarial model simultaneously.”
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360
Understanding Neural Networks Through Deep Visualization
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,358 · cited 2×
“[40] Deep Visualization Toolbox PH L Specific img 221.”
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · cited 1×
“Deep Taylor decomposition [41] is a method that decomposes a neural network’s output, for given input instance, into contributions of this instance by backpropagating the explanations from the output layer to the input.”
Delving into Transferable Adversarial Examples and Black-box Attacks
Liu, Y., Chen, X., Liu, C., & Song, D. (2016) · arXiv · doi:10.48550/arxiv.1611.02770 · Cited by 1,310 · cited 1×
“Liu et al. [131] performed a thorough investigation on the transferability of both non-targeted and targeted adversarial examples while using models and datasets of large scale, concluding that while transferring non-targeted adversarial examples can be very effective in fooling neural networks,…”
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · cited 1×
“In [73], three different data preprocessing techniques to ensure fairness in classification tasks are analysed.”
A Reductions Approach to Fair ClassificationPreprint
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J. C., & Wallach, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02453 · Cited by 1,285 · cited 1×
“Agarwal et al. [95] introduced a systematic framework that incorporates many other previously outlined definitions of fairness, treating them as special cases.”
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · cited 1×
“In [42], the idea of Concept Activation Vectors (CAVs) was introduced, providing a human-friendly interpretation of a neural network internal state; an intuition of how sensitive a prediction is to a user-defined concept and how important the concept is to the classification itself.”
Adaptive deconvolutional networks for mid and high level feature learning
Zeiler, M. D., Taylor, G. W., & Fergus, R. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126474 · Cited by 1,217 · cited 2×
“Deconvolution, as proposed in [32], is a technique for visualizing Convolutional Neural Networks (CNNs or ConvNets) by utilising De-convolutional Networks (DeconvNets or DCNNs), as initially proposed in [33].”
(2013) · Cited by 1,126
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory
Cukier, R. I., Fortuin, C. M., Shuler, K. E., Petschek, A. G., & Schaibly, J. H. (1973) · The Journal of Chemical Physics · doi:10.1063/1.1680571 · Cited by 1,046 · cited 1×
“Cukier et al. [104] proposed the Fourier Amplitude Sensitivity Test (FAST) method to improve the approximation of Sobol’s indices.”
A new uncertainty importance measure
Borgonovo, E. (2007) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2006.04.015 · Cited by 1,026 · cited 1×
“One example is Borgonovo’s [112] distribution or density based sensitivity indices, which measure the distance or the divergence between the unconditional output distribution and output distribution when conditioned on one or more input variables.”
Decision-based adversarial attacks: Reliable attacks against black-box machine learning models
Brendel, W., Rauber, J., & Bethge, M. (2017) · arXiv · doi:10.48550/arxiv.1712.04248 · Cited by 1,018 · cited 1×
“Brendel et al. [135] highlighted the lack of scientific studies regarding decision-based adversarial attacks and pinpointed to the benefits and the versatility of such attacks, namely that they can be used against any black-box model, require only the observing of the model’s final decisions, are…”
McCullagh, P., & Nelder, J. A. (2019) · doi:10.1201/9780203753736 · Cited by 1,013
Saliency detection by multi-context deep learning
Zhao, R., Ouyang, W., Li, H., & Wang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298731 · Cited by 995 · cited 1×
“Furthermore, in [26], a task-specific pre-training scheme was designed in order to make the multi-context modeling suited for saliency detection.”
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Optimized Pre-Processing for Discrimination Prevention
Calmon, F. P., Wei, D., Vinzamuri, B., Ramamurthy, K. N., & Varshney, K. R. (2017) · neural information processing systems · Cited by 978
Adversarial Attacks on Neural Networks for Graph Data
Zügner, D., Akbarnejad, A., & Günnemann, S. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220078 · Cited by 968 · cited 1×
“In the works of Zugner et al. [138] and Dai et al. [139], adversarial examples in graph-structured data were studied.”
HotFlip: White-Box Adversarial Examples for Text Classification
Ebrahimi, J., Rao, A., Lowd, D., & Dou, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2006 · Cited by 922 · cited 2×
“Such words or phrases are identified as more or less influential based on the influence of their individual characters, similarly to [150].”
Adversarial Attacks on Neural Network PoliciesPreprint
Huang, S., Papernot, N., Goodfellow, I., Duan, Y., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02284 · Cited by 921 · cited 1×
“Adversarial example vulnerability also exists in deep reinforcement learning modelling, as demonstrated by Huang et al. [145].”
Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6311 · Cited by 918 · cited 1×
“…which follows the synonyms substitution strategy, but replaces words that are based on the word saliency and classification probability TextFooler [161] generates adversarial examples for text by utilising word embedding distance and part-of-speech matching to first identify the most important…”
Generalized Additive Models: Some Applications
Hastie, T., & Tibshirani, R. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478440 · Cited by 870
Consumer credit-risk models via machine-learning algorithms
Khandani, A. E., Kim, A. J., & Lo, A. W. (2010) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2010.06.001 · Cited by 864
Generating Natural Language Adversarial Examples
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivastava, M., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1316 · Cited by 787 · cited 2×
“Alzantot et al. [153] proposed a method to generate adversarial examples for text while using a population-based genetic algorithm.”
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · cited 1×
“In [43], an interpretability technique for neural networks operating in the natural language processing (NLP) domain was proposed.”
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · cited 1×
“In [87], three approaches for making the naive Bayes classifier discrimination-free were proposed.”
Explanations based on the Missing: Towards Contrastive Explanations with Pertinent NegativesPreprint
Dhurandhar, A., Chen, P., Luss, R., Tu, C., Ting, P., Shanmugam, K., & Das, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07623 · Cited by 757 · cited 2×
“Originally proposed in [51], the contrastive explanations method (CEM) is capable of generating, what the authors call, contrastive explanations for any black box model.”
HopSkipJumpAttack: A Query-Efficient Decision-Based Attack
Chen, J., Jordan, M. I., & Wainwright, M. J. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00045 · Cited by 718 · cited 1×
“Introduced in [130], HopSkipJumpAttack is a group of adversarial-example generating algorithms that rely on binary information regarding the decision boundary and Monte Carlo methods in order to approximate the direction of the gradient.”
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694 · cited 1×
“Following up from other studies demonstrating that the exclusion of sensitive features cannot fully eradicate discrimination from model decisions, Kamishima et al. [99] presented and analysed three major causes of unfairness in machine learning: prejudice, underestimation, and negative legacy.”
Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
Gao, J., Lanchantin, J., Soffa, M. L., & Qi, Y. (2018) · 2018 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00016 · Cited by 693
Learning to Explain: An Information-Theoretic Perspective on Model InterpretationPreprint
Chen, J., Song, L., Wainwright, M. J., & Jordan, M. I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07814 · Cited by 690 · cited 1×
“L2X [58] is a real-time instance-wise feature selection method that can also be used for model interpretation.”
Fairness Constraints: Mechanisms for Fair Classification
Zafar, M. B., Valera, I., Rodriguez, M. G., & Gummadi, K. P. (2015) · arXiv · doi:10.48550/arxiv.1507.05259 · Cited by 645 · cited 2×
“In [94], a new metric for evaluating decision boundary fairness both in terms of disparate treatment and disparate impact at the same time, with respect to one or more sensitive features was introduced.”
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
Ren, S., Deng, Y., He, K., & Che, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1103 · Cited by 635 · cited 1×
“Ren et al. [160] proposed a greedy algorithm for textual adversarial example generation , alled probability weighted word saliency (PWWS), which follows the synonyms substitution strategy, but replaces words that are based on the word saliency and classification probability TextFooler [161]…”
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · cited 1×
“Following up from their previous work [94], Zafar et al. [93] introduced a novel notion of unfairness, which was defined through the rates of misclassification, called disparate mistreatment.”
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“Another survey that attempted to categorise the existing explainability methods is this of Guidotti et al. [19].”
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks
Iyyer, M., Wieting, J., Gimpel, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1170 · Cited by 615 · cited 1×
“Samanta and Mehta [147] as well as Iyyer et al. [148] proposed methods for generating adversarial sentences that are both grammatically correctly and in agreement with the syntax of the original sentences.”
Polikar, R. (2012) · Ensemble Machine Learning · doi:10.1007/978-1-4419-9326-7_1 · Cited by 600
Adversarial Attack on Graph Structured DataPreprint
Dai, H., Li, H., Tian, T., Huang, X. S., Wang, L., Zhu, J., & Song, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02371 · Cited by 593 · cited 1×
“In the works of Zugner et al. [138] and Dai et al. [139], adversarial examples in graph-structured data were studied.”
BERT-ATTACK: Adversarial Attack Against BERT Using BERT
Li, L., Ma, R., Guo, Q., Xue, X., & Qiu, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.500 · Cited by 574
EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples
Chen, P.-Y., Sharma, Y., Zhang, H., Yi, J., & Hsieh, C.-J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11302 · Cited by 569
Accurate intelligible models with pairwise interactions
Lou, Y., Caruana, R., Gehrke, J., & Hooker, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487579 · Cited by 564 · cited 2×
“…on real medical data, where naturally interpretable generalized additive models with pairwise interactions (GA 2 Ms), as originally proposed in [66], achieved state-of-the-art accuracy, showing that GA 2 Ms are the first step towards deploying interpretable high-accuracy models in applications…”
(2016) · neural information processing systems · Cited by 556
The Shapley Value: Essays in Honor of Lloyd S. Shapley
(1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528446 · Cited by 527 · cited 1×
“Under this variation, a much desired property, which is known as completeness or Efficiency [28] or Summation to Delta [29], is satisfied: the attributions sum up to the target output minus the target output that was evaluated at the baseline.”
BAE: BERT-based Adversarial Examples for Text Classification
Garg, S., & Ramakrishnan, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.498 · Cited by 478
Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 469 · cited 2×
“The fact that the majority of notions or definitions of machine learning fairness merely focus on predefined social segments was criticised in [96].”
Building Classifiers with Independency Constraints
Calders, T., Kamiran, F., & Pechenizkiy, M. (2009) · 2009 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2009.83 · Cited by 468 · cited 1×
“Finally the idea behind the third preprocessing technique, as initially presented in [75], is to apply different weights to the instances of the dataset based on frequency counts with respect to the sensitive column.”
Derivative based global sensitivity measures and their link with global sensitivity indices
Sobol’, I. M., & Kucherenko, S. (2009) · Mathematics and Computers in Simulation · doi:10.1016/j.matcom.2009.01.023 · Cited by 445 · cited 1×
“Being introduced by Sobol and Kucherenko [114], the method of derivative-based global sensitivity measures (DGSM) is based on finding the averages of local derivatives while using Monte Carlo or Quasi Monte Carlo sampling methods.”
Global sensitivity measures from given data
Plischke, E., Borgonovo, E., & Smith, C. L. (2013) · European Journal of Operational Research · doi:10.1016/j.ejor.2012.11.047 · Cited by 429 · cited 1×
“Building on from the work of Borgonovo, Plischke et al. [113] introduced a new class of estimators for approximating density-based sensitivity measures, independent of the sampling generation method used.”
Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers
Binder, A., Montavon, G., Lapuschkin, S., Müller, K.-R., & Samek, W. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-44781-0_8 · Cited by 418
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · cited 1×
“The second technique is called “massaging the dataset” and it was originally proposed in [74].”
Delayed Impact of Fair Machine Learning
Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., & Hardt, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/862 · Cited by 373
Deep Text Classification Can be Fooled
Liang, B., Li, H., Su, M., Bian, P., Li, X., & Shi, W. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/585 · Cited by 370
TextBugger: Generating Adversarial Text Against Real-world Applications
Li, J., Ji, S., Du, T., Li, B., & Wang, T. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23138 · Cited by 343 · cited 1×
“Textbugger [157] works in similar fashion, providing improvements over DeepWordBug through the introduction of novel scoring functions.”
Simple Black-Box Adversarial Attacks on Deep Neural Networks
Narodytska, N., & Kasiviswanathan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.172 · Cited by 320 · cited 1×
“In their study [129], Narodytska et al. focused on generating adversarial examples for any deep convolutional neural network without prior knowledge of the internal workings of the network in question.”
Random balance designs for the estimation of first order global sensitivity indices
Tarantola, S., Gatelli, D., & Mara, T. A. (2006) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2005.06.003 · Cited by 319 · cited 1×
“Saltelli et al. [105] improved upon FAST to compute the total-effect indices, while Tarantola et al. [106] extended random balance designs, applied by Satterthwait in regression problems, to sensitivity analysis for non-linear, non-additive models by combining them with FAST (RBD-FAST).”
Supersparse linear integer models for optimized medical scoring systems
Ustun, B., & Rudin, C. (2016) · Machine Learning · doi:10.1007/s10994-015-5528-6 · Cited by 313 · cited 1×
“Ustun and Rudin [64] proposed Supersparse Linear Integer Models (SLIM), a type of predictive system that only allows for additions, subtraction, and multiplications of input features to generate predictions, thus being highly interpretable.”
Adversarial Attacks on Neural Networks for Graph Data
Zügner, D., Akbarnejad, A., & Günnemann, S. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/872 · Cited by 301
Decision Theory for Discrimination-Aware Classification
Kamiran, F., Karim, A., & Zhang, X. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.45 · Cited by 300 · cited 1×
“Kamiran et al. [78] pointed out that many of the methods that make classifiers aware of discriminatory biases require data modifications or algorithm tweaks and they are not very flexible with respect to multiple sensitive feature handling and control over the performance vs. discrimination trade-off.”
The Social Cost of Strategic Classification
Milli, S., Miller, J. P., Dragan, A. D., & Hardt, M. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287576 · Cited by 285 · cited 1×
“Milli et al. [83] also studied how individuals adjust their behaviour strategically to manipulate decision rules in order to gain favourable treatment from decision-making models.”
Spatially Transformed Adversarial Examples
Xiao, C., Zhu, J.-Y., Li, B., He, W., Liu, M., & Song, D. X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02612 · Cited by 275 · cited 1×
“Spatially Transformed Attack: Xiao et al. [126] proposed perturbing images by performing a slight spatial transformation such as translating, rotating and/or distorting the image features.”
Learning how to explain neural networks: PatternNet and PatternAttribution
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K. R., Erhan, D., Kim, B., & Dähne, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05598 · Cited by 264 · cited 2×
“[44] iNNvestigate PH L Specific img 41.”
Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J. M., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02012 · Cited by 263 · cited 2×
“[97], building on from [92], studied the problem of producing calibrated probability scores, the end goal of many machine learning applications, while, at the same time, ensuring fair decisions across different demographic segments.”
Pathologies of Neural Models Make Interpretations Difficult
Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., & Boyd-Graber, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1407 · Cited by 245 · cited 1×
“Feng et al. [159] introduced a process, called “input reduction”, which can expose issues regarding overconfidence and oversensitivity in natural language processing models.”
The Disparate Effects of Strategic Manipulation
Hu, L. J., Immorlica, N., & Vaughan, J. W. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287597 · Cited by 239 · cited 2×
“In [82] strategic manipulation models were studied and adapted in an environment of social inequality, in which different social groups have to pay different costs of manipulation.”
From Parity to Preference-based Notions of Fairness in ClassificationPreprint
Zafar, M. B., Valera, I., Gomez-Rodriguez, M., Gummadi, K. P., & Weller, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00010 · Cited by 231 · cited 1×
“In another work by Zafar et al. [100], it was pointed out that many of the existing notions of fairness, regarding treatment or impact, are too rigorous and restrictive and, as a result, tend to hinder the overall model performance.”
Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Celis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287586 · Cited by 231 · cited 1×
“Celis et al. [98] highlighted that, although efforts have been made in recent studies to achieve fairness with respect to some particular metric, some important metrics have been ignored, while some of the proposed algorithms are not supported by a solid theoretical background.”
Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
Kümmerer, M., Theis, L., & Bethge, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.1045 · Cited by 231 · cited 1×
“An improvement over the initial method was proposed in [24], where the well-known Krizhevsky network [25] was utilised in order to outperform state-of-the-art saliency models by a large margin, increasing the amount of explained information by 67% when compared to state-of-the art.”
Towards Crafting Text Adversarial SamplesPreprint
Samanta, S., & Mehta, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02812 · Cited by 229 · cited 1×
“Samanta and Mehta [147] as well as Iyyer et al. [148] proposed methods for generating adversarial sentences that are both grammatically correctly and in agreement with the syntax of the original sentences.”
Towards the first adversarially robust neural network model on MNIST
Schott, L., Rauber, J., Bethge, M., & Brendel, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09190 · Cited by 225 · cited 1×
“Schott et al. also developed a high-accuracy, robust against adversarial attacks, image classification model that utilises the analysis by synthesis approach [143].”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples
Cheng, M., Yi, J., Chen, P.-Y., Zhang, H., & Hsieh, C.-J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5767 · Cited by 217
Fairness in Learning: Classic and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., & Roth, A. (2016) · arXiv · doi:10.48550/arxiv.1605.07139 · Cited by 199 · cited 1×
“Building on from [90], where the concept of meritocratic fairness was introduced, Kearns et al. [91] performed a more comprehensive analysis on the broader issue of realising superior guarantees in terms of performance, while relaxing the model assumptions.”
Le, H., & Viviani, J. (2017) · Research in International Business and Finance · doi:10.1016/j.ribaf.2017.07.104 · Cited by 199
RISE: Randomized Input Sampling for Explanation of Black-box Models
Petsiuk, V., Das, A., & Saenko, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07421 · Cited by 196 · cited 1×
“In order to interpret the predictions of deep neural networks for images, the RISE algorithm [39] creates a saliency map for any black-box model, indicating how important each pixel of the image with respect to the network’s prediction.”
Word-level Textual Adversarial Attacking as Combinatorial Optimization
Zang, Y., Qi, F., Yang, C., Liu, Z., Zhang, M., Liu, Q., & Sun, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.540 · Cited by 195
Li, Y., Li, L., Wang, L., Zhang, T., & Gong, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00441 · Cited by 191 · cited 1×
“NATTACK: instead of seeking an optimal adversarial example Li et al. [122] considered fitting a probability distribution in a neighbourhood centered around a given example, with the assumption being that any example generated from this distribution is a good adversary candidate.”
Learning Universal Adversarial Perturbations with Generative Models
Hayes, J., & Danezis, G. (2018) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00015 · Cited by 187 · cited 1×
“Hayes and Danezis [142] introduced a generative universal adversarial example framework, whereby image perturbations are produced by a generative model, such that, when incorporated into a normal, non-adversarial instance, they transform it to an adversarial instance.”
Grgić-Hlača, N., Zafar, M. B., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11296 · Cited by 181 · cited 2×
“[89] procedurally_fair_learning F G Agnostic tab 22 2018”
Adversarial Training Methods for Semi-Supervised Text Classification
Miyato, T., Dai, A. M., & Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07725 · Cited by 179 · cited 1×
“Miyato et al. [149] proposed applying perturbations to the word embeddings in a recurrent neural network instead of the original input.”
Boolean Decision Rules via Column GenerationPreprint
Dash, S., Günlük, O., & Wei, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09901 · Cited by 175 · cited 1×
“Boolean Rule Column Generation [68] is a technique that utilises Boolean rules, either in their disjunctive normal form (DNF) or in their conjunctive normal form (CNF), in order to create predictive models.”
Explanations of Model Predictions with live and breakDown Packages
Staniak, M., & Biecek, P. (2019) · The R Journal · doi:10.32614/rj-2018-072 · Cited by 172 · cited 2×
“The breakDown method, as proposed in [62], is similar to SHAP [48], as both, based on the conditioned responses of a black-box model, attempt to attribute them proportionally to the input features.”
Did the Model Understand the Question?
Mudrakarta, P. K., Taly, A., Sundararajan, M., & Dhamdhere, K. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1176 · Cited by 166 · cited 1×
“Another work, closely related to the integrated gradients method, was proposed in [30], where attributions are used in order to help identify weaknesses of three question-answer models better than the conventional models, while also to provide workflow superiority.”
Provably Minimally-Distorted Adversarial ExamplesPreprint
Carlini, N., Katz, G., Barrett, C., & Dill, D. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.10207 · Cited by 162
Runaway Feedback Loops in Predictive Policing
Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., & Venkatasubramanian, S. (2017) · arXiv · doi:10.48550/arxiv.1706.09847 · Cited by 161 · cited 2×
“[81] ML-fairness-gym F G Agnostic tab 45.”
Machine Learning in Healthcare
Callahan, A., & Shah, N. H. (2017) · Key Advances in Clinical Informatics · doi:10.1016/b978-0-12-809523-2.00019-4 · Cited by 160
TED: Teaching AI to Explain its Decisions
Hind, M., Wei, D., Campbell, M. S., Codella, N., Dhurandhar, A., Mojsilović, A., Ramamurthy, K. N., & Varshney, K. R. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314273 · Cited by 160 · cited 1×
“Hind et al. [71] introduced TED, a framework for producing local explanations that satisfy the complexity mental model of a domain.”
Efficient Data Representation by Selecting Prototypes with Importance Weights
Gurumoorthy, K. S., Dhurandhar, A., Cecchi, G., & Aggarwal, C. C. (2019) · Proceedings · doi:10.1109/icdm.2019.00036 · Cited by 157 · cited 1×
“Protodash: the work that was detailed in [55] regarding prototypes was extended in [56] by associating non-negative weightings to prototypes corresponding to their contribution, consequently creating a unifying coherent framework for both prototypes and criticisms/outliers.”
An effective algorithm for computing global sensitivity indices (EASI)
Plischke, E. (2009) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2009.11.005 · Cited by 143 · cited 1×
“The RBD-FAST method was further improved in terms of computational efficiency by Plischke [107], while Tissot et al. introduced a bias correction method in order to improve estimation accuracy [108].”
Explaining the Explainer: A First Theoretical Analysis of LIME
Garreau, D., & von Luxburg, U. (2020) · arXiv · doi:10.48550/arxiv.2001.03447 · Cited by 143 · cited 1×
“In 2020, the first theoretical analysis of LIME [46] was published, validating the significance and meaningfulness of LIME, but also proving that poor choices in terms of parameters could lead LIME to missing out on important features.”
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations
Tan, S., Joty, S., Kan, M., & Socher, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.263 · Cited by 138 · cited 1×
“In their work [164], Tan et al. proposed Morpheus, which is a method for generating textual adversarial examples by greedily perturbing the inflections of the original words in the text to find the inflected forms with the greatest loss increase, only taking into considerations the inflections that…”
Houdini: Fooling Deep Structured Visual and Speech Recognition Models with Adversarial Examples
Cissé, M., Adi, Y., Neverova, N., & Keshet, J. (2017) · neural information processing systems · Cited by 133
Is BERT Really Robust? Natural Language Attack on Text Classification and EntailmentPreprint
Jin, D., Jin, Z., Zhou, J., & Szolovits, P. (2019) · arXiv.org · doi:10.48550/arxiv.1907.11932 · Cited by 132 · cited 1×
“…which follows the synonyms substitution strategy, but replaces words that are based on the word saliency and classification probability TextFooler [161] generates adversarial examples for text by utilising word embedding distance and part-of-speech matching to first identify the most important…”
Explanations of Model Predictions with Live and breakDown Packages
Staniak, M., & Biecek, P. (2018) · Lincoln (University of Nebraska) · Cited by 128
Bias correction for the estimation of sensitivity indices based on random balance designs
Tissot, J.-Y., & Prieur, C. (2012) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2012.06.010 · Cited by 120 · cited 1×
“The RBD-FAST method was further improved in terms of computational efficiency by Plischke [107], while Tissot et al. introduced a bias correction method in order to improve estimation accuracy [108].”
Natural Language Adversarial Defense through Synonym EncodingPreprint
Wang, X., Jin, H., Yang, Y., & He, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06723 · Cited by 98 · cited 1×
“A similar idea was later also proposed by Kuleshov et al. [154], which uses word replacement by greedy heuristics, while later Wang et al. [155] improved upon the genetic algorithm, achieving not only higher success rates, but also lower word substitution rates and more transferable adversarial…”
Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete DataPreprint
Yang, P., Chen, J., Hsieh, C., Wang, J.-L., & Jordan, M. I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.12316 · Cited by 93 · cited 1×
“Yang et al. [146] focussed on generating adversarial examples for discrete data such as text.”
Machine Learning in Healthcare Informatics
Dua, S., Acharya, U. R., & Dua, P. (2013) · Intelligent systems reference library · doi:10.1007/978-3-642-40017-9 · Cited by 89
Accurate, reliable and fast robustness evaluationPreprint
Brendel, W., Rauber, J., Kümmerer, M., Ustyuzhaninov, I., & Bethge, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01003 · Cited by 87 · cited 1×
“Brendel et al. [136] also developed a novel family of gradient-based adversarial attacks that not only performed better than previous gradient-based attacks, but were more effective against gradient-masking, more efficient in terms of querying the defended model, and able to optimise for a variety…”
Adversarial Examples for Natural Language Classification Problems
Kuleshov, V., Thakoor, S., Lau, T., & Ermon, S. (2018) · Cited by 72
Fair Algorithms for Learning in Allocation Problems
Elzayn, H., Jabbari, S., Jung, C., Kearns, M., Neel, S., Roth, A., & Schutzman, Z. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287571 · Cited by 70
Zafar, M. R., & Khan, N. M. (2019) · arXiv · doi:10.48550/arxiv.1906.10263 · Cited by 69 · cited 2×
“[47] DLIME PH L Agnostic img txt tab 7.”
EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial ExamplesPreprint
Chen, P., Sharma, Y., Zhang, H., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04114 · Cited by 63 · cited 1×
“In [133], a novel approach that uses an elastic net-based regularisation framework (the combination of the L 1 and L 2 norms) to generate adversarial instances against deep neural networks was proposed.”
Zafar, M. R., & Khan, N. (2023) · doi:10.32920/22734359 · Cited by 51
Generating Adversarial Text Samples
Samanta, S., & Mehta, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-76941-7_71 · Cited by 46
Generalized Linear Rule ModelsPreprint
Wei, D., Dash, S., Gao, T., & Günlük, O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01761 · Cited by 38 · cited 1×
“Generalized Linear Rule Models [69], which are often referred to as rule ensembles, are Generalized Linear Models (GLMs) [70] that are linear combinations of rule-based features.”
Improving Simple Models with Confidence ProfilesPreprint
Dhurandhar, A., Shanmugam, K., Luss, R., & Olsen, P. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07506 · Cited by 34 · cited 1×
“[63] proposed transferring knowledge from highperforming pre-trained deep neural networks to a low performing, but interpretable non-complex model to improve its performance.”
Meritocratic Fairness for Infinite and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., Neel, S., & Roth, A. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278764 · Cited by 24 · cited 1×
“Building on from [90], where the concept of meritocratic fairness was introduced, Kearns et al. [91] performed a more comprehensive analysis on the broader issue of realising superior guarantees in terms of performance, while relaxing the model assumptions.”
Unique Sharp Local Minimum in L1-minimization Complete Dictionary Learning
Wang, Y., Wu, S., & Yu, B. (2020) · Journal of Machine Learning Research · Cited by 9
Fast Differentiable Clipping-Aware Normalization and RescalingPreprint
Rauber, J., & Bethge, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07677 · Cited by 5 · cited 2×
“The proposed clipping methods prior to [144] were relatively slow and they only provided approximations to the optimal solution, thus diminishing the effectiveness of the produced. adversarial examples.”
Research works printed but not linked: 3
- Luss, R., Chen, P.Y., Dhurandhar, A., Sattigeri, P., Zhang, Y., Shanmugam, K., and Tu, C.C. (2019). Generating contrastive explanations with monotonic attribute functions. arXiv.
- Van Looveren, A., and Klaise, J. (2019). Interpretable counterfactual explanations guided by prototypes. arXiv.
- Wang, X., Jin, H., and He, K. (2019). Natural language adversarial attacks and defenses in word level. arXiv.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century
Maleki Varnosfaderani, S., & Forouzanfar, M. (2024) · Bioengineering · doi:10.3390/bioengineering11040337 · Cited by 931
“Interpretability ensures that AI models provide clear explanations for their decisions, fostering trust with clinicians who can understand the reasoning behind recommendations [153].”
BackgroundThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“LLM-based agents have limitations because of hallucination [228], the lack of training data and tuning for specific fields [229], untransparent decision-making process [230] and susceptibility to adversarial attacks [231].”
Metadata
- Authors
- Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S.
- Affiliations
- University of Patras (GR)
- Corresponding author
- Linardatos, Pantelis
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2020-12-25
- Volume
- 23
- Issue
- 1
- Pages
- 18
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC7824368
- OpenAlex
- W3116286104
- Cited by
- 2,922
- References
- 162
- Citations by year
- 2026: 485 · 2025: 702 · 2024: 701 · 2023: 565 · 2022: 370 · 2021: 97 · 2020: 1
- FWCI
- 8.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, model interpretability, black-box models, medical applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.