General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Open accessThe DOI matches its Crossref record; 133 of the 133 references it cites were found in a registry or an index.
Abstract
An increasing number of model-agnostic interpretation techniques for machine learning (ML) models such as partial dependence plots (PDP), permutation feature importance (PFI) and Shapley values provide insightful model interpretations, but can lead to wrong conclusions if applied incorrectly. We highlight many general pitfalls of ML model interpretation, such as using interpretation techniques in the wrong context, interpreting models that do not generalize well, ignoring feature dependencies, interactions, uncertainty estimates and issues in high-dimensional settings, or making unjustified causal interpretations, and illustrate them with examples. We focus on pitfalls for global methods that describe the average model behavior, but many pitfalls also apply to local methods that explain individual predictions. Our paper addresses ML practitioners by raising awareness of pitfalls and identifying solutions for correct model interpretation, but also addresses ML researchers by discussing open issues for further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 2 Assuming one-fits-all interpretability . A default xgboost regression model that minimizes the mean squared error (MSE) was fitted on 20 independently and uniformly distributed features to predict another independent, uniformly sampled target. In this setting, predicting the (unconditional) mean E [ Y ] in a constant model is optimal. The learner overfits due to a small training data size. Mean marginal SHAP (red, error bars indicate 0 . 05 and 0 . 95 quantiles) exposes all mechanistically used features. In contrast, PFI on test data (blue, error bars indicate 0 . 05 and 0 . 95 quantiles) considers all features to be irrelevant, since no feature contributes to the generalization performance.](https://figures.cochat.ai/paper-figures/d89fa50113fe4204a130eac0040da274/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-2.png)


![Fig. 5 Confusing linear correlation with dependence . Highly dependent features X 1 and X 2 that have a correlation close to zero. A test ( H 0 : Features are independent) using Pearson correlation is not significant, but for HSIC, the H 0 -hypothesis gets rejected. Data from [80].](https://figures.cochat.ai/paper-figures/90068dfc976d4ead9a6c7cbb19095454/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-5.png)

![Fig. 7 Misleading effect due to interactions . Simulation example with interactions: Y = 3 X 1 -6 X 2 + 12 X 2 1 ( X 3 ≥ 0) + ϵ with X 1 , X 2 , X 3 i.i.d. ∼ U [ -1 , 1] and ϵ i.i.d. ∼ N (0 , 0 . 3). A random forest with 500 trees is fitted on 1000 observations. Effects are calculated on 200 randomly sampled (training) observations. A, B: PDP (yellow) and ICE curves of X 1 and X 2 ; C: Derivative ICE curves and their standard deviation of X 2 ; D: 2-dimensional PDP of X 2 and X 3 .](https://figures.cochat.ai/paper-figures/acbe517b1cfe4f9b8d1eb0df227240b1/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-7.png)
![Fig. 8 Ignoring model and approximation uncertainty . PDP for X 1 with Y = 0 · X 1 + ∑ 10 j =2 X j + ϵ i with X 1 , . . . , X 10 ∼ U [0 , 1] and ϵ i ∼ N (0 , 0 . 9). Left: PDP for X 1 of a random forest trained on 100 data points. Middle: Multiple PDPs (10x) for the model from left plots, but with different samples (each n=100) for PDP estimation. Right: Repeated (10x) data samples of n=100 and newly fitted random forest.](https://figures.cochat.ai/paper-figures/bf95f6cae70f49538e3b3bfd16542e68/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-8.png)


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, these methods can be misleading when correlations exist between input variables 107,109 .”
From Machine learning in concrete science: applications, challenges, and best practices (2022) · cites this paper as [109]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 1×
“Global methods include the partial dependence plot (PDP) [30], partial importance (PI) [16], accumulated local affects (ALE) [2], or the permutation feature importance (PFI) [10,28,16].”
MethodScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 2×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundRegression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · cited 1×
“An alternative is to directly use models that implicitly perform feature selection such as LASSO [95] or component-wise boosting [84] as they can produce sparse models with fewer features.”
MethodElements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 5×
“The variance of the estimates reduces in O ( 1 m ), where m is the number of evaluated orderings [21,67].”
BackgroundMethodInfluentialA Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“Since MCP is well known in statistics, we refer the practitioner to [24] for an overview and discussion of alternative adjustment methods, such as the Bonferroni-Holm method [44].”
Method"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2005) · doi:10.1002/047174882x · Cited by 13,791
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 3×
“However, further research is required, as there is no single perfect definition of interpretability, but rather multiple depending on the context [25,82].”
BackgroundMethodFrom local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · cited 1×
“In such scenarios, one should quantify the relevance of a feature regarding the model’s prediction (and not the model’s generalization error) using methods like the SHAP importance [65].”
MethodWrappers for feature subset selection
Kohavi, R., & John, G. H. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(97)00043-x · Cited by 8,947 · cited 1×
“Wrapper selection approaches are model-agnostic and algorithms like greedy forward selection or subset selection procedures [52,4], which start from an empty model and iteratively add relevant (subsets of) features if needed, even allow to measure the relevance of features for predictive…”
MethodScikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 2×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundMultivariate Adaptive Regression Splines
Friedman, J. H. (1991) · The Annals of Statistics · doi:10.1214/aos/1176347963 · Cited by 8,000
Model Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261 · cited 1×
“Before a model is fitted, groupings could already be exploited for dimensionality reduction, for example by selecting groups of features by the group LASSO [104].”
MethodWhat's wrong with Bonferroni adjustments
Perneger, T. V. (1998) · BMJ · doi:10.1136/bmj.316.7139.1236 · Cited by 5,900
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Likewise, Lipton [62] criticizes interpretable ML for its lack of causal conclusions, trust, and insights, but the author does not discuss any pitfalls in detail.”
BackgroundMultiple comparisons among means
Dunn, O. J. (1961) · Journal of the American Statistical Association · doi:10.1080/01621459.1961.10482090 · Cited by 5,006 · cited 1×
“One of the most popular MCP adjustment methods is the Bonferroni correction [26], which rejects a null hypothesis if its p-value is smaller than α/p , with p as the number of tests.”
MethodStatistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
Regression Shrinkage and Selection via The Lasso: A Retrospective
Tibshirani, R. (2011) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2011.00771.x · Cited by 3,589
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · cited 1×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundHastie, T., & Tibshirani, R. (1986) · Statistical Science · doi:10.1214/ss/1177013604 · Cited by 3,409
Detecting Novel Associations in Large Data Sets
Reshef, D. N., Reshef, Y. A., Finucane, H. K., Grossman, S. R., McVean, G., Turnbaugh, P. J., Lander, E. S., Mitzenmacher, M., & Sabeti, P. C. (2011) · Science · doi:10.1126/science.1205438 · Cited by 3,339 · cited 1×
“In addition, there are information-theoretical measures, such as (conditional) mutual information [20] or the maximal information coefficient (MIC) [80], that can however be difficult Feature X 1 F ea t u r e X 2 Fig.”
BackgroundConditional variable importance for random forests
Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., & Zeileis, A. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-307 · Cited by 3,254
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 1×
“However, further research is required, as there is no single perfect definition of interpretability, but rather multiple depending on the context [25,82].”
BackgroundUnderstanding Machine Learning: From Theory to Algorithms
Shalev-Shwartz, S., & Ben-David, S. (2014) · doi:10.1017/cbo9781107298019 · Cited by 3,115
(2015) · Cited by 2,857
Permutation importance: a corrected feature importance measure
Altmann, A., Toloşi, L., Sander, O., & Lengauer, T. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq134 · Cited by 2,843 · cited 2×
“For all other methods that provide p-values or confidence intervals, such as PIMP (Permutation IMPortance) [1], which is a testing approach for PFI, MCP is often ignored in practice to the best of our knowledge, with some exceptions[89,100].”
MethodMeasuring and testing dependence by correlation of distances
Székely, G. J., Rizzo, M. L., & Bakirov, N. K. (2007) · The Annals of Statistics · doi:10.1214/009053607000000505 · Cited by 2,684 · cited 1×
“Other important measures are e.g. the distance correlation [94], the randomized dependence coefficient (RDC) [63], or the alternating conditional expectations (ACE) algorithm [11].”
BackgroundExplaining prediction models and individual predictions with feature contributions
Štrumbelj, E., & Kononenko, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0679-x · Cited by 2,286 · cited 1×
“Local methods include the individual conditional expectation (ICE) curves [31], individual conditional importance (ICI) [16], local interpretable model-agnostic explanations (LIME) [81], Shapley values [92] and SHapley Additive exPlanations (SHAP) [67,66] or counterfactual explanations [98,22].”
MethodOn the Surprising Behavior of Distance Metrics in High Dimensional Space
Aggarwal, C. C., Hinneburg, A., & Keim, D. A. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44503-x_27 · Cited by 2,083
Elements of Causal Inference: Foundations and Learning Algorithms
Peters, J., Janzing, D., & Schölkopf, B. (2017) · OAPEN (The OAPEN Foundation) · Cited by 2,010
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861 · cited 3×
“Particularly for continuous interactions where ICE curves start at different intercepts, we recommend the use of derivative or centered ICE curves, which eliminate differ-ences in intercepts and leave only differences due to interactions [31].”
MethodGraph-based approach for airborne light detection and ranging segmentation
Vilariño, D. L., Cabaleiro, J. C., Martínez, J., Rivera, F. F., & Pena, T. F. (2017) · Journal of Applied Remote Sensing · doi:10.1117/1.jrs.11.015020 · Cited by 1,721
Estimating Optimal Transformations for Multiple Regression and Correlation
Breiman, L., & Friedman, J. H. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10478157 · Cited by 1,710 · cited 1×
“Other important measures are e.g. the distance correlation [94], the randomized dependence coefficient (RDC) [63], or the alternating conditional expectations (ACE) algorithm [11].”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 1×
“Interactions between features can be added manually or algorithmically (e.g. via a forward greedy search) [15].”
BackgroundMeasuring Statistical Dependence with Hilbert-Schmidt Norms
Gretton, A., Bousquet, O., Smola, A., & Schölkopf, B. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564089_7 · Cited by 1,560 · cited 1×
“Kernel-based measures, such as kernel canonical correlation analysis (KCCA) [5] or the Hilbert-Schmidt independence criterion (HSIC) [37], are commonly used.”
MethodVisualizing the effects of predictor variables in black box supervised learning models
Apley, D. W., & Zhu, J. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12377 · Cited by 1,535 · cited 6×
“…partial dependence plots [30], inferring behavior from smartphone usage [90,89] with the help of permutation feature importance [91] and accumulated local effect plots [2], or understanding the relation between critical illness and health records [59] using Shapley additive explanations (SHAP) [67].”
BackgroundMethodInfluentialStatistical and machine learning forecasting methods: Concerns and ways forward
Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018) · PLOS ONE · doi:10.1371/journal.pone.0194889 · Cited by 1,528 · cited 1×
“…UCI benchmark data repository and concluded that such models often should be preferred due to their inherent interpretability; Makridakis et al. [68] systematically compared various ML models (including long-short-term-memory models and multi-layer neural networks) to statistical models (e.g.…”
BackgroundConsistent Individualized Feature Attribution for Tree Ensembles
Lundberg, S. M., Erion, G. G., & Lee, S.-I. (2018) · arXiv · doi:10.48550/arxiv.1802.03888 · Cited by 1,430 · cited 2×
“Based on Shapley values, Lundberg et al. [66] proposed SHAP interaction values, and Casalicchio et al. [16] proposed a fair attribution of the importance of interactions to the individual features.”
MethodPredictive learning via rule ensembles
Friedman, J. H., & Popescu, B. E. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas148 · Cited by 1,315 · cited 2×
“Interesting 2-way interactions can be selected by another method such as the H-statistic [29].”
MethodBenchmarking state-of-the-art classification algorithms for credit scoring: an update of research
Lessmann, S., Baesens, B., Seow, H.-V., & Thomas, L. C. (2015) · European Journal of Operational Research · doi:10.1016/j.ejor.2015.05.030 · Cited by 1,284 · cited 1×
“…model performs as well as AdaBoost and even better than an SVM in predicting heart disease from electronic health record data; Baesens et al. [6] showed that simple interpretable classifiers perform competitively for credit scoring, and in an update to the study the authors note that “the…”
MethodInterpretable Machine Learning: Fundamental Principles and 10 Grand ChallengesPreprint
Rudin, C., Chen, C.-F., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2021) · Statistics Survey · doi:10.48550/arxiv.2103.11251 · Cited by 1,254 · cited 1×
“Related Work: Rudin et al. [83] present principles for interpretability and discuss challenges for model interpretation with a focus on inherently interpretable models.”
BackgroundExplaining machine learning classifiers through diverse counterfactual explanations
Mothilal, R. K., Sharma, A., & Tan, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372850 · Cited by 1,233
Saliency, Scale and Image Description
Kadir, T., & Brady, M. (2001) · International Journal of Computer Vision · doi:10.1023/a:1012460413855 · Cited by 1,195 · cited 1×
“Model-specific methods, in contrast are tied to a certain model class (e.g. saliency maps [49] for gradient-based models, such as neural networks), and are mainly considered out-of-scope for this work.”
Methodpdp: An R Package for Constructing Partial Dependence Plots
Greenwell, B. (2017) · The R Journal · doi:10.32614/rj-2017-016 · Cited by 1,189 · cited 1×
“By default, many implementations of interpretability methods use an equidistant grid to perturb feature values [34,70,76], although some also allow using user-defined values.”
BackgroundFisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165 · cited 1×
“Global methods include the partial dependence plot (PDP) [30], partial importance (PI) [16], accumulated local affects (ALE) [2], or the permutation feature importance (PFI) [10,28,16].”
MethodCounterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Self-citationMutual Information Neural Estimation.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A. C., & Hjelm, D. (2018) · International Conference on Machine Learning · Cited by 920
Benchmarking state-of-the-art classification algorithms for credit scoring
Baesens, B., Van Gestel, T., Viaene, S., Stepanova, M., Suykens, J., & Vanthienen, J. (2003) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601545 · Cited by 904
Model selection and model averaging
Claeskens, G., & Hjort, N. L. (2008) · Cited by 872
Classifier Technology and the Illusion of Progress
Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000060 · Cited by 723 · cited 1×
“However, there are several examples where interpretable models have proven to be serious competitors: More than 15 years ago, Hand [41] demonstrated that simple models often achieve more than 90% of the predictive power of potentially highly complex models across the UCI benchmark data repository…”
Backgroundiml: An R package for Interpretable Machine Learning
Molnar, C. (2018) · Journal of Open Source Software · doi:10.21105/joss.00786 · Cited by 718 · cited 1×
“By default, many implementations of interpretability methods use an equidistant grid to perturb feature values [34,70,76], although some also allow using user-defined values.”
BackgroundSelf-citationPanning for Gold: ‘Model-X’ Knockoffs for High Dimensional Controlled Variable Selection
Candès, E., Fan, Y., Janson, L., & Lv, J. (2018) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12265 · Cited by 632
Model Selection and Model Averaging
Claeskens, G., & Hjort, N. L. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511790485 · Cited by 629
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504 · cited 1×
“Das et al. [23] survey methods for explainable AI and discuss challenges with a focus on saliency maps for neural networks.”
BackgroundLiebetrau, A. (1983) · Sage eBooks · doi:10.4135/9781412984942 · Cited by 479
Causal Interpretations of Black-Box Models
Zhao, Q., & Hastie, T. (2021) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2019.1624293 · Cited by 462 · cited 1×
“The PDP between a feature and the target can be interpreted as the respective average causal effect if the model performs well and the set of remaining variables is a valid adjustment set [106].”
Backgroundmlr3: A modern object-oriented machine learning framework in R
Lang, M., Binder, M., Richter, J., Schratz, P., Pfisterer, F., Coors, S., Au, Q., Casalicchio, G., Kotthoff, L., & Bischl, B. (2019) · Journal of Open Source Software · doi:10.21105/joss.01903 · Cited by 449 · cited 1×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundSelf-citationLauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448 · cited 1×
“Applications of interpretable machine learning (IML) include understanding pre-evacuation decision-making [107] with partial dependence plots [30], inferring behavior from smartphone usage [90,89] with the help of permutation feature importance [91] and accumulated local effect plots [2], or understanding the relation…”
MethodWu, J., Roy, J., & Stewart, W. F. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181de9e17 · Cited by 373 · cited 1×
“…gradient boosting and neural networks did not outperform logistic regression in predicting fetal growth abnormalities; Similarly, Wu et al. [103] have shown that a logistic regression model performs as well as AdaBoost and even better than an SVM in predicting heart disease from electronic…”
MethodA Simple and Effective Model-Based Variable Importance MeasurePreprint
Greenwell, B., Boehmke, B. C., & McCarthy, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04755 · Cited by 357 · cited 1×
“Another similar interaction score based on partial dependencies is defined by [35].”
BackgroundPredicting personality from patterns of behavior collected with smartphones
Stachl, C., Au, Q., Schoedel, R., Gosling, S. D., Harari, G. M., Buschek, D., Völkel, S. T., Schuwerk, T., Oldemeier, M., Ullmann, T., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1920484117 · Cited by 348 · cited 3×
“Interpreting the results in this use case seems to be challenging since features were dependent and single feature effects were either small or non-linear [90].”
BackgroundMethodSelf-citationVisualizing the Feature Importance for Black Box Models
Casalicchio, G., Molnar, C., & Bischl, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-10925-7_40 · Cited by 325
Self-citationStable feature selection for biomarker discovery
He, Z., & Yu, W. (2010) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2010.07.002 · Cited by 309
Measures of Association: How to Choose?
Khamis, H. (2008) · Journal of Diagnostic Medical Sonography · doi:10.1177/8756479308317006 · Cited by 284 · cited 1×
“For categorical or mixed features, separate dependence measures have been proposed, such as Kendall’s rank correlation coefficient for ordinal features, or the phi coefficient and Goodman & Kruskal’s lambda for nominal features [51].”
MethodAlgorithmic recourse: From counterfactual explanations to interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445899 · Cited by 274 · cited 1×
“Without assumptions about the underlying causal structure, IML methods cannot account for these adaptions and guide action [50].”
BackgroundTrue to the Model or True to the Data?Preprint
Chen, H., Janizek, J. D., Lundberg, S., & Lee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16234 · Cited by 273 · cited 2×
“Therefore, these methods are said to be true to the model but not true to the data [18].”
BackgroundMethodMulti-Objective Counterfactual Explanations
Dandl, S., Molnar, C., Binder, M., & Bischl, B. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58112-1_31 · Cited by 271
Self-citationHooker, G., Mentch, L., & Zhou, S. (2021) · Statistics and Computing · doi:10.1007/s11222-021-10057-z · Cited by 264
Generalized Functional ANOVA Diagnostics for High-Dimensional Functions of Dependent Variables
Hooker, G. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x237892 · Cited by 220 · cited 3×
“Furthermore, Hooker [46] considers dependent features and decomposes the predictions in main and interaction effects.”
BackgroundMethodAgainst Interpretability: a Critical Examination of the Interpretability Problem in Machine Learning
Krishnan, M. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00372-9 · Cited by 218 · cited 1×
“Krishnan [55] criticizes the general conceptual foundation of interpretability, but does not dispute the usefulness of available methods.”
BackgroundFeature relevance quantification in explainable AI: A causal problem
Janzing, D., Minorics, L., & Blöbaum, P. (2019) · arXiv · doi:10.48550/arxiv.1910.13413 · Cited by 207 · cited 4×
“All the aforementioned methods extrapolate under dependent features (see also Section 5.1), but satisfy sensitivity, i.e. are zero if a feature is not used by the model [48,21,53].”
BackgroundInfluentialGoldberg, L. R. (2019) · Quantitative Finance · doi:10.1080/14697688.2019.1655928 · Cited by 206
Common Errors in Statistics (and How to Avoid Them)
Good, P. I., & Hardin, J. W. (2009) · Wiley eBooks · doi:10.1002/9780470473924 · Cited by 196
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations
Laugel, T., Lesot, M.-J., Marsala, C., Renard, X., & Detyniecki, M. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/388 · Cited by 194
Understanding Global Feature Contributions With Additive Importance Measures
Covert, I., Lundberg, S., & Lee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00668 · Cited by 191
Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation
Bischl, B., Mersmann, O., Trautmann, H., & Weihs, C. (2012) · Evolutionary Computation · doi:10.1162/evco_a_00069 · Cited by 186 · cited 1×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundSelf-citationSimultaneous Statistical Inference: With Applications in the Life Sciences
Dickhaus, T. (2014) · Springer eBooks · doi:10.1007/978-3-642-45182-9 · Cited by 184 · cited 1×
“Since MCP is well known in statistics, we refer the practitioner to [24] for an overview and discussion of alternative adjustment methods, such as the Bonferroni-Holm method [44].”
MethodMatejka, J., & Fitzmaurice, G. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025912 · Cited by 173 · cited 2×
“Data from [69]. to estimate [99,7].”
BackgroundKuhle, S., Maguire, B., Zhang, H., Hamilton, D., Allen, A. C., Joseph, K. S., & Allen, V. M. (2018) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-018-1971-2 · Cited by 166 · cited 1×
“…and the Theta method) in time series forecasting tasks and found that the latter consistently show greater predictive accuracy; Kuhle et al. [56] found that random forests, gradient boosting and neural networks did not outperform logistic regression in predicting fetal growth abnormalities;…”
MethodGrouped graphical Granger modeling for gene expression regulatory networks discovery
Lozano, A. C., Abe, N., Liu, Y., & Rosset, S. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp199 · Cited by 159 · cited 1×
“Examples where features can naturally be grouped include the grouping of sensor data [17], time-lagged features [64], or one-hot-encoded categorical features and interaction terms [36].”
BackgroundModelling and interpreting pre-evacuation decision-making using machine learning
Zhao, X., Lovreglio, R., & Nilsson, D. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103140 · Cited by 156 · cited 1×
“Applications of interpretable machine learning (IML) include understanding pre-evacuation decision-making [107] with partial dependence plots [30], inferring behavior from smartphone usage [90,89] with the help of permutation feature importance [91] and accumulated local effect plots [2], or…”
MethodEstimation of Mutual Information: A Survey
Walters-Williams, J., & Li, Y. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02962-2_49 · Cited by 145 · cited 1×
“Data from [69]. to estimate [99,7].”
Grouped variable importance with random forests and application to multiple functional data analysis
Gregorutti, B., Michel, B., & Saint-Pierre, P. (2015) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2015.04.002 · Cited by 145 · cited 2×
“For model interpretation, various papers extended feature importance methods from single features to groups of features [4,36,97,102].”
BackgroundBoosting additive models using component-wise P-Splines
Schmid, M. C., & Hothorn, T. (2008) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.09.009 · Cited by 144 · cited 2×
“An alternative is to directly use models that implicitly perform feature selection such as LASSO [95] or component-wise boosting [84] as they can produce sparse models with fewer features.”
BackgroundMethodI Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144
vip: Variable Importance Plots
Greenwell, B. M., & Boehmke, B. (2018) · CRAN: Contributed Packages · doi:10.32614/cran.package.vip · Cited by 133
Discovering additive structure in black box functions
Hooker, G. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014122 · Cited by 124 · cited 3×
“Interaction strength usually decreases quickly with increasing interaction size, and one should only consider d -way interactions when all their ( d − 1)-way interactions were significant [45].”
BackgroundMethodInfluentialThe Randomized Dependence CoefficientPreprint
López-Paz, D., Hennig, P., & Schölkopf, B. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.7717 · Cited by 123 · cited 2×
“the distance correlation [94], the randomized dependence coefficient (RDC) [63], or the alternating conditional expectations (ACE) algorithm [11].”
BackgroundMolnar, C., König, G., Bischl, B., & Casalicchio, G. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00901-9 · Cited by 120
Self-citationGrouped feature importance and combined features effect plot
Au, Q., Herbinger, J., Stachl, C., Bischl, B., & Casalicchio, G. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00840-5 · Cited by 116 · cited 4×
“For model interpretation, various papers extended feature importance methods from single features to groups of features [4,36,97,102].”
BackgroundMethodInfluentialSelf-citationExploring the cloud of variable importance for the set of all good models
Dong, J., & Rudin, C. D. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00264-0 · Cited by 113
Please Stop Permuting Features: An Explanation and Alternatives.Preprint
Hooker, G., & Mentch, L. (2019) · arXiv (Cornell University) · Cited by 102
Common Errors in Statistics (And How to Avoid Them)
Good, P. I., & Hardin, J. W. (2012) · Wiley eBooks · doi:10.1002/9781118360125 · Cited by 96
Causal interpretation rules for encoding and decoding models in neuroimaging
Weichwald, S., Meyer, T., Özdenizci, O., Schölkopf, B., Ball, T., & Grosse-Wentrup, M. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.01.036 · Cited by 95 · cited 2×
“…known whether a model is deployed in a causal or anti-causal setting – i.e. whether the model attempts to predict an effect from its causes or the other way round – a partial identification of the causal roles based on feature relevance is possible (under strong and non-testable assumptions) [101].”
BackgroundMethodSelf-citationKönig, G., Molnar, C., Bischl, B., & Grosse‐Wentrup, M. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413090 · Cited by 82 · cited 2×
“Recent work formalizes the implications of marginal and conditional interpretation techniques [18,48,54,53,21].”
BackgroundSelf-citationRobustness in machine learning explanations: does it matter?
Hancox-Li, L. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372836 · Cited by 81
Statistical Dependence: Beyond Pearson’s ρ
Tjøstheim, D., Otneim, H., & Støve, B. (2022) · Statistical Science · doi:10.1214/21-sts823 · Cited by 78
Algorithmic Recourse: from Counterfactual Explanations to Interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2020) · arXiv · doi:10.48550/arxiv.2002.06278 · Cited by 76 · cited 1×
“Without assumptions about the underlying causal structure, IML methods cannot account for these adaptions and guide action [50].”
BackgroundSelecting Useful Groups of Features in a Connectionist Framework
Chakraborty, D., & Pal, N. R. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.910730 · Cited by 72 · cited 1×
“Examples where features can naturally be grouped include the grouping of sensor data [17], time-lagged features [64], or one-hot-encoded categorical features and interaction terms [36].”
BackgroundQuantifying Model Complexity via Functional Decomposition for Better Post-hoc Interpretability
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-43823-4_17 · Cited by 64 · cited 1×
“Some steps have been made towards quantifying model complexity, such as using functional decomposition and quantifying the complexity of the components [71] or measuring the stability of predictions [79].”
BackgroundSelf-citationLooking Deeper into Tabular LIMEPreprint
Garreau, D., & Ulrike Von Luxburg, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.11092 · Cited by 60
Measuring the stability of results from supervised statistical learning
Philipp, M., Rusch, T., Hornik, K., & Strobl, C. (2018) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2018.1473779 · Cited by 56 · cited 1×
“Some steps have been made towards quantifying model complexity, such as using functional decomposition and quantifying the complexity of the components [71] or measuring the stability of predictions [79].”
BackgroundFeature Interaction in Terms of Prediction Performance
Oh, S. (2019) · Applied Sciences · doi:10.3390/app9235191 · Cited by 55 · cited 1×
“Instead of decomposing the partial dependence function, [74] uses the predictive performance to measure interaction strength.”
MethodScholbeck, C. A., Molnar, C., Heumann, C., Bischl, B., & Casalicchio, G. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_18 · Cited by 53 · cited 2×
“Since many of the interpretation methods work by similar principles of manipulating data and “probing” the model [85], they also share many pitfalls.”
BackgroundSelf-citationBehavioral Patterns in Smartphone Usage Predict Big Five Personality TraitsPreprint
Stachl, C., Au, Q., Schoedel, R., et al. (2019) · doi:10.31234/osf.io/ks4vd · Cited by 49 · cited 2×
“For all other methods that provide p-values or confidence intervals, such as PIMP (Permutation IMPortance) [1], which is a testing approach for PFI, MCP is often ignored in practice to the best of our knowledge, with some exceptions[89,100].”
MethodResampling Strategies for Model Assessment and Selection
Simon, R. M. (2007) · Springer eBooks · doi:10.1007/978-0-387-47509-7_8 · Cited by 48 · cited 1×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundHyperparameter Optimization: Foundations, Algorithms, Best Practices and Open ChallengesPreprint
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05847 · Cited by 46
Self-citationSAM: The Sensitivity of Attribution Methods to Hyperparameters
Bansal, N., Agarwal, C., & Nguyen, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00870 · Cited by 44
Defining Locality for Surrogates in Post-hoc Interpretablity
Laugel, T., Renard, X., Lesot, M., Marsala, C., & Detyniecki, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07498 · Cited by 44
The many Shapley values for model explanation
Sundararajan, M., & Najmi, A. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1908.08474 · Cited by 35 · cited 1×
“Similarly, conditional SAGE and conditional SHAP value functions sample the remaining features conditional on the feature of interest and therefore violate sensitivity [48,93,21,53].”
BackgroundDo Not Trust Additive Explanations
Gosiewska, A., & Biecek, P. (2019) · arXiv · doi:10.48550/arxiv.1903.11420 · Cited by 34
VINE: Visualizing Statistical Interactions in Black Box ModelsPreprint
Britton, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00561 · Cited by 33 · cited 1×
“A, B: PDP (yellow) and ICE curves of X 1 and X 2 ; C: Derivative ICE curves and their standard deviation of X 2 ; D: 2-dimensional PDP of X 2 and X 3 . visualizations are based on clustering homogeneous ICE curves, such as visual interaction effects (VINE) [13] or [105].”
MethodEffect of Superpixel Aggregation on Explanations in LIME – A Case Study with Biological Data
Schallner, L., Rabold, J., Scholz, O., & Schmid, U. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_13 · Cited by 31
On the Art and Science of Machine Learning Explanations
Hall, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02909 · Cited by 30 · cited 1×
“Hall [40] discusses recommendations for the application of particular interpretation methods but does not address general pitfalls.”
BackgroundBrenning, A. (2023) · Machine Learning · doi:10.1007/s10994-023-06327-8 · Cited by 26
A Causal Perspective on Meaningful and Robust Algorithmic RecoursePreprint
König, G., Freiesleben, T., & Grosse‐Wentrup, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07853 · Cited by 26
Self-citationImproving LIME Robustness with Smarter Locality SamplingPreprint
Saito, S., Chua, E., Capel, N., & Hu, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12302 · Cited by 25
A unified approach for inference on algorithm-agnostic variable importance
Williamson, B. D., Gilbert, P. B., Simon, N., & Carone, M. (2020) · Cited by 22
Interpreting neural decoding models using grouped model reliance
Valentin, S., Harkotte, M., & Popov, T. G. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007148 · Cited by 15 · cited 1×
“For model interpretation, various papers extended feature importance methods from single features to groups of features [4,36,97,102].”
BackgroundTesting conditional independence in supervised learning algorithms
Watson, D. S., & Wright, M. N. (2021) · Machine Learning · doi:10.1007/s10994-021-06030-6 · Cited by 13 · cited 3×
“For all other methods that provide p-values or confidence intervals, such as PIMP (Permutation IMPortance) [1], which is a testing approach for PFI, MCP is often ignored in practice to the best of our knowledge, with some exceptions[89,100].”
BackgroundMethodtotalvis: A Principal Components Approach to Visualizing Total Effects in Black Box Models
Seedorff, N., & Brown, G. (2021) · SN Computer Science · doi:10.1007/s42979-021-00560-5 · Cited by 10 · cited 1×
“Only recently, [4,12,86] attempted to answer this question by using dimension-reduction techniques (such as PCA) before applying the interpretation method.”
MethodZhang, X., Wang, Y., & Li, Z. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02255-z · Cited by 8 · cited 1×
“A, B: PDP (yellow) and ICE curves of X 1 and X 2 ; C: Derivative ICE curves and their standard deviation of X 2 ; D: 2-dimensional PDP of X 2 and X 3 . visualizations are based on clustering homogeneous ICE curves, such as visual interaction effects (VINE) [13] or [105].”
MethodStatistical dependence: Beyond Pearson's $\rho$Preprint
Tjøstheim, D., Otneim, H., & Støve, B. (2018) · arXiv (Cornell University) · Cited by 7
ICIE 1.0: A Novel Tool for Interactive Contextual Interaction Explanations
van der Zon, S. B., Duivesteijn, W., van Ipenburg, W., Veldsink, J., & Pechenizkiy, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-13463-1_6 · Cited by 5 · cited 1×
“Work by Zon et al. [108] followed a similar idea by proposing an interactive visualization tool to group Shapley values with regards to interacting features that need to be defined by the user.”
MethodDecomposition of Global Feature Importance into Direct and Associative Components (DEDACT)Preprint
König, G., Freiesleben, T., Bischl, B., Casalicchio, G., & Grosse‐Wentrup, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08086 · Cited by 4 · cited 5×
“Similarly, conditional SAGE and conditional SHAP value functions sample the remaining features conditional on the feature of interest and therefore violate sensitivity [48,93,21,53].”
BackgroundMethodInfluentialSelf-citationStatistical dependence: Beyond Pearson's $ρ$Preprint
Tjøstheim, D., Otneim, H., & Støve, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10455 · Cited by 4
Towards Explaining Hyperparameter Optimization via Partial Dependence Plots
Moosbauer, J., Herbinger, J., Casalicchio, G., Lindauer, M., & Bischl, B. (2021) · International Conference on Machine Learning · Cited by 3
Self-citation(2014) · doi:10.5555/2627435.2697065 · cited 1×
“The major driving force behind this development has been a considerable outperformance of ML over traditional models on many prediction tasks [27].”
Background
Research works printed but not linked: 7
- Brenning, A.: Transforming feature space to interpret machine learning models. arXiv:2104.04295 (2021) · arXiv
- Freiesleben, T.: Counterfactual explanations & adversarial examples-common grounds, essential differences, and potential transfers. arXiv preprint arXiv:2009.05487 (2020) · arXiv
- Curran Associates, Inc. (2017). https://doi.org/ 10.5555/3295222.3295230 · doi:10.5555/3295222.3295230
- Molnar, C., Freiesleben, T., König, G., Casalicchio, G., Wright, M.N., Bischl, B.: Relating the partial dependence plot and permutation feature importance to the data generating process. arXiv preprint arXiv:2109.01433 (2021) · arXiv
- Semenova, L., Rudin, C., Parr, R.: A study in Rashomon curves and volumes: a new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755 (2021) 103. · arXiv
- Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning, pp. 3319-3328. PMLR (2017) 111.
- Alzate, C., et al. (eds.) MIDAS/PAP -2018. LNCS (LNAI), vol. 11054, pp. 81-94. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-13463-1 6 · DOI
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME
Salih, A. M., Raisi‐Estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400304 · Cited by 897
“Despite the limitations of SHAP and LIME in terms of uncertainty estimates, generalization, non-linear dependencies (with LIME), feature dependencies, and inability to infer causality [13], they hold substantial value for explaining and interpreting complex machine learning models.”
BackgroundInterpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
Self-citationExplainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_2 · Cited by 379
Self-citationMachine learning in concrete science: applications, challenges, and best practices
Li, Z., Yoon, J., Zhang, R., Rajabipour, F., Srubar III, W. V., Dabo, I., & Radlińska, A. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00810-x · Cited by 388
Metadata
- Authors
- Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B.
- Affiliations
- Leibniz Institute for Prevention Research and Epidemiology - BIPS (DE)
- University of Vienna (AT)
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Molnar, Christoph
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2022
- Pages
- 39-68
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3194357095
- Cited by
- 211
- References
- 130
- Citations by year
- 2026: 41 · 2025: 56 · 2024: 48 · 2023: 51 · 2022: 11
- FWCI
- 22.95
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 106 d)
- Locations
- Lecture notes in computer science (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Deutsche Forschungsgemeinschaft (437611051)
- Bundesministerium für Bildung und Forschung (01IS18036A)
- Graduate School of Systemic Neurosciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- model-agnostic interpretability, explainable machine learning, partial dependence plots, permutation importance, Shapley value, feature interaction, uncertainty estimation, high-dimensional data, causal interpretation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.