The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Open accessAbstract
An increasing number of model-agnostic interpretation techniques for machine learning (ML) models such as partial dependence plots (PDP), permutation feature importance (PFI) and Shapley values provide insightful model interpretations, but can lead to wrong conclusions if applied incorrectly. We highlight many general pitfalls of ML model interpretation, such as using interpretation techniques in the wrong context, interpreting models that do not generalize well, ignoring feature dependencies, interactions, uncertainty estimates and issues in high-dimensional settings, or making unjustified causal interpretations, and illustrate them with examples. We focus on pitfalls for global methods that describe the average model behavior, but many pitfalls also apply to local methods that explain individual predictions. Our paper addresses ML practitioners by raising awareness of pitfalls and identifying solutions for correct model interpretation, but also addresses ML researchers by discussing open issues for further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 2 Assuming one-fits-all interpretability . A default xgboost regression model that minimizes the mean squared error (MSE) was fitted on 20 independently and uniformly distributed features to predict another independent, uniformly sampled target. In this setting, predicting the (unconditional) mean E [ Y ] in a constant model is optimal. The learner overfits due to a small training data size. Mean marginal SHAP (red, error bars indicate 0 . 05 and 0 . 95 quantiles) exposes all mechanistically used features. In contrast, PFI on test data (blue, error bars indicate 0 . 05 and 0 . 95 quantiles) considers all features to be irrelevant, since no feature contributes to the generalization performance.](https://figures.cochat.ai/paper-figures/d89fa50113fe4204a130eac0040da274/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-2.png)


![Fig. 5 Confusing linear correlation with dependence . Highly dependent features X 1 and X 2 that have a correlation close to zero. A test ( H 0 : Features are independent) using Pearson correlation is not significant, but for HSIC, the H 0 -hypothesis gets rejected. Data from [80].](https://figures.cochat.ai/paper-figures/90068dfc976d4ead9a6c7cbb19095454/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-5.png)

![Fig. 7 Misleading effect due to interactions . Simulation example with interactions: Y = 3 X 1 -6 X 2 + 12 X 2 1 ( X 3 ≥ 0) + ϵ with X 1 , X 2 , X 3 i.i.d. ∼ U [ -1 , 1] and ϵ i.i.d. ∼ N (0 , 0 . 3). A random forest with 500 trees is fitted on 1000 observations. Effects are calculated on 200 randomly sampled (training) observations. A, B: PDP (yellow) and ICE curves of X 1 and X 2 ; C: Derivative ICE curves and their standard deviation of X 2 ; D: 2-dimensional PDP of X 2 and X 3 .](https://figures.cochat.ai/paper-figures/acbe517b1cfe4f9b8d1eb0df227240b1/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-7.png)
![Fig. 8 Ignoring model and approximation uncertainty . PDP for X 1 with Y = 0 · X 1 + ∑ 10 j =2 X j + ϵ i with X 1 , . . . , X 10 ∼ U [0 , 1] and ϵ i ∼ N (0 , 0 . 9). Left: PDP for X 1 of a random forest trained on 100 data points. Middle: Multiple PDPs (10x) for the model from left plots, but with different samples (each n=100) for PDP estimation. Right: Repeated (10x) data samples of n=100 and newly fitted random forest.](https://figures.cochat.ai/paper-figures/bf95f6cae70f49538e3b3bfd16542e68/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-8.png)


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, these methods can be misleading when correlations exist between input variables 107,109 .”
From Machine learning in concrete science: applications, challenges, and best practices (2022) · cites this paper as [109]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 1×
“Global methods include the partial dependence plot (PDP) [30], partial importance (PI) [16], accumulated local affects (ALE) [2], or the permutation feature importance (PFI) [10,28,16].”
MethodScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 2×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundRegression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · cited 1×
“An alternative is to directly use models that implicitly perform feature selection such as LASSO [95] or component-wise boosting [84] as they can produce sparse models with fewer features.”
MethodElements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 5×
“The variance of the estimates reduces in O ( 1 m ), where m is the number of evaluated orderings [21,67].”
BackgroundMethodInfluentialA Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“Since MCP is well known in statistics, we refer the practitioner to [24] for an overview and discussion of alternative adjustment methods, such as the Bonferroni-Holm method [44].”
Method"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2005) · doi:10.1002/047174882x · Cited by 13,791
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 3×
“However, further research is required, as there is no single perfect definition of interpretability, but rather multiple depending on the context [25,82].”
BackgroundMethod
Research works printed but not linked: 6
- Brenning, A.: Transforming feature space to interpret machine learning models. arXiv:2104.04295 (2021) · arXiv
- Freiesleben, T.: Counterfactual explanations & adversarial examples-common grounds, essential differences, and potential transfers. arXiv preprint arXiv:2009.05487 (2020) · arXiv
- König, G., Molnar, C., Bischl, B., Grosse-Wentrup, M.: Relative feature importance. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9318–9325. IEEE (2021). https://doi.org/10.1109/ICPR48806.2021.9413090 · DOI
- Molnar, C., Freiesleben, T., König, G., Casalicchio, G., Wright, M.N., Bischl, B.: Relating the partial dependence plot and permutation feature importance to the data generating process. arXiv preprint arXiv:2109.01433 (2021) · arXiv
- Semenova, L., Rudin, C., Parr, R.: A study in Rashomon curves and volumes: a new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755 (2021) · arXiv
- Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning, pp. 3319–3328. PMLR (2017)
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME
Salih, A. M., Raisi‐Estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400304 · Cited by 897
“Despite the limitations of SHAP and LIME in terms of uncertainty estimates, generalization, non-linear dependencies (with LIME), feature dependencies, and inability to infer causality [13], they hold substantial value for explaining and interpreting complex machine learning models.”
BackgroundInterpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
Self-citationExplainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_2 · Cited by 379
Self-citationMachine learning in concrete science: applications, challenges, and best practices
Li, Z., Yoon, J., Zhang, R., Rajabipour, F., Srubar III, W. V., Dabo, I., & Radlińska, A. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00810-x · Cited by 388
Metadata
- Authors
- Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B.
- Affiliations
- Leibniz Institute for Prevention Research and Epidemiology - BIPS (DE)
- University of Vienna (AT)
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Molnar, Christoph
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2022
- Pages
- 39-68
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3194357095
- Cited by
- 211
- References
- 130
- Citations by year
- 2026: 41 · 2025: 56 · 2024: 48 · 2023: 51 · 2022: 11
- FWCI
- 22.95
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 106 d)
- Locations
- Lecture notes in computer science (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Deutsche Forschungsgemeinschaft (437611051)
- Bundesministerium für Bildung und Forschung (01IS18036A)
- Graduate School of Systemic Neurosciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- model-agnostic interpretability, explainable machine learning, partial dependence plots, permutation importance, Shapley value, feature interaction, uncertainty estimation, high-dimensional data, causal interpretation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models. https://app.cochat.ai/papers/10.1007/978-3-031-04083-2_4
- The paper
- Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022). General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-031-04083-2_4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.