Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.

General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models

Open access

Abstract

An increasing number of model-agnostic interpretation techniques for machine learning (ML) models such as partial dependence plots (PDP), permutation feature importance (PFI) and Shapley values provide insightful model interpretations, but can lead to wrong conclusions if applied incorrectly. We highlight many general pitfalls of ML model interpretation, such as using interpretation techniques in the wrong context, interpreting models that do not generalize well, ignoring feature dependencies, interactions, uncertainty estimates and issues in high-dimensional settings, or making unjustified causal interpretations, and illustrate them with examples. We focus on pitfalls for global methods that describe the average model behavior, but many pitfalls also apply to local methods that explain individual predictions. Our paper addresses ML practitioners by raising awareness of pitfalls and identifying solutions for correct model interpretation, but also addresses ML researchers by discussing open issues for further research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-031-04083-2_4

Cites

132 of 132 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 2 Assuming one-fits-all interpretability . A default xgboost regression model that minimizes the mean squared error (MSE) was fitted on 20 independently and uniformly distributed features to predict another independent, uniformly sampled target. In this setting, predicting the (unconditional) mean E [ Y ] in a constant model is optimal. The learner overfits due to a small training data size. Mean marginal SHAP (red, error bars indicate 0 . 05 and 0 . 95 quantiles) exposes all mechanistically used features. In contrast, PFI on test data (blue, error bars indicate 0 . 05 and 0 . 95 quantiles) considers all features to be irrelevant, since no feature contributes to the generalization performance.
Fig. 2 Assuming one-fits-all interpretability . A default xgboost regression model that minimizes the mean squared error (MSE) was fitted on 20 independently and uniformly distributed features to predict another independent, uniformly sampled target. In this setting, predicting the (unconditional) mean E [ Y ] in a constant model is optimal. The learner overfits due to a small training data size. Mean marginal SHAP (red, error bars indicate 0 . 05 and 0 . 95 quantiles) exposes all mechanistically used features. In contrast, PFI on test data (blue, error bars indicate 0 . 05 and 0 . 95 quantiles) considers all features to be irrelevant, since no feature contributes to the generalization performance.CC-BY · From the PDF
Fig. 3 Bad model generalization . Top: Performance estimates on training and test data for a linear regression model (underfitting), a random forest (overfitting) and a support vector machine with radial basis kernel (good fit). The three features are drawn from a uniform distribution, and the target was generated as Y = X 2 1 + X 2 -5 X 1 X 2 + ϵ , with ϵ ∼ N (0 , 5). Bottom: PDPs for the data-generating process (DGP) - which is the ground truth - and for the three models.
Fig. 3 Bad model generalization . Top: Performance estimates on training and test data for a linear regression model (underfitting), a random forest (overfitting) and a support vector machine with radial basis kernel (good fit). The three features are drawn from a uniform distribution, and the target was generated as Y = X 2 1 + X 2 -5 X 1 X 2 + ϵ , with ϵ ∼ N (0 , 5). Bottom: PDPs for the data-generating process (DGP) - which is the ground truth - and for the three models.CC-BY · From the PDF
Fig. 4 Interpretation with extrapolation . Illustration of artificial data points generated by three different perturbation approaches. The black dots refer to observed data points and the red crosses to the artificial data points.
Fig. 4 Interpretation with extrapolation . Illustration of artificial data points generated by three different perturbation approaches. The black dots refer to observed data points and the red crosses to the artificial data points.CC-BY · From the PDF
Fig. 5 Confusing linear correlation with dependence . Highly dependent features X 1 and X 2 that have a correlation close to zero. A test ( H 0 : Features are independent) using Pearson correlation is not significant, but for HSIC, the H 0 -hypothesis gets rejected. Data from [80].
Fig. 5 Confusing linear correlation with dependence . Highly dependent features X 1 and X 2 that have a correlation close to zero. A test ( H 0 : Features are independent) using Pearson correlation is not significant, but for HSIC, the H 0 -hypothesis gets rejected. Data from [80].CC-BY · From the PDF
Fig. 6 Misunderstanding conditional interpretation . A linear model was fitted on the data-generating process modeled using a linear Gaussian structural causal model. The entailed directed acyclic graph is depicted on the left. For illustrative purposes, the original model coefficients were updated such that not only feature X 3 , but also feature X 2 is used by the model. PFI on test data considers both X 3 and X 2 to be relevant. In contrast, conditional feature importance variants either only consider X 3 to be relevant (CFI) or consider all features to be relevant (conditional SAGE value function).
Fig. 6 Misunderstanding conditional interpretation . A linear model was fitted on the data-generating process modeled using a linear Gaussian structural causal model. The entailed directed acyclic graph is depicted on the left. For illustrative purposes, the original model coefficients were updated such that not only feature X 3 , but also feature X 2 is used by the model. PFI on test data considers both X 3 and X 2 to be relevant. In contrast, conditional feature importance variants either only consider X 3 to be relevant (CFI) or consider all features to be relevant (conditional SAGE value function).CC-BY · From the PDF
Fig. 7 Misleading effect due to interactions . Simulation example with interactions: Y = 3 X 1 -6 X 2 + 12 X 2 1 ( X 3 ≥ 0) + ϵ with X 1 , X 2 , X 3 i.i.d. ∼ U [ -1 , 1] and ϵ i.i.d. ∼ N (0 , 0 . 3). A random forest with 500 trees is fitted on 1000 observations. Effects are calculated on 200 randomly sampled (training) observations. A, B: PDP (yellow) and ICE curves of X 1 and X 2 ; C: Derivative ICE curves and their standard deviation of X 2 ; D: 2-dimensional PDP of X 2 and X 3 .
Fig. 7 Misleading effect due to interactions . Simulation example with interactions: Y = 3 X 1 -6 X 2 + 12 X 2 1 ( X 3 ≥ 0) + ϵ with X 1 , X 2 , X 3 i.i.d. ∼ U [ -1 , 1] and ϵ i.i.d. ∼ N (0 , 0 . 3). A random forest with 500 trees is fitted on 1000 observations. Effects are calculated on 200 randomly sampled (training) observations. A, B: PDP (yellow) and ICE curves of X 1 and X 2 ; C: Derivative ICE curves and their standard deviation of X 2 ; D: 2-dimensional PDP of X 2 and X 3 .CC-BY · From the PDF
Fig. 8 Ignoring model and approximation uncertainty . PDP for X 1 with Y = 0 · X 1 + ∑ 10 j =2 X j + ϵ i with X 1 , . . . , X 10 ∼ U [0 , 1] and ϵ i ∼ N (0 , 0 . 9). Left: PDP for X 1 of a random forest trained on 100 data points. Middle: Multiple PDPs (10x) for the model from left plots, but with different samples (each n=100) for PDP estimation. Right: Repeated (10x) data samples of n=100 and newly fitted random forest.
Fig. 8 Ignoring model and approximation uncertainty . PDP for X 1 with Y = 0 · X 1 + ∑ 10 j =2 X j + ϵ i with X 1 , . . . , X 10 ∼ U [0 , 1] and ϵ i ∼ N (0 , 0 . 9). Left: PDP for X 1 of a random forest trained on 100 data points. Middle: Multiple PDPs (10x) for the model from left plots, but with different samples (each n=100) for PDP estimation. Right: Repeated (10x) data samples of n=100 and newly fitted random forest.CC-BY · From the PDF
Fig. 9 Failure to scale to high-dimensional settings . Comparison of the number of features with significant importance - once with and once without Bonferronicorrected significance levels for a varying number of added noise variables. Datasets were sampled from Y = 2 X 1 + 2 X 2 2 + ϵ with X 1 , X 2 , ϵ ∼ N (0 , 1). X 3 , X 4 , ..., X p ∼ N (0 , 1) are additional noise variables with p ranging between 2 and 1000. For each p , we sampled two datasets from this data-generating process - one to train a random forest with 500 trees on and one to test whether feature importances differed from 0 using PIMP. In all experiments, X 1 and X 2 were correctly identified as important.
Fig. 9 Failure to scale to high-dimensional settings . Comparison of the number of features with significant importance - once with and once without Bonferronicorrected significance levels for a varying number of added noise variables. Datasets were sampled from Y = 2 X 1 + 2 X 2 2 + ϵ with X 1 , X 2 , ϵ ∼ N (0 , 1). X 3 , X 4 , ..., X p ∼ N (0 , 1) are additional noise variables with p ranging between 2 and 1000. For each p , we sampled two datasets from this data-generating process - one to train a random forest with 500 trees on and one to test whether feature importances differed from 0 using PIMP. In all experiments, X 1 and X 2 were correctly identified as important.CC-BY · From the PDF
Fig. 10 Causal graph
Fig. 10 Causal graphCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 130 references

Research works printed but not linked: 6

  1. Brenning, A.: Transforming feature space to interpret machine learning models. arXiv:2104.04295 (2021) · arXiv
  2. Freiesleben, T.: Counterfactual explanations & adversarial examples-common grounds, essential differences, and potential transfers. arXiv preprint arXiv:2009.05487 (2020) · arXiv
  3. König, G., Molnar, C., Bischl, B., Grosse-Wentrup, M.: Relative feature importance. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9318–9325. IEEE (2021). https://doi.org/10.1109/ICPR48806.2021.9413090 · DOI
  4. Molnar, C., Freiesleben, T., König, G., Casalicchio, G., Wright, M.N., Bischl, B.: Relating the partial dependence plot and permutation feature importance to the data generating process. arXiv preprint arXiv:2109.01433 (2021) · arXiv
  5. Semenova, L., Rudin, C., Parr, R.: A study in Rashomon curves and volumes: a new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755 (2021) · arXiv
  6. Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning, pp. 3319–3328. PMLR (2017)

Cited by

See all 313

Metadata

Authors
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B.
Affiliations
  • Leibniz Institute for Prevention Research and Epidemiology - BIPS (DE)
  • University of Vienna (AT)
  • Ludwig-Maximilians-Universität München (DE)
Corresponding author
Molnar, Christoph
Venue
Lecture Notes in Computer Science
Published
2022
Pages
39-68
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
OpenAlex
W3194357095
Cited by
211
References
130
Citations by year
2026: 41 · 2025: 56 · 2024: 48 · 2023: 51 · 2022: 11
FWCI
22.95
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Deutsche Forschungsgemeinschaft (437611051)
  • Bundesministerium für Bildung und Forschung (01IS18036A)
  • Graduate School of Systemic Neurosciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
model-agnostic interpretability, explainable machine learning, partial dependence plots, permutation importance, Shapley value, feature interaction, uncertainty estimation, high-dimensional data, causal interpretation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models. https://app.cochat.ai/papers/10.1007/978-3-031-04083-2_4
The paper
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022). General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-031-04083-2_4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.