General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Open accessThe DOI matches its Crossref record; 133 of the 133 references it cites were found in a registry or an index.
Abstract
An increasing number of model-agnostic interpretation techniques for machine learning (ML) models such as partial dependence plots (PDP), permutation feature importance (PFI) and Shapley values provide insightful model interpretations, but can lead to wrong conclusions if applied incorrectly. We highlight many general pitfalls of ML model interpretation, such as using interpretation techniques in the wrong context, interpreting models that do not generalize well, ignoring feature dependencies, interactions, uncertainty estimates and issues in high-dimensional settings, or making unjustified causal interpretations, and illustrate them with examples. We focus on pitfalls for global methods that describe the average model behavior, but many pitfalls also apply to local methods that explain individual predictions. Our paper addresses ML practitioners by raising awareness of pitfalls and identifying solutions for correct model interpretation, but also addresses ML researchers by discussing open issues for further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 2 Assuming one-fits-all interpretability . A default xgboost regression model that minimizes the mean squared error (MSE) was fitted on 20 independently and uniformly distributed features to predict another independent, uniformly sampled target. In this setting, predicting the (unconditional) mean E [ Y ] in a constant model is optimal. The learner overfits due to a small training data size. Mean marginal SHAP (red, error bars indicate 0 . 05 and 0 . 95 quantiles) exposes all mechanistically used features. In contrast, PFI on test data (blue, error bars indicate 0 . 05 and 0 . 95 quantiles) considers all features to be irrelevant, since no feature contributes to the generalization performance.](https://figures.cochat.ai/paper-figures/d89fa50113fe4204a130eac0040da274/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-2.png)


![Fig. 5 Confusing linear correlation with dependence . Highly dependent features X 1 and X 2 that have a correlation close to zero. A test ( H 0 : Features are independent) using Pearson correlation is not significant, but for HSIC, the H 0 -hypothesis gets rejected. Data from [80].](https://figures.cochat.ai/paper-figures/90068dfc976d4ead9a6c7cbb19095454/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-5.png)

![Fig. 7 Misleading effect due to interactions . Simulation example with interactions: Y = 3 X 1 -6 X 2 + 12 X 2 1 ( X 3 ≥ 0) + ϵ with X 1 , X 2 , X 3 i.i.d. ∼ U [ -1 , 1] and ϵ i.i.d. ∼ N (0 , 0 . 3). A random forest with 500 trees is fitted on 1000 observations. Effects are calculated on 200 randomly sampled (training) observations. A, B: PDP (yellow) and ICE curves of X 1 and X 2 ; C: Derivative ICE curves and their standard deviation of X 2 ; D: 2-dimensional PDP of X 2 and X 3 .](https://figures.cochat.ai/paper-figures/acbe517b1cfe4f9b8d1eb0df227240b1/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-7.png)
![Fig. 8 Ignoring model and approximation uncertainty . PDP for X 1 with Y = 0 · X 1 + ∑ 10 j =2 X j + ϵ i with X 1 , . . . , X 10 ∼ U [0 , 1] and ϵ i ∼ N (0 , 0 . 9). Left: PDP for X 1 of a random forest trained on 100 data points. Middle: Multiple PDPs (10x) for the model from left plots, but with different samples (each n=100) for PDP estimation. Right: Repeated (10x) data samples of n=100 and newly fitted random forest.](https://figures.cochat.ai/paper-figures/bf95f6cae70f49538e3b3bfd16542e68/general-pitfalls-of-model-agnostic-interpretation-methods-for-machine-learning-figure-8.png)


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, these methods can be misleading when correlations exist between input variables 107,109 .”
From Machine learning in concrete science: applications, challenges, and best practices (2022) · cites this paper as [109]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 1×
“Global methods include the partial dependence plot (PDP) [30], partial importance (PI) [16], accumulated local affects (ALE) [2], or the permutation feature importance (PFI) [10,28,16].”
MethodScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 2×
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundRegression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · cited 1×
“An alternative is to directly use models that implicitly perform feature selection such as LASSO [95] or component-wise boosting [84] as they can produce sparse models with fewer features.”
MethodElements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 5×
“The variance of the estimates reduces in O ( 1 m ), where m is the number of evaluated orderings [21,67].”
BackgroundMethodInfluentialA Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“Since MCP is well known in statistics, we refer the practitioner to [24] for an overview and discussion of alternative adjustment methods, such as the Bonferroni-Holm method [44].”
Method"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2005) · doi:10.1002/047174882x · Cited by 13,791
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 3×
“However, further research is required, as there is no single perfect definition of interpretability, but rather multiple depending on the context [25,82].”
BackgroundMethod
Research works printed but not linked: 7
- Brenning, A.: Transforming feature space to interpret machine learning models. arXiv:2104.04295 (2021) · arXiv
- Freiesleben, T.: Counterfactual explanations & adversarial examples-common grounds, essential differences, and potential transfers. arXiv preprint arXiv:2009.05487 (2020) · arXiv
- Curran Associates, Inc. (2017). https://doi.org/ 10.5555/3295222.3295230 · doi:10.5555/3295222.3295230
- Molnar, C., Freiesleben, T., König, G., Casalicchio, G., Wright, M.N., Bischl, B.: Relating the partial dependence plot and permutation feature importance to the data generating process. arXiv preprint arXiv:2109.01433 (2021) · arXiv
- Semenova, L., Rudin, C., Parr, R.: A study in Rashomon curves and volumes: a new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755 (2021) 103. · arXiv
- Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning, pp. 3319-3328. PMLR (2017) 111.
- Alzate, C., et al. (eds.) MIDAS/PAP -2018. LNCS (LNAI), vol. 11054, pp. 81-94. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-13463-1 6 · DOI
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME
Salih, A. M., Raisi‐Estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400304 · Cited by 897
“Despite the limitations of SHAP and LIME in terms of uncertainty estimates, generalization, non-linear dependencies (with LIME), feature dependencies, and inability to infer causality [13], they hold substantial value for explaining and interpreting complex machine learning models.”
BackgroundInterpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
Self-citationExplainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_2 · Cited by 379
Self-citationMachine learning in concrete science: applications, challenges, and best practices
Li, Z., Yoon, J., Zhang, R., Rajabipour, F., Srubar III, W. V., Dabo, I., & Radlińska, A. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00810-x · Cited by 388
Avoiding common machine learning pitfalls
Lones, M. A. (2024) · Patterns · doi:10.1016/j.patter.2024.101046 · Cited by 170
“41 For this reason, you should take care when using either deep learning or XAI for models that are going to make high-stakes or safety-critical decisions; see Rudin 19 for more on this.”
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
How Interpretable Machine Learning Can Benefit Process Understanding in the Geosciences
Jiang, S., Sweet, L., Blougouras, G., Brenning, A., Li, W., Reichstein, M., Denzler, J., Shangguan, W., Yu, G., Huang, F., et al. (2024) · Earth's Future · doi:10.1029/2024ef004540 · Cited by 182
“On the other hand, complex ML models, such as deep neural networks (NNs) and boosting algorithms, often outperform simpler models in terms of accuracy, but lack inherent interpretability (Dramsch, 2020; Molnar et al., 2020).”
InfluentialRyo, M., Angelov, B., Mammola, S., Kass, J. M., Benito, B. M., & Hartig, F. (2021) · Ecography · doi:10.1111/ecog.05360 · Cited by 182
Using SHAP Values and Machine Learning to Understand Trends in the Transient Stability Limit
Hamilton, R. I., & Papadopoulos, P. N. (2023) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2023.3248941 · Cited by 215
“LIME can also only provide additive explanations without separation of main effects and interactions [21].”
BackgroundInfluentialMolnar, C., Freiesleben, T., König, G., Herbinger, J., Reisinger, T., Casalicchio, G., Wright, M. N., & Bischl, B. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44064-9_24 · Cited by 237
Self-citationInterpretable dropout prediction: towards XAI-based personalized intervention
Nagy, M., & Molontay, R. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00331-8 · Cited by 179
“A central element of this work is model interpretability and explainability, which are crucial in machine learning, especially in high-risk environments where predictions have implications (Adadi & Berrada, 2018; Gunning et al., 2019; Molnar, 2020).”
InfluentialAyano, Y. M., Schwenker, F., Dufera, B. D., & Debelee, T. G. (2022) · Diagnostics · doi:10.3390/diagnostics13010111 · Cited by 168
“Furthermore, the local surrogate model may spuriously approximate the complex ML models, i.e., the explanation outcome may have no real connection with the ML model [158,159].”
ResultLu, S.-C., Swisher, C. L., Chung, C. D., Jaffray, D. A., & Sidey-Gibbons, C. J. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1129380 · Cited by 150
Vale, D., El-Sharif, A., & Ali, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00142-y · Cited by 124
Deep learning for wildfire risk prediction: Integrating remote sensing and environmental data
Xu, Z., Li, J., Cheng, S. L., Xue, R., Zhao, Y., He, H., Guan, H., Sharma, A., Erxleben, M., Chang, R., et al. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.06.002 · Cited by 130
Pinheiro, M. G., Madeira, S. C., & Francisco, A. P. (2023) · Applied Energy · doi:10.1016/j.apenergy.2022.120493 · Cited by 119
Beyond prediction: methods for interpreting complex models of soil variation
Wadoux, A. M. J. ‐., & Molnar, C. (2022) · Geoderma · doi:10.1016/j.geoderma.2022.115953 · Cited by 115
Self-citationAI/ML for Network Security: The Emperor has no Clothes
Jacobs, A. S., Beltiukov, R., Willinger, W., Ferreira, R. A., Gupta, A. K., & Granville, L. Z. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560609 · Cited by 157
Moreno-Sánchez, P. A., Isla, G. G., Corino, V., Vehkaoja, A. T., Brukamp, K., van Gils, M., & Mainardi, L. T. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108235 · Cited by 122
Machine learning for an explainable cost prediction of medical insurance
Orji, U. E., & Ukwandu, E. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100516 · Cited by 128
“Generating ICE plots can be time-consuming, especially for large datasets or complex models (Molnar, 2020).”
BackgroundResultGrouped feature importance and combined features effect plot
Au, Q., Herbinger, J., Stachl, C., Bischl, B., & Casalicchio, G. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00840-5 · Cited by 116
“Although the so-called conditional PFI provides a solution to this problem, its interpretation is different and “must be interpreted as the additional, unique contribution of a feature given all remaining features we condition on were known” (Molnar et al. 2020a).”
BackgroundMethodInfluentialSelf-citationCross-Validation Strategy Impacts the Performance and Interpretation of Machine Learning Models
Sweet, L., Müller, C., Anand, M., & Zscheischler, J. (2023) · Artificial Intelligence for the Earth Systems · doi:10.1175/aies-d-23-0026.1 · Cited by 130
A review of evaluation approaches for explainable AI with applications in cardiology
Salih, A. M., Galazzo, I. B., Gkontra, P., Rauseo, E., Lee, A. M., Lekadir, K., Radeva, P. I., Petersen, S. E., & Menegaz, G. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10852-w · Cited by 106
Long short-term memory models of water quality in inland water environments
Pyo, J., Pachepsky, Y. A., Kim, S., Abbas, A., Kim, M., Kwon, Y. S., Ligaray, M. V., & Cho, K. H. (2023) · Water Research X · doi:10.1016/j.wroa.2023.100207 · Cited by 125
Bankruptcy prediction using machine learning and Shapley additive explanations
Nguyen, H. H., Viviani, J., & Jabeur, S. B. (2023) · Review of Quantitative Finance and Accounting · doi:10.1007/s11156-023-01192-x · Cited by 95
“While SHAP dependence plots provide insights into the relationship between features and model output, they do not establish causality because machine learning algorithms are designed to exploit associations rather than model causal relationships (Molnar et al. 2022).”
The Intriguing Relation Between Counterfactual Explanations and Adversarial Examples
Freiesleben, T. (2021) · Minds and Machines · doi:10.1007/s11023-021-09580-9 · Cited by 117
Self-citationInterpretable machine learning for psychological research: Opportunities and pitfalls.
Henninger, M., Debelak, R., Rothacher, Y., & Strobl, C. (2023) · Psychological Methods · doi:10.1037/met0000560 · Cited by 74
Explainable AI In Education : Current Trends, Challenges, And Opportunities
Rachha, A., & Seyam, M. (2023) · SoutheastCon 2023 · doi:10.1109/southeastcon51012.2023.10115140 · Cited by 68
“Molnar et al [36] gives a comprehensible review of explainability techniques in model agnostic XAI [36, 63].”
BackgroundFreiesleben, T., König, G., Molnar, C., & Tejero-Cantero, Á. (2024) · Minds and Machines · doi:10.1007/s11023-024-09691-z · Cited by 82
“Unfortunately, drawing inferences with current IML raises epistemological issues (Molnar et al. 2022): it is often unclear what target quantity IML methods estimate (Lundberg et al. 2021), is it properties of the model or of the phenomenon (Chen et al. 2020, Hooker et al. 2021)?”
Self-citationA Review of Application of Machine Learning in Storm Surge Problems
Qin, Y., Su, C., Chu, D., Zhang, J., & Song, J. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11091729 · Cited by 80
Explainable AI for industrial fault diagnosis: A systematic review
Cação, J., Santos, J., & Antunes, M. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100905 · Cited by 86
Janßen, A., Bennis, F. C., & Mathôt, R. A. A. (2022) · Pharmaceutics · doi:10.3390/pharmaceutics14091814 · Cited by 65
A Guide to Feature Importance Methods for Scientific Inference
Ewald, F. K., Bothmann, L., Wright, M. N., Bischl, B., Casalicchio, G., & König, G. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63797-1_22 · Cited by 77
“Although this approach has potential pitfalls, it can serve as a viable alternative for gaining insights into the inherent patterns and relationships within the observed data, particularly when the generalization error of the ML model is small [43].”
BackgroundMethodInfluentialSelf-citationExplainable Machine Learning with Prior Knowledge: An Overview
Beckh, K., Müller, S., Jakobs, M., Toborek, V., Tan, H., Fischer, R., Welke, P., Houben, S., & von Rueden, L. (2021) · Lancaster EPrints (Lancaster University) · doi:10.48550/arxiv.2105.10172 · Cited by 68
Challenges and opportunities of ML and explainable AI in large-sample hydrology
Slater, L. J., Blougouras, G., Deng, L., Deng, Q., Ford, E., van Dijke, A. J. H., Huang, F., Jiang, S., Liu, Y., Moulds, S., et al. (2025) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2024.0287 · Cited by 73
“Several promising methods are emerging to attempt to assess causality with ML models, but they are not yet widely applied in hydrology.”
A Machine Learning Explainability Tutorial for Atmospheric Sciences
Flora, M. L., Potvin, C. K., McGovern, A., & Handler, S. L. (2023) · Artificial Intelligence for the Earth Systems · doi:10.1175/aies-d-23-0018.1 · Cited by 77
Tideman, L. E. M., Migas, L. G., Djambazova, K. V., Patterson, N. H., Caprioli, R. M., Spraggins, J. M., & de Plas, R. V. (2021) · Analytica Chimica Acta · doi:10.1016/j.aca.2021.338522 · Cited by 54
Dear XAI Community, We Need to Talk! Fundamental Misconceptions in Current XAI Research
Freiesleben, T., & König, G. (2023) · Communications in Computer and Information Science · doi:10.1007/978-3-031-44064-9_3 · Cited by 32
Self-citationChen, X., Liu, M., Wang, Z., & Wang, Y. (2024) · Sensors · doi:10.3390/s24165223 · Cited by 55
“Model interpretability involves examining parameters or feature statistics within the black-box model [29,30].”
BackgroundExplainable AI in healthcare: to explain, to predict, or to describe?
Carriero, A., de Hond, A., Cappers, B. C. M., Paulovich, F. V., Abeln, S., Moons, K. G. M., & van Smeden, M. (2025) · Diagnostic and Prognostic Research · doi:10.1186/s41512-025-00213-8 · Cited by 55
“The post-hoc explainable AI methods typically prioritize human understanding over a faithful (completely correct) description of how the model computes predictions.”
Counterfactual Explanations for Models of Code
Cito, J., Dillig, I., Murali, V., & Chandra, S. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip55303.2022.9794112 · Cited by 66
Forcher, L., Beckmann, T., Wohak, O., Romeike, C., Graf, F., & Altmann, S. (2023) · Science and Medicine in Football · doi:10.1080/24733938.2023.2239766 · Cited by 47
“While, this study provides an insightful model interpretation approach to increase the explainability of the model, the causality of the findings cannot be established (Holzinger et al. 2019; Molnar et al. 2020).”
Bloch, L., & Friedrich, C. M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108029 · Cited by 48
Learning and actioning general principles of cancer cell drug sensitivity
Carli, F., Chiaro, P. D., Morelli, M., Arora, C., Bisceglia, L., Rosa, N. D. O., Cortesi, A., Franceschi, S., Lessi, F., Stefano, A. L. D., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-56827-5 · Cited by 49
“To determine the importance of genes, we utilized two distinct but complementary methods 53 : SHAP (SHapley Additive exPlanations) 20 importance and Permutation Importance 54 .”
Machine learning methods for prediction of cancer driver genes: a survey paper
de Andrades, R. S., & Recamonde‐Mendoza, M. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac062 · Cited by 47
“However, we raise awareness that pitfalls in model interpretability may produce incorrect conclusions [133] just like any other step in ML methodology, thus demanding careful use.”
BackgroundDesigning Inherently Interpretable Machine Learning ModelsPreprint
Sudjianto, A., & Zhang, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.01743 · Cited by 52
Wind power plants hybridised with solar power: A generation forecast perspective
Couto, A., & Estanqueiro, A. I. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.138793 · Cited by 41
Peichl, M., Thober, S., Samaniego, L., Hansjürgens, B., & Marx, A. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-6523-2021 · Cited by 42
Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
Weerts, H., Pfisterer, F., Feurer, M., Eggensperger, K., Bergman, E., Awad, N. H., Vanschoren, J., Pechenizkiy, M., Bischl, B., & Hutter, F. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14747 · Cited by 53
“Going beyond reporting fairness metric scores, more comprehensive evaluations could include grounding metrics in real-world quantities (Corbett-Davies & Goel, 2018), the use of disaggregated evaluations (Barocas et al., 2021) and integration of interpretability tools (Molnar et al., 2022).”
BackgroundSelf-citationNúñez, J., Cortés, C. B., & Yáñez, M. A. (2023) · Water · doi:10.3390/w15193369 · Cited by 39
Model learning with personalized interpretability estimation (ML-PIE)
Virgolin, M., Lorenzo, A. D., Randone, F., Medvet, E., & Wahde, M. (2021) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3449726.3463166 · Cited by 43
König, G., Molnar, C., Bischl, B., & Grosse‐Wentrup, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68787-8 · Cited by 36
Self-citationWang, R., Kuo, P., Chen, L.-C., Seastedt, K. P., Gichoya, J. W., & Celi, L. A. (2024) · EBioMedicine · doi:10.1016/j.ebiom.2024.105047 · Cited by 39
“66 69 The extreme complexity of radiographic images renders many of the explanations provided by machines opaque to human comprehension.”
Background(2025) · doi:10.1038/s41467-025-57181-2 · Cited by 34
Predicting T cell receptor functionality against mutant epitopes
Drost, F., Dorigatti, E., Straub, A., Hilgendorf, P., Wagner, K. I., Heyer, K., Montes, M. L., Bischl, B., Busch, D. H., Schober, K., et al. (2024) · Cell Genomics · doi:10.1016/j.xgen.2024.100634 · Cited by 39
Self-citationExplainable AI for trustworthy intelligent process monitoring
Johannssen, A., Qiu, P., Yeganeh, A., & Chukhrova, N. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.111407 · Cited by 43
Explaining a Deep Reinforcement Learning (DRL)-Based Automated Driving Agent in Highway Simulations
Bellotti, F., Lazzaroni, L., Capello, A., Cossu, M., Gloria, A. D., & Berta, R. (2023) · IEEE Access · doi:10.1109/access.2023.3259544 · Cited by 35
“Very interesting and useful, [59] warns of many general pitfalls of ML model interpretation, such as using interpretation techniques in the wrong context, interpreting models that do not generalize well, ignoring feature dependencies, interactions, uncertainty estimates and issues in…”
BackgroundGAM Changer: Editing Generalized Additive Models with Interactive VisualizationPreprint
Wang, Z. J., Kale, A. M., Nori, H., Stella, P., Nunnally, M., Chau, D. H., Vorvoreanu, M., Vaughan, J. W., & Caruana, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.03245 · Cited by 37
“Feature correlations and confounders convolute ML interpretation [24].”
BackgroundInterpretable artificial intelligence in radiology and radiation oncology
Cui, S., Traverso, A., Niraula, D., Zou, J., Luo, Y., Owen, D., Naqa, I. E., & Wei, L. (2023) · British Journal of Radiology · doi:10.1259/bjr.20230142 · Cited by 36
Aldeia, G. S. I., & de França, F. O. (2022) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-022-09435-x · Cited by 38
“Several works in the literature explores and proposes different ways of generating synthetic data sets to evaluate explanations [14, 41, 46, 47, 49, 51].”
BackgroundA general steganographic framework for neural network models
Yang, Z., Wang, Z., & Zhang, X. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119250 · Cited by 32
Liu, X., Taylor, M. P., Aelion, C. M., & Dong, C. (2021) · Environmental Science & Technology · doi:10.1021/acs.est.1c01097 · Cited by 39
Amenaghawon, A. N., Igemhokhai, S., Eshiemogie, S. A., Ugbodu, F., & Evbarunegbe, N. I. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25432 · Cited by 31
The promise of automated machine learning for the genetic analysis of complex traits
Manduchi, E., Romano, J. D., & Moore, J. H. (2021) · Human Genetics · doi:10.1007/s00439-021-02393-x · Cited by 31
“…point to keep in mind to avoid misleading interpretations, is that permutation importance does not separate between main and interaction effects, but reflects both the importance of a feature as well as that of its interactions with the other features (Casalicchio et al. 2019; Molnar et al. 2021).”
BackgroundMethodTakefuji, Y. (2025) · Water Research · doi:10.1016/j.watres.2025.123514 · Cited by 28
“Consequently, these methods inherently introduce biases because each employs distinct assumptions and limitations that may skew the interpretation of a feature ’ s role in the model ’ s predictions (Bilodeau, 2024; Cross, 2024; Momenzadeh, 2022; Huang, 2024; Kumar, 2021; Lones, 2024; Molnar, 2022).”
Confronting Machine Learning with Financial Research
Lommers, K., Harzli, O. E., & Kim, J. (2021) · The Journal of Financial Data Science · doi:10.2139/ssrn.3788349 · Cited by 30
Beyond the black box with biologically informed neural networks
Selby, D., Sprang, M., Ewald, J., & Vollmer, S. J. (2025) · Nature Reviews Genetics · doi:10.1038/s41576-025-00826-1 · Cited by 27
Interpretable Machine Learning for Survival Analysis
Langbein, S., Krzyziński, M., Spytek, M., Baniecki, H., Biecek, P., & Wright, M. N. (2025) · Biometrical Journal · doi:10.1002/bimj.70089 · Cited by 34
iPDP: On Partial Dependence Plots in Dynamic Modeling Scenarios
Muschalik, M., Fumagalli, F., Jagtani, R., Hammer, B., & Hüllermeier, E. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44064-9_11 · Cited by 36
“Furthermore, it is possible to use the dynamic quantiles instead as grid points depending on the application scenario [28,37].”
BackgroundCausal Explanations and XAIPreprint
Beckers, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.13169 · Cited by 34
“Yet there is also a growing interest in using ML-models for action-guidance, and that involves predictions about the second rung of the ladder, interventions (Lipton, 2018; Molnar et al., 2021; Schölkopf et al., 2021).”
BackgroundBrenning, A. (2023) · Machine Learning · doi:10.1007/s10994-023-06327-8 · Cited by 26
Saleh, H., El-Rashidy, N. M., Elaziz, M. A., Aseeri, A. O., & El–Sappagh, S. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00514-z · Cited by 30
“The objective function is the local minimization of L without any assumptions about f because LIME is model-agnostic.”
Beyond model interpretability: socio-structural explanations in machine learning
Smart, A., & Kasirzadeh, A. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02056-1 · Cited by 25
Mortier, S., Hamedpour, A., Bussmann, B., Wandji, R. T., Latré, S., Sigurdsson, B. D., Schepper, T. D., & Verdonck, T. (2024) · Ecological Informatics · doi:10.1016/j.ecoinf.2024.102730 · Cited by 27
Artificial intelligence and digital twins for the personalised prediction of hypertension risk
Naik, A., Nalepa, J., Wijata, A. M., Mahon, J., Mistry, D., Knowles, A. T., Dawson, E. A., Lip, G. Y. H., Olier, I., & Ortega‐Martorell, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110718 · Cited by 28
Inherently Interpretable Machine Learning: A Contrasting Paradigm to Post-hoc Explainable AI
Zschech, P., Weinzierl, S., & Kraus, M. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00964-0 · Cited by 35
Folgado, D., Barandas, M., Famiglini, L., Santos, R., Cabitza, F., & Gamboa, H. F. S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101955 · Cited by 28
Harnessing Prior Knowledge for Explainable Machine Learning: An Overview
Beckh, K., Müller, S., Jakobs, M., Toborek, V., Tan, H., Fischer, R., Welke, P., Houben, S., & von Rueden, L. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00038 · Cited by 33
“[13] point out pitfalls of explainability methods that require additional knowledge by the user in order to gain reliable new insights and to prevent false conclusions.”
BackgroundAin, Q. U., Nazir, R., Nawaz, A., Shahbaz, H., Dilshad, A., Mufti, I. U., & Iftikhar, M. (2024) · Journal of Clinical and Translational Pathology · doi:10.14218/jctp.2023.00061 · Cited by 27
“82 Supervised implications of these regulations require strong security of data storage, access, and management and secure data storage.”
Henninger, M., & Strobl, C. (2024) · Behaviormetrika · doi:10.1007/s41237-024-00253-2 · Cited by 29
“Third, interpreting the ML model’s prediction can help to better understand the underlying social science processes, but also to assess whether the ML model makes reasonable predictions (see Lavelle-Hill et al. 2023; Molnar et al. 2020; Murdoch et al. 2019, for a discussion).”
Limitations of XAI Methods for Process-Level Understanding in the Atmospheric Sciences
Silva, S. J., & Keller, C. A. (2023) · Artificial Intelligence for the Earth Systems · doi:10.1175/aies-d-23-0045.1 · Cited by 28
Interpretable machine learning for time-to-event prediction in medicine and healthcare
Baniecki, H., Sobieski, B., Szatkowski, P., Bombiński, P., & Biecek, P. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103026 · Cited by 28
“We foresee several limitations of the introduced methods related to general limitations of post-hoc interpretability methods [55].”
Can local explanation techniques explain linear additive models?
Rahnama, A. H. A., Bütepage, J., Geurts, P., & Boström, H. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00971-3 · Cited by 23
Raufi, B., & Longo, L. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4010048 · Cited by 19
“These methods are effective, as they avoid common mistakes such as using a “one-size-fits-all” approach to interpretability, poor model generalization, over-reliance on complex models for explainability, and neglecting feature dependence [62].”
Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics
Klein, L., Lüth, C. T., Schlegel, U., Bungert, T. J., El‐Assady, M., & Jäger, P. F. (2024) · neural information processing systems · doi:10.52202/079017-2142 · Cited by 40
How much do we see? On the explainability of partial dependence plots for credit risk scoring
Szepannaek, G., & Lübke, K. (2023) · Argumenta Oeconomica · doi:10.15611/aoe.2023.1.07 · Cited by 25
Predicting the Occurrence of Metabolic Syndrome Using Machine Learning Models
Τρίγκα, Μ., & Dritsas, E. (2023) · Computation · doi:10.3390/computation11090170 · Cited by 20
Preventing deception with explanation methods using focused sampling
Vreš, D., & Robnik‐Šikonja, M. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00900-w · Cited by 21
“Molnar et al. (2021) point out some pitfalls of model agnostic explanation methods when they are not used correctly.”
User-centric evaluation of explainability of AI with and for humans: A comprehensive empirical study
Bobek, S., Korycińska, P., Krakowska, M., Mozolewski, M., Rak, D., Zych, M., Wójcik, M., & Nalepa, G. J. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103625 · Cited by 27
Variable Importance in High-Dimensional Settings Requires Grouping
Chamma, A., Thirion, B., & Engemann, D. A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.28997 · Cited by 19
Marginal effects for non-linear prediction functions
Scholbeck, C. A., Casalicchio, G., Molnar, C., Bischl, B., & Heumann, C. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00993-x · Cited by 22
Self-citationDecoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks
Blücher, S., Vielhaben, J., & Strodthoff, N. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2401.06654 · Cited by 30
Importance of 3D convolution and physics on a deep learning coastal fog model
Kamangir, H., Krell, E., Collins, W. G., King, S. A., & Tissot, P. E. (2022) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2022.105424 · Cited by 16
“Molnar (2020) further discusses the advantages and disadvantages of SHAP.”
BackgroundMethodInfluentialMaking sense of the black‐boxes: Toward interpretable text classification using deep learning models
Tao, J., Zhou, L., & Hickey, K. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24642 · Cited by 20
“Another key challenge of model interpretations identified in recent studies (Azodi et al., 2020; Molnar, Casalicchio, & Bischl, 2020) is their dependence on features used in building the models, which calls for superior methods for feature perturbation.”
BackgroundLima, H. S., de Oliveira, G. F. V., Ferreira, R. D. S., de Castro, A. G., Silva, L. C. F., Ferreira, L. D. S., Oliveira, D. A. D. S., da Silva, L. F., Kasuya, M. C. M., de Paula, S. O., et al. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.120559 · Cited by 15
Cherry on Top or Real Need? A Review of Explainable Machine Learning in Kidney Transplantation
de Souza, A. A., Stubbs, A., Hesselink, D. A., Baan, C. C., & Boer, K. (2024) · Transplantation · doi:10.1097/tp.0000000000005063 · Cited by 15
Beyond Pixels: Enhancing LIME with Hierarchical Features and Segmentation Foundation Models
Knab, P., Marton, S., & Bartelt, C. (2025) · Frontiers in artificial intelligence and applications · doi:10.3233/faia250827 · Cited by 17
“Moreover, humans often struggle to interpret the explanations, as the highlighted areas do not align with our intuitive understanding (Molnar et al., 2022; Kim et al., 2022).”
Santoliquido, F., Dupletsa, U., Tissino, J., Branchesi, M., Iacovelli, F., Iorio, G., Mapelli, M., Gerosa, D., Harms, J., & Pasquato, M. (2024) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202450381 · Cited by 15
“However, this approximation is only valid under certain conditions, most notably, the high-S / N limit (Vallisneri 2008).”
BackgroundInitialization Noise in Image Gradients and Saliency Maps
Woerl, A.-C., Disselhoff, J., & Wand, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00176 · Cited by 21
“Limitations of attribution: Explaining phenomena by data-driven learning remains challenging [9, 22].”
Background
Metadata
- Authors
- Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B.
- Affiliations
- Leibniz Institute for Prevention Research and Epidemiology - BIPS (DE)
- University of Vienna (AT)
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Molnar, Christoph
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2022
- Pages
- 39-68
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3194357095
- Cited by
- 211
- References
- 130
- Citations by year
- 2026: 41 · 2025: 56 · 2024: 48 · 2023: 51 · 2022: 11
- FWCI
- 22.95
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 106 d)
- Locations
- Lecture notes in computer science (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Deutsche Forschungsgemeinschaft (437611051)
- Bundesministerium für Bildung und Forschung (01IS18036A)
- Graduate School of Systemic Neurosciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- model-agnostic interpretability, explainable machine learning, partial dependence plots, permutation importance, Shapley value, feature interaction, uncertainty estimation, high-dimensional data, causal interpretation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.