The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations
Open accessAbstract
Post-hoc interpretability approaches have been proven to be powerful tools to generate explanations for the predictions made by a trained black-box model. However, they create the risk of having explanations that are a result of some artifacts learned by the model instead of actual knowledge from the data. This paper focuses on the case of counterfactual explanations and asks whether the generated instances can be justified, i.e. continuously connected to some ground-truth data. We evaluate the risk of generating unjustified counterfactual examples by investigating the local neighborhoods of instances whose predictions are to be explained and show that this risk is quite high for several datasets. Furthermore, we show that most state of the art approaches do not differentiate justified from unjustified counterfactual examples, leading to less useful explanations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Generated counterfactual samples should be justified, meaning that there is a continuous path from the counterfactual to a sample in the ground-truth training data.
“They argue that generated counterfactual samples should be”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [140]
The notion of 'design transparency' proposed by Michele Loi and his co-authors [52] is close to what the paper described as 'justifications for experts' in Section 4.1.
“As far as operationalization is concerned, the notion of 'design transparency' proposed by Michele Loi and his co-authors [52] is close to what we described as 'justifications for experts' in Section 4.1.”
From Beyond explainability: justifiability and contestability of algorithmic decision systems (2022) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
(1996) · Cited by 20,179 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
A Global Geometric Framework for Nonlinear Dimensionality Reduction
Tenenbaum, J. B., Silva, V. D., & Langford, J. C. (2000) · Science · doi:10.1126/science.290.5500.2319 · Cited by 13,585 · see citing works
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061 · see citing works
How to Explain Individual Classification DecisionsPreprint
Baehrens, D., Schroeter, T. S., Stefan Harmeling, B., Kawanabe, M., Hansen, K., & Mueller, K.-R. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0912.1128 · Cited by 743 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Towards Robust Interpretability with Self-Explaining Neural NetworksPreprint
Alvarez-Melis, D., & Alvarez-Melis, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 545 · see citing works
Counterfactual reasoning and learning systems: the example of computational advertising
Bottou, L., Peters, J., Quiñonero-Candela, J., Charles, D. X., Chickering, D. M., Portugaly, E., Ray, D., Simard, P., & Snelson, E. (2013) · Cited by 497 · see citing works
Local Rule-Based Explanations of Black Box Decision SystemsPreprint
Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., & Giannotti, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10820 · Cited by 323 · see citing works
To Trust Or Not To Trust A ClassifierPreprint
Jiang, H., Kim, B., Guan, M. Y., & Gupta, M. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11783 · Cited by 320 · see citing works
The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification
Kim, B., Rudin, C. D., & Shah, J. (2014) · Cited by 185 · see citing works
Comparison-Based Inverse Classification for Interpretability in Machine Learning
Laugel, T., Lesot, M., Marsala, C., Renard, X., & Detyniecki, M. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-91473-2_9 · Cited by 141 · see citing works
Self-citationPlease Stop Explaining Black Box Models for High Stakes Decisions.Preprint
Rudin, C. D. (2018) · arXiv (Cornell University) · Cited by 138 · see citing works
Turner, R. (2016) · Machine learning for signal processing ... · doi:10.1109/mlsp.2016.7738872 · Cited by 77 · see citing works
Generalized Inverse Classification
Lash, M. T., Lin, Q., Street, W. N., Robinson, J. G., & Ohlmann, J. W. (2017) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611974973.19 · Cited by 64 · see citing works
Making Tree Ensembles InterpretablePreprint
Hara, S., & Hayashi, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05390 · Cited by 60 · see citing works
Defining Locality for Surrogates in Post-hoc InterpretablityPreprint
Laugel, T., Renard, X., Lesot, M., Marsala, C., & Detyniecki, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07498 · Cited by 58 · see citing works
Self-citationUnderstanding classifier errors by examining influential neighbors
Kabra, M., Robie, A. A., & Branson, K. M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299017 · Cited by 23 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Mark W. Craven and Jude W. Shavlik. Extracting tree-structured representations of trained neural networks. Advances in Neural Information Processing Systems, 8:24-30, 1996.
- Michael Lash, Qihang Lin, Nick Street, Jennifer Robinson, and Jeffrey Ohlmann. Generalized in- verse classification. In Proc. of the 2017 SIAM Int. Conf. on Data Mining, pages 162-170, 2017.
Cited by
Sorted by citations
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063 · see citing works
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · see citing works
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · see citing works
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · see citing works
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works
Stepin, I., Alonso, J. M., Catala, A., & Pereira-Farina, M. (2021) · IEEE Access · doi:10.1109/access.2021.3051315 · Cited by 376 · see citing works
Chou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.003 · Cited by 302 · see citing works
Neural Prototype Trees for Interpretable Fine-grained Image Recognition
Nauta, M., van Bree, R., & Seifert, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01469 · Cited by 269 · see citing works
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267 · see citing works
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211 · see citing works
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135 · see citing works
Vale, D., El-Sharif, A., & Ali, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00142-y · Cited by 124 · see citing works
XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112 · see citing works
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works
Beyond explainability: justifiability and contestability of algorithmic decision systems
Henin, C., & Le Métayer, D. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01251-8 · Cited by 71 · see citing works
Darvishvand, L., Kamkari, B., Huang, M. J., & Hewitt, N. J. (2025) · Sustainable Cities and Society · doi:10.1016/j.scs.2025.106492 · Cited by 42 · see citing works
Unjustified Classification Regions and Counterfactual Explanations in Machine Learning
Laugel, T., Lesot, M.-J., Marsala, C., Renard, X., & Detyniecki, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-46147-8_3 · Cited by 28 · see citing works
Self-citationA framework to contest and justify algorithmic decisions
Henin, C., & Le Métayer, D. (2021) · AI and Ethics · doi:10.1007/s43681-021-00054-3 · Cited by 14 · see citing works
- Authors
- Laugel, T., Lesot, M.-J., Marsala, C., Renard, X., & Detyniecki, M.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- AXA (France) (FR)
- Venue
- Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
- Published
- 2019-08
- Pages
- 2801-2807
- OpenAlex
- W2964027122
- Cited by
- 194
- References
- 21
- Citations by year
- 2026: 13 · 2025: 27 · 2024: 25 · 2023: 34 · 2022: 34 · 2021: 36 · 2020: 21 · 2019: 4
- FWCI
- 9.09
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- AXA Research Fund
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- post-hoc explanation, counterfactual explanations, black-box models, model artifacts, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
