Explaining Explanations in AI
Open accessThe DOI matches its Crossref record; 98 of the 100 references it cites were found in a registry or an index; none of the 58 sources its claims cite is checked for support yet.
Abstract
Recent work on interpretability in machine learning and AI has focused on the building of simplified models that approximate the true criteria used to make decisions. These models are a useful pedagogical device for teaching trained professionals how to predict what decisions will be made by the complex system, and most importantly how the system might break. However, when considering any such model it's important to remember Box's maxim that "All models are wrong but some are useful." We focus on the distinction between these models and explanations in philosophy and sociology. These models can be understood as a "do it yourself kit" for explanations, allowing a practitioner to directly answer "what if questions" or generate contrastive explanations without external assistance. Although a valuable ability, giving these models as explanations appears more difficult than necessary, and other forms of explanation may not have the same trade-offs. We contrast the different schools of thought on what makes an explanation, and suggest that machine learning might benefit from viewing the problem more broadly.
Overview
CoChat Summary
TL;DR
The paper argues that most explainable AI (xAI) methods produce simplified approximations of black-box models, which function more like scientific models than explanations as studied in philosophy, cognitive science, and social science. It calls for contrastive, selective, and socially interactive explanations that support contestability and accountability.
Key findings
- xAI’s dominant approach retro-fits local or global simplified models—linear/gradient-based approximations or decision-tree methods—over complex algorithms; these are useful for experts and debugging but can be misleading to non-experts when their domain and limitations are undocumented.
- Work in the “explanation sciences” indicates that everyday explanations are contrastive, selective, and social: people ask why P rather than Q, select one or two causes from many, and explanations involve interactive dialogue and argumentation.
- Contrastive explanation methods (e.g., Martens and Provost 2013; Wachter et al. 2018) directly return alternative data points and avoid some approximation/domain problems, but raise relevance concerns if the counterfactual does not correspond to a user’s interests.
- The paper argues xAI should develop interactive post-hoc methods that make algorithmic decisions contestable and facilitate dialogue among users, developers, systems, and stakeholders; it states the field largely fails this task at present.
Methods
- Conceptual/position paper; no empirical study, dataset, or quantitative analysis is reported.
- Reviews xAI interpretability approaches—transparency versus post-hoc interpretation, local/global approximations, and linear/gradient versus tree-based methods—and contrasts them with philosophical, cognitive science, and social science literature on explanation.
- Uses argumentation and synthesis, with illustrative examples such as Newtonian physics, a bee/spider example, and image binarization choices.
Limitations
- Stated: a complete overview of the philosophy of explanation is unfeasible; the paper focuses solely on ways of providing explanations; contrastive theories remain debated; and choosing relevant contrast cases is not straightforward.
- Evident: no empirical validation, user study, or comparative evaluation of explanation methods is presented; claims about misleading non-experts and the need for interactive methods are argued rather than tested.
- Evident: the analysis relies on secondary literature and selected examples, with no new data or formal metrics.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 8 Not valid · 42 Not checked · 64 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“Recent work on interpretability in machine learning and AI has focused on the building of simplified models that approximate the true criteria used to make decisions.”
- Also stated in: § 3 EXPLAINABLE AI · p. 3 · plain
“These simplified models approximate the true criteria used to make decisions (e.g. Baehrens et al. (2010); Ribeiro et al. (2016); Selvaraju et al. (2016); Simonyan et al. (2013)).”
What the sources say
- Not checked yetcites Baehrens et al., 2010
- Not checked yetcites Ribeiro et al., 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3 EXPLAINABLE AI · p. 3 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We then examine research on explanations in philosophy, cognitive science, and the social sciences which suggests that 'why-questions' (e.g. 'why did the model exhibit that behaviour?') require explanations that are contrastive, selective, and socially interactive.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that these approximations function more like scientific models than the types of scientific and 'everyday' explanations considered in philosophy, cognitive science, and psychology.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If this comparison holds, it follows that the majority of currently available methods will, at best, produce locally reliable but globally misleading explanations of model functionality.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A statement the paper makes itself — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, local approximations are accurate representations only of a specific domain or 'slice' of a model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 114 claims and to verify the 58 references still open.
- Probably not a substantial claim. 42
- It may state two separate claims as one. 28
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There can be multiple ways of making data and code accessible, and there can be multiple ways to justify a decision and different degrees of transparency in the decision (Miller 2019; Mittelstadt et al. 2019).”
From Explaining Why the Computer Says No: Algorithmic Transparency Affects the Perceived Trustworthiness of Automated Decision‐Making (2023) · cites this paper as Mittelstadt et al. 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For users to trust a system, they need to know over which domain a model is reliable, where it is uncertain, and where it is likely to break down86.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Critics of the current principles-based approach to AI governance have emphasized a number of limits to operationalizing fairness, the practical limits of providing algorithmic explainability or transparency, and the lack of professional accountability mechanisms needed to ensure their consistent implementation [119 -121].”
From Artificial intelligence, systemic risks, and sustainability (2021) · cites this paper as [121]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Explainability tools can (at least currently) provide only an approximation of the decision-making functions in black-box systems (Mittelstadt et al., 2019)”
From Hidden Risks: Artificial Intelligence and Hermeneutic Harm (2025) · cites this paper as Mittelstadt et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Citizens want to receive clear reasons for a (negative) decision to assess the fairness of a decision (De Fine Licht and De Fine Licht 2020; Tyler 2006) or to be able to contest an algorithmic decision (Mittelstadt et al. 2019).”
From Explaining Why the Computer Says No: Algorithmic Transparency Affects the Perceived Trustworthiness of Automated Decision‐Making (2023) · cites this paper as Mittelstadt et al. 2019
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 12×
“However, very few stop to check what they are agreeing to, and to find out what explainable AI means to other people involved in the discussion (Lipton, 2016).”
BackgroundInfluentialDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 4×
“These simplified models approximate the true criteria used to make decisions (e.g. Baehrens et al. (2010); Ribeiro et al. (2016); Selvaraju et al. (2016); Simonyan et al. (2013)).”
BackgroundMethodInfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1×
“Poorly interpretable models “are opaque in the sense that if one is a recipient of the output of the algorithm (the classification decision), rarely does one have any concrete sense of how or why a particular classification has been arrived at from inputs” (Burrell, 2016, p.1).”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“…interpretation methods due to this potential to mislead recipients, for example by falsely attributing a decision to an irrelevant feature, or a more acceptable feature (e.g. post code, ’leadership’) that serves as a proxy for a legally protected feature (cf. Barocas and Selbst (2016); Kim (2016)).”
BackgroundLearning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · cited 3×
“…should we only pay attention to contrasting solutions closest to the original solution as suggested by DeepLift (illustrated by Figure 1 centre; (Shrikumar et al., 2017, 2016)); uniformly weight over all possible values, suggested by SHAPE (Lundberg and Lee, 2017) (Figure 1 right); or pay as…”
MethodMethods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 2×
“e: transparency and post-hoc interpretation. Transparency addresses how a model functions internally, whereas post-hoc interpretations concern how the model behaves (Lepri et al., 2017; Lipton, 2016; Montavon et al., 2017). Transparency can be further specified according to its target. Respectively, a mechanistic unde…”
Models and Analogies in Science
Ardley, G. W. R., & Philosophical Studies, W. S. F. T. N. U. O. I. (1965) · Philosophical Studies · doi:10.5840/philstudies19651438 · Cited by 1,762
Proceedings for the Annual Meeting of the Cognitive Science Society
Dziemianko, M., Clarke, A. D. F., & Keller, F. (2013) · Cited by 1,726
Hidden factors and hidden topics: understanding rating dimensions with review text
McAuley, J., & Leskovec, J. (2013) · Proceedings of the 7th ACM conference on Recommender systems · doi:10.1145/2507157.2507163 · Cited by 1,564 · cited 2×
“According to Lipton (2016), approaches to post-hoc interpretability include verbal (natural language) explanations (e.g. McAuley and Leskovec (2013)), visualisations and interactive interfaces (e.g. Simonyan et al. (2013); Tam-agnini et al. (2017)), local explanations or approximations (e.g. Fong…”
Robustness in the Strategy of Scientific Model Building
Box, G. E. P. (1979) · Robustness in Statistics · doi:10.1016/b978-0-12-438150-6.50018-2 · Cited by 1,562 · cited 4×
“accurate descriptions of the phenomena they are interested in, but which may prove inaccurate in a larger domain. It is in this context that Box’s maxim, “All models are wrong, but some are useful,” (Box, 1979) should be understood. Explainable AI generates approximate simple models and calls them ‘explanations’, sugg…”
InfluentialThe Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080 · cited 2×
“…of a system (i.e. as a type of ‘debugging’; (Kulesza et al., 2015)) and help developers and humans working with a system learn from it (Samek et al., 2017), and enhance the trust between individuals subject to a decision and the system itself (Citron and Pasquale, 2014; Hildebrandt…”
BackgroundDatta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · cited 2×
“…seek to explain how a fixed model leads to a particular prediction, either by fitting a simpler, local model around a particular decision (Ribeiro et al., 2016), or by perturbing variables to measure how the prediction changes (Adler et al., 2016; Datta et al., 2016; Simonyan et al., 2013).”
BackgroundMethodLepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · cited 2×
“parent by explanations at a minimum of three levels: “at the level of the entire model, at the level of individual components (e.g., parameters), and at the level of a particular training algorithm” (Lepri et al., 2017). A model, its component parts, or its training/learning algorithm can thus be said to be transparen…”
BackgroundProceedings of the AMIA Symposium
Schulz, S., Daumke, P., Smith, B., & Hahn, U. (2005) · Cited by 606
Extracting Tree-Structured Representations of Trained Networks
Craven, M. W., & SHAVLIK, J. W. (1995) · neural information processing systems · Cited by 602
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 1×
“…to comply with relevant legislation (Doshi-Velez et al., 2017), verify and improve the functionality of a system (i.e. as a type of ‘debugging’; (Kulesza et al., 2015)) and help developers and humans working with a system learn from it (Samek et al., 2017), and enhance the trust between…”
BackgroundExplanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
Gregor, S., & Benbasat, I. (1999) · MIS Quarterly · doi:10.2307/249487 · Cited by 535 · cited 1×
“Gregor and Benbasat (1999) support the importance of abnormality, suggesting that users request explanations when an anomaly or abnormal event is detected.”
BackgroundThe Black Box Society: The Secret Algorithms That Control Money and Information
(2015) · Choice Reviews Online · doi:10.5860/choice.190706 · Cited by 533 · cited 1×
“they have become even more important (Bodo et al., 2017; Kroll et al., 2016; Nissenbaum, 1996; Olhede and Wolfe, 2018; Pasquale, 2015; Selbst and Barocas, 2018; Veale and Edwards, 2018).”
Background(2014) · Digital Commons at University of Maryland Carey Law (University of Maryland Francis King Carey School of Law) · Cited by 524
How the Machine 'Thinks:' Understanding Opacity in Machine Learning AlgorithmsPreprint
Burrell, J. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2660674 · Cited by 511
Knowledge-based causal attribution: The abnormal conditions focus model.
Hilton, D. J., & Slugoski, B. R. (1986) · Psychological Review · doi:10.1037/0033-295x.93.1.75 · Cited by 503
Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 496 · cited 1×
“…should we only pay attention to contrasting solutions closest to the original solution as suggested by DeepLift (illustrated by Figure 1 centre; (Shrikumar et al., 2017, 2016)); uniformly weight over all possible values, suggested by SHAPE (Lundberg and Lee, 2017) (Figure 1 right); or pay as…”
MethodAccountability of AI Under the Law: The Role of Explanation
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., et al. (2017) · doi:10.2139/ssrn.3064761 · Cited by 457 · cited 1×
“Explanations can be necessary to comply with relevant legislation (Doshi-Velez et al., 2017), verify and improve the functionality of a system (i.e. as a type of ‘debugging’; (Kulesza et al., 2015)) and help developers and humans working with a system learn from it (Samek et al., 2017), and enhance…”
BackgroundAlgorithmic Accountability and Public Reason
Binns, R. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0263-5 · Cited by 457 · cited 2×
“Justification has recently received increasing attention by scholars discussing algorithmic accountability (e.g. Binns (2017); Biran and McKeown (2014); Hildebrandt and Koops (2010); Selbst and Barocas (2018); Weller (2017)).”
BackgroundConversational processes and causal explanation.
Hilton, D. J. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.1.65 · Cited by 456 · cited 2×
“Explanations can be conceived as involving one or more explainers and explainees engaging in information transfer through dialogue, visual representation, or other means (Hilton, 1990), often to correct information or knowledge assymetry (Lim and Dey, 2009).”
Background(1979) · Elsevier eBooks · doi:10.1016/c2013-0-11050-1 · Cited by 454
Comprehensible credit scoring models using rule extraction from support vector machines
Martens, D., Baesens, B., Van Gestel, T., & Vanthienen, J. (2007) · European Journal of Operational Research · doi:10.1016/j.ejor.2006.04.051 · Cited by 448 · cited 1×
“Historically, much work has focused on global approximations of models (e.g. Craven and Shavlik (1996); Martens et al. (2007); Sanchez et al. (2015), including approaches based on clustering (Chen et al., 2016), integer programming (Zeng et al., 2017), and rule lists (Wang and Rudin, 2015)).”
BackgroundThe Intuitive Appeal of Explainable Machines
Selbst, A. D., & Barocas, S. (2018) · doi:10.2139/ssrn.3126971 · Cited by 392 · cited 4×
“abermas, 1984)). Justification has recently received increasing attention by scholars discussing algorithmic accountability (e.g. Binns (2017); Biran and McKeown (2014); Hildebrandt and Koops (2010); Selbst and Barocas (2018); Weller (2017)). However, justification currently occupies an uneasy position in xAI, in that…”
InfluentialData-Driven Discrimination at Work
Kim, P. T. (2017) · William and Mary law review · Cited by 383
Accountability in a computerized society
Nissenbaum, H. (1996) · Science and Engineering Ethics · doi:10.1007/bf02639315 · Cited by 346 · cited 1×
“These issues are not unique to computerised decision-making systems, but with the growth of machine learning based systems they have become even more important (Bodo et al., 2017; Kroll et al., 2016; Nissenbaum, 1996; Olhede and Wolfe, 2018; Pasquale, 2015; Selbst and Barocas, 2018; Veale and Edwards, 2018). What dist…”
BackgroundThe Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification
Kim, B., Rudin, C. D., & Shah, J. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 340
Grad-CAM: Why did you say that?
Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07450 · Cited by 309
Lipton, P. (1990) · Royal Institute of Philosophy Supplement · doi:10.1017/s1358246100005130 · Cited by 281 · cited 9×
“However, very few stop to check what they are agreeing to, and to find out what explainable AI means to other people involved in the discussion (Lipton, 2016).”
InfluentialAuditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280
Auditing Black-Box Models for Indirect Influence
Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0011 · Cited by 280
Scientific Representation and the Semantic View of Theories
FRIGG, R. (2006) · THEORIA · doi:10.1387/theoria.553 · Cited by 276 · cited 2×
“Rather, the reviewed empirical evidence indicates that humans psychologically prefer contrastive explanations (Miller, 2017, p.18)(Rehder, 2003, 2006).”
BackgroundExplaining Data-Driven Document Classifications1
Martens, D., & Provost, F. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.1.04 · Cited by 264 · cited 3×
“ues arise in the fitting of models: if the domain the model is fitted to does not capture examples relevant to the intended audience, then it is unlikely to be useful. The first approach described by Martens and Provost (2013) is explicitly designed for use on discrete data, and focuses on the particular problem of wh…”
BackgroundMethodInfluentialAssessing demand for intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2009) · Proceedings of the 11th international conference on Ubiquitous computing · doi:10.1145/1620545.1620576 · Cited by 246 · cited 3×
“Explanations can be conceived as involving one or more explainers and explainees engaging in information transfer through dialogue, visual representation, or other means (Hilton, 1990), often to correct information or knowledge assymetry (Lim and Dey, 2009).”
Interpretable classification models for recidivism prediction
Zeng, J., Ustun, B., & Rudin, C. (2017) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12227 · Cited by 220 · cited 1×
“Historically, much work has focused on global approximations of models (e.g. Craven and Shavlik (1996); Martens et al. (2007); Sanchez et al. (2015), including approaches based on clustering (Chen et al., 2016), integer programming (Zeng et al., 2017), and rule lists (Wang and Rudin, 2015)).”
BackgroundMiller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218 · cited 1×
“ity of currently available methods will, at best, produce locally reliable but globally 2The dichotomy between explanations in machine learning and elsewhere is well illustrated by the recent work by Miller et al. (2017) who found that none of the papers on a curated reading list for xAI made use of work from the soci…”
BackgroundA Causal-Model Theory of Conceptual Representation and Categorization.
Rehder, B. (2003) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.29.6.1141 · Cited by 199
Ruben, D.-H. (2003) · Routledge eBooks · doi:10.4324/9780203169308 · Cited by 191
Transparency: Motivations and ChallengesPreprint
Weller, A. (2017) · arXiv (Cornell University) · doi:10.17863/cam.101505 · Cited by 188
Causal and Constitutive Explanation Compared
Ylikoski, P. (2013) · Erkenntnis · doi:10.1007/s10670-013-9513-9 · Cited by 186 · cited 1×
“A given cause of set of causes may be incomplete insofar as they are not the sole cause of the event, but nonetheless convey useful information to the recipient in a given context or for a given purpose (Ylikoski, 2013).”
Contemporary science and natural explanation: commonsense conceptions of causality
(1988) · Choice Reviews Online · doi:10.5860/choice.26-1476 · Cited by 185 · cited 1×
“Further, to be informative, explanations should not be entirely reducible to presuppositions, or beliefs that the recipient of the explanation already holds (Hesslow, 1988).”
BackgroundThe Challenges of Ambient Law and Legal Protection in the Profiling Era
Hildebrandt, M., & Koops, B. (2010) · The Modern Law Review · doi:10.1111/j.1468-2230.2010.00806.x · Cited by 177 · cited 2×
“…as a type of ‘debugging’; (Kulesza et al., 2015)) and help developers and humans working with a system learn from it (Samek et al., 2017), and enhance the trust between individuals subject to a decision and the system itself (Citron and Pasquale, 2014; Hildebrandt and Koops, 2010; Zarsky, 2013).”
(1991) · Choice Reviews Online · doi:10.5860/choice.28-5038 · Cited by 164
Veale, M., & Edwards, L. (2018) · Computer Law & Security Review · doi:10.1016/j.clsr.2017.12.002 · Cited by 162 · cited 1×
“…are not unique to computerised decision-making systems, but with the growth of machine learning based systems they have become even more important (Bodo et al., 2017; Kroll et al., 2016; Nissenbaum, 1996; Olhede and Wolfe, 2018; Pasquale, 2015; Selbst and Barocas, 2018; Veale and Edwards, 2018).”
Background(2015) · International Conference on Artificial Intelligence and Statistics · Cited by 162
Explanation, Invariance, and Intervention
Woodward, J. (1997) · Philosophy of Science · doi:10.1086/392584 · Cited by 156 · cited 1×
“The explanations requested are thus not full scientific explanations, as they need not appeal to general relationships or scientific laws, but rather at most to causal relationships between the set of variables in a given model (Woodward, 1997).”
BackgroundMantelero, A. (2016) · Computer Law & Security Review · doi:10.1016/j.clsr.2016.01.014 · Cited by 147 · cited 1×
“…developers, professionals working in tandem with a system (e.g. expert labelers of training cases; see Berendt and Preibusch (2017) for an overview of human actors involved in algorithmic decision-making), and to individuals or groups affected by a system’s outputs (Mantelero, 2016; Weller, 2017).”
BackgroundBodó, B., Helberger, N., Irion, K., Borgesius, F. Z., Möller, J. E., van de Velde, B., Bol, N., van Es, B., & de Vreese, C. H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.25337 · Cited by 132
Visual explanation of evidence in additive classifiers
Poulin, B., Eisner, R., Szafron, D., Lu, P., Greiner, R., Wishart, D. S., Fyshe, A., Pearcy, B., Macdonell, C., & Anvik, J. (2006) · NPARC · Cited by 131
The growing ubiquity of algorithms in society: implications, impacts and innovations
Olhede, S. C., & Wolfe, P. J. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0364 · Cited by 120 · cited 1×
“…are not unique to computerised decision-making systems, but with the growth of machine learning based systems they have become even more important (Bodo et al., 2017; Kroll et al., 2016; Nissenbaum, 1996; Olhede and Wolfe, 2018; Pasquale, 2015; Selbst and Barocas, 2018; Veale and Edwards, 2018).”
BackgroundMental Models and Causal Explanation: Judgements of Probable Cause and Explanatory Relevance
Hilton, D. J. (1996) · Thinking & Reasoning · doi:10.1080/135467896394447 · Cited by 117 · cited 4×
“s them irrelevant to the purposes of the explainer or recipient of an explanation (the ’explainee’). This type of selection is essential to reduce long causal chains to a cognitively manageable size (Hilton, 1996). In xAI, selection often takes the form of key features or evidence being emphasised in explanation inter…”
InfluentialThe Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification
Kim, B., Rudin, C. D., & Shah, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01161 · Cited by 117 · cited 2×
“Case-based explanation methods (Caruana et al., 1999; Kim et al., 2014) for non-cased based machine learning involve using the trained model as a distance metric to determine which cases in the training data set are most similar to the case or decision to be explained.”
BackgroundMethodHow Incorporating Feedback Mechanisms in a DSS Affects DSS Evaluations
Kayande, U., De Bruyn, A., Lilien, G. L., Rangaswamy, A., & van Bruggen, G. H. (2009) · Information Systems Research · doi:10.1287/isre.1080.0198 · Cited by 116 · cited 1×
“…machine learning models, it is perhaps most useful to always treat explanation generation as an interactive process, initially involving a mix of human and automated actors, at a minimum an inquirer (e.g. a developer, user) and the model or system (Kayande et al., 2009; Martens and Provost, 2013).”
BackgroundCounterfactuals and Explanation
Kment, B. (2006) · Mind · doi:10.1093/mind/fzl261 · Cited by 111 · cited 2×
“In recent decades, work in the philosophy of science and epis-temology has paid increasing attention to theories of contrastive explanations and counterfactual causality (e.g. Kment (2006); Lewis (1973); Ruben (2004); Woodward and Zalta (2003)).”
A new dialectical theory of explanation
Walton, D. (2004) · Philosophical Explorations · doi:10.1080/1386979032000186863 · Cited by 105 · cited 1×
“Walton (2004, 2007) takes a similar approach in proposing a dialectical theory of everyday explanations, which suggests that giving explanations involves not only transfer of information or causal claims, but also argumentative support for these claims.”
BackgroundHerfel, W. E., Krajewski, W. A., Niiniluoto, I., & Wójcicki, R. (1995) · BRILL eBooks · doi:10.1163/9789004457430 · Cited by 105
Explanation and categorization: How “why?” informs “what?”
Lombrozo, T. (2008) · Cognition · doi:10.1016/j.cognition.2008.10.007 · Cited by 100 · cited 1×
“Lombrozo (2009) for example demonstrated that the type of explanation provided (e.g. mechanistic, functional) can influence the recipient’s view of the importance of features in categorising a phenomenon (Poulin et al., 2006).”
BackgroundCase-based explanation of non-case-based learning methods.
Caruana, R., Kangarloo, H., Dionisio, J. D. N., Sinha, U., & Johnson, D. (1999) · PubMed · Cited by 94
Automation, Algorithms, and Politics| Auditing for Transparency in Content Personalization Systems
Mittelstadt, B. (2016) · International journal of communication · Cited by 92
Self-citationArgumentation-Based Inference and Decision Making--A Medical Perspective
Fox, J., Glasspool, D., Grecu, D., Modgil, S., South, M., & Patkar, V. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.102 · Cited by 86 · cited 2×
“These findings, that explanations are given via conversation and resemble argumentation, reveal a mechanistic link between explanation and justification as a type of discourse (Fox et al., 2007).”
Background(2017) · arXiv (Cornell University) · Cited by 86
Interpretability via Model Extraction
Bastani, O., Kim, C., & Bastani, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09773 · Cited by 85 · cited 1×
“A trade-off inherently occurs between the insightfulness of the approximated model, the simplicity of the presented function, and the size of the domain to which is applies and remains valid (Bastani et al., 2017; Lakkaraju et al., 2017).”
BackgroundAttribution in conversational context: Effect of mutual knowledge on explanation‐giving
Slugoski, B. R., Lalljee, M., Lamb, R., & Ginsburg, G. P. (1993) · European Journal of Social Psychology · doi:10.1002/ejsp.2420230302 · Cited by 79 · cited 1×
“They should further be relevant to the question asked by the recipient (epistemic relevance; (Miller, 2017), or what Slugoski et al .”
Gale, G. (2003) · Cambridge University Press eBooks · doi:10.1017/chol9780521591041.052 · Cited by 77
Interpretability in Machine Learning – Principles and Practice
Lisboa, P. J. G. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-03200-9_2 · Cited by 72 · cited 1×
“‘Interpretability’ refers to the degree of human comprehensibility of a given ‘black-box’ model or decision (Lisboa, 2013; Miller, 2017).”
BackgroundWhen similarity and causality compete in category-based property generalization
Rehder, B. (2006) · Memory & Cognition · doi:10.3758/bf03193382 · Cited by 71
Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations
Tamagnini, P., Krause, J., Dasgupta, A., & Bertini, E. (2017) · Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/3077257.3077260 · Cited by 69 · cited 1×
“Visualisation techniques can visually demonstrate the relative influence of features or particular pixels (e.g. in the case of an image classifier), or provide an interface for users to explore textual or visual explanations (Poulin et al., 2006; Tamagnini et al., 2017).”
Explaining in conversation: Towards an argument model
Antaki, C., & Leudar, I. (1992) · European Journal of Social Psychology · doi:10.1002/ejsp.2420220206 · Cited by 61 · cited 1×
“Conversational explanations are used to both offer causes of an event and back claims as to the truth or relevance of these causes (Antaki and Leudar, 1992).”
BackgroundHow to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54 · cited 2×
“These simplified models approximate the true criteria used to make decisions (e.g. Baehrens et al. (2010); Ribeiro et al. (2016); Selvaraju et al. (2016); Simonyan et al. (2013)).”
BackgroundMethodComprehensible Credit Scoring Models Using Rule Extraction from Support Vector MachinesPreprint
Martens, D., Baesens, B., Gestel, T. V., et al. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.878283 · Cited by 48
Interpretable Clustering via Discriminative Rectangle Mixture Model
Chen, J., Chang, Y., Hobbs, B. D., Castaldi, P. J., Cho, M. H., Silverman, E. K., & Dy, J. (2016) · Proceedings · doi:10.1109/icdm.2016.0097 · Cited by 43 · cited 2×
“level. Historically, much work has focused on global approximations of models (e.g. Craven and Shavlik (1996); Martens et al. (2007); Sanchez et al. (2015), including approaches based on clustering (Chen et al., 2016), integer programming (Zeng et al., 2017), and rule lists (Wang and Rudin, 2015)). In contrast, local…”
Dialogical Models of Explanation.
Walton, D. (2007) · Cited by 35
Berendt, B., & Preibusch, S. (2017) · Big Data · doi:10.1089/big.2016.0055 · Cited by 32 · cited 2×
“and Koops, 2010; Zarsky, 2013). As these purposes suggest, explanations can be offered to expert developers, professionals working in tandem with a system (e.g. expert labelers of training cases; see Berendt and Preibusch (2017) for an overview of human actors involved in algorithmic decision-making), and to individua…”
BackgroundTowards Extracting Faithful and Descriptive Representations of Latent Variable Models
Rockt, T., Riedel, S., & Singh, S. (2015) · UCL Discovery (University College London) · Cited by 31
High Dimensional Spaces, Deep Learning and Adversarial ExamplesPreprint
Dube, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.00634 · Cited by 30
Veale, M., & Edwards, L. (2017) · LawArXiv (OSF Preprints) · doi:10.31228/osf.io/y25ag · Cited by 29
Masulli, F., Pasi, G., & Yager, R. R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-03200-9 · Cited by 27
Interpretable & Explorable Approximations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01154 · Cited by 23 · cited 1×
“A trade-off inherently occurs between the insightfulness of the approximated model, the simplicity of the presented function, and the size of the domain to which is applies and remains valid (Bastani et al., 2017; Lakkaraju et al., 2017).”
BackgroundDo Social Norms Influence Causal Inferences
Samland, J., & Waldmann, M. R. (2014) · eScholarship (California Digital Library) · Cited by 16
Birchall, C. (2016) · International journal of communication · Cited by 14
Auditing Black-box Models by Obscuring FeaturesPreprint
Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2016) · arXiv (Cornell University) · Cited by 11
Explaining Data-Driven Document Classifications
Martens, D., & Provost, F. J. (2013) · Faculty Digital Archive (New York University Florence) · Cited by 11
Causal and constitutive explanation comparedPreprint
Ylikoski, P. (2017) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/6dajt · Cited by 5
McClure, J., Sutton, R. M., & Hilton, D. (2003) · Kent Academic Repository (University of Kent) · Cited by 1
Research works printed but not linked: 7
- Or Biran and Kathleen McKeown. 2014. Justification narratives for individual classifications. In Proceedings of the AutoML workshop at ICML , Vol. 2014.
- J. Habermas. 1984. The Theory of Communicative Action: Volume 1: Reason and the Rationalization of Society . Beacon, Boston.
- Joshua A. Kroll, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- David Lewis. 1973. Counterfactuals . Blackwell, Oxford.
- Zachary C. Lipton. 2016. The Mythos of Model Interpretability. arXiv:1606.03490 [cs, stat] (10 6 2016). http://arxiv.org/abs/1606. 03490 arXiv: 1606.03490. · arXiv
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry (2014), 1-23.
- Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology forthcoming (2018).
Cited by
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
The Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911
“Enormous technical efforts are undertaken to meet ethical targets in the fields of accountability and explainable AI (Mittelstadt et al. 2019), fairness and discrimination aware data mining (Gebru et al.”
BackgroundThe Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907
“Enormous technical efforts are undertaken to meet ethical targets in the fields of accountability and explainable AI (Mittelstadt et al. 2019), fairness and discrimination aware data mining (Gebru et al.”
BackgroundBias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043
“Given that a decision will often not be reasoned (hence the reasons will be unknown), it will be difficult to establish that it was made on the basis of a biased decision-making process (Mittelstadt et al., 2019).”
Explainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040
“Concerning explanations, it has also been argued that there is a gap of expectations between ML and so-called explanation sciences such as law, cognitive science, philosophy, and the social sciences [28].”
Background
Metadata
- Authors
- Mittelstadt, B., Russell, C., & Wachter, S.
- Affiliations
- University of Oxford (GB)
- University of Surrey (GB)
- Venue
- Proceedings of the Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2019-01-29
- Pages
- 279-288
- OpenAlex
- W2898694742
- Semantic Scholar
- cf4bbe1e39c6f0f88aa69da2a95125b47de0ec7b
- Cited by
- 794
- References
- 100
- Citations by year
- 2026: 79 · 2025: 99 · 2024: 110 · 2023: 117 · 2022: 115 · 2021: 125 · 2020: 107 · 2019: 40
- FWCI
- 37.01
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Surrey Research Insight Open Access (The University of Surrey) (submittedVersion)
- SSRN Electronic Journal (submittedVersion)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/N510129/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Philosophy of Biology and Scientific Explanation
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainable AI, interpretability, philosophy of explanation, contrastive explanations, what-if questions, teaching model
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.