Problems with Shapley-value-based explanations as feature importance measures

“Many recently proposed “local” methods (Ribeiro et al., 2016; Lundberg & Lee, 2017; Lundberg et al., 2018) define a value function vf,x : 2 → R that depends on a specific data instance x to explain how each feature contributes to the output of the function on this instance.”

“Since the notion of the sum of two games is not especially meaningful, the Additivity axiom has been described by game theorists as “mathematically convenient” and “not nearly so innocent as the other two” Osborne and Rubinstein [1994]. The choice to constrain the value to be unique in this way has implications for wh…”

“Instead, Miller (2019) argues that humans are selective about explanations: certain contrasts are more meaningful than others.”

“This perspective is closer in spirit to the “humangrounded metrics” that Doshi-Velez & Kim (2017) describe in comparison with the “functionally-grounded evaluation” of the previous section.”

“TreeSHAP (Lundberg et al., 2018), an efficient algorithm for calculating SHAP values on additive tree-based models such as random forests and gradient boosting machines, can estimate EXS̄ |XS [f(xS ,XS̄)] by observing what proportion”

“Slack et al. (2020) demonstrate how to exploit this sensitivity by devising models which illegally discriminate on some protected feature for in-distribution samples, but exhibit different behavior on the out-of-distribution samples used by KernelSHAP so as to simulate “fairness” in the resulting explanations.”

“A variety of methods to assign feature influence using the Shapley value have recently been developed (Lipovetsky & Conklin, 2001; Štrumbelj & Kononenko, 2014; Lundberg et al., 2018; Datta et al., 2016b; Merrick & Taly, 2019; Frye et al., 2019; Aas et al., 2019).”

“Shapley-value-based explanations are primarily used for purposes of normative evaluation: deciding whether a model’s behavior is acceptable (Bhatt et al., 2020).”

“Kaur et al. (2020) conducted a contextual inquiry and survey of data scientists to observe their interpretation of interpretability tools including the SHAP Python package.”

“Ustun et al. (2019) further specify a way to model this problem by searching for changes within characteristics which are actually mutable; they call this the “actionable recourse” problem (with a corresponding analysis by Venkatasubramanian & Alfano (2020)).”

“We use the framework set out by Selbst and Barocas [2018], who argue that there are three general motivations behind the call for explanations in AI.”

“He attributes this insight to work by Lipton (1990).”

“Kruskal (1987) and Lipovetsky & Conklin (2001) proposed using the Shapley value to analyze global feature importance in linear regression by using the value function vf (S) to represent theR(2) of a linear model f built on predictors S, to decompose the variance explained additively between the features. Owen & Prieur…”

“If we are willing to consider intervention distributions (Section 2.2.1), then the framework provided by Merrick & Taly (2019) provides a slightly different contrastive explanation: in their setting, the Shapley value assignment can be thought of as a set of answers to the question, “Why f(x)…”

“ight be able to alter their own situation to approximate q. This setup has been formalized as the “counterfactual explanation” problem byWachter et al.(2017) (with an analysis of hidden assumptions byBarocas et al. (2020)).Ustun et al.(2019) further specify a way to model this problem by searching for changes within c…”

“This setup has been formalized as the “counterfactual explanation” problem by Wachter et al. (2017) (with an analysis of hidden assumptions by Barocas et al.”

“Method vf,x(S) v̂f,x(S) KernelSHAP, Shapley sampling values EXS̄ |XS [f(xS ,XS̄)] ED[f(xS ,XS̄)] QII, FAE, Interventional TreeSHAP ED[f(xS ,XS̄)] ED[f(xS ,XS̄)] Conditional TreeSHAP, Frye et al. (2019), Aas et al. (2019) EXS̄ |XS [f(xS ,XS̄)] EXS̄ |XS [f(xS ,XS̄)]”

“1), then the framework provided by Merrick and Taly [2019] provides a slightly different contrastive explanation: in their setting, the Shapley value assignment can be thought of as a set of answers to the question, “Why f(x) rather than f(r)?”, where r is chosen from the reference distribution.”

“Slack et al. (2020) demonstrate how to exploit this sensitivity by devising models which illegally discriminate on some protected feature for in-distribution samples, but exhibit different behavior on the out-of-distribution samples used by KernelSHAP so as to simulate “fairness” in the resulting explanations.”

“The change from unconditional to conditional permutation changes the respective interpretation method [78,7], or, in worst case, can break it [57,113,62].”

“More recently there has been a lot of work on the tradeoffs of variants of the original S HAP explanations, e.g., Sundararajan and Najmi (2020), Kumar et al. (2020), Janzing, Minorics, and Bloebaum (2020), Merrick and Taly (2020), and Aas, Jullum, and Løland (2019).”

“Having two popular types of Shapley value explanations has been cited as a weakness [22], but this issue is not unique to Shapley values; it is encountered by a large number of model explanation methods [26], and it is fundamental to understanding feature importance with correlated or statistically…”

“Figure 13 shows more examples of MNIST explanations using the combination of the conditional distribution and Shapley value (i.e., LossSHAP).”

“There have, however, been a number of recent criticisms of the Shapley value in ML [37]–[39], suggesting to us that the Shapley value and its alternatives may be further developed considerably over the coming years.”

“The SHAP value 𝜙 𝑖 for a feature 𝑖 , and calculate its average impact across every combination of features, represented in Eq.”

“It is important to better understand what these limitations are [11].”

“Understanding the meaning and implications of Shapley values in complex models can be challenging for non-experts [55].”

“While recent work has gone so far as to suggest that having two separate approaches presents an irreconcilable problem with using Shapley values for feature attribution (Kumar et al., 2020), in this paper, we argue that rather than representing some critical flaw in using the Shapley value for feature attribution, eac…”

“SHAP is valued for generating both local and global explanations and for its applicability to tree-based models such as XGBoost and random forests [106,107].”

“I. E. Kumar et al. (2020) criticize Shapley-value-based explanations, such as described above, for their reliance on the additivity axiom (Hart, 1989) and lack of human-groundedness and contrastiveness.”

“[25] I E Kumar, S Venkatasubramanian, C Scheidegger, and S Friedler.”

“These models provide a way to shed some light on black box decision making but importance must not be conflated with actual explanations as they can often be altered without affecting the output prediction [126], [127].”

“One of the downsides of feature importance is that it may lead to confirmation bias, both in the model derivation phase [37] and in the post-hoc interpretation of the output.”

“However, these approaches have been also criticized since they can yield misleading information about the relative importance of features for predictions [149, 150].”

“Kumar et al. [ 109 ] criticise Shapley-value-based explanations, such as described in [ 21,22,102 ] , for their reliance on the additivity axiom [ 103 ] and lack of human-groundedness and contrastiveness.”

“Thus, this kind of approach requires further modelling of how the features are correlated [94].”

“However, these methods can produce incorrect and misleading explanations primarily because they focus on the correlation between the input and output of algorithms as opposed to their causal relationship [36, 44, 24, 33, 62, 4].”

“…study, we consider the questions about the ‘‘human-centric’’ lack of significance in explainability and the impacts on comprehension’s mental models of users very pertinent – both as a consequence of the lack of explanations completeness and the information overload generated by XAI methods [218].”

“Shapley values, however, are limited in their ability to provide useful causal explanations [159].”

“Lastly, there is a growing body of statistical work contesting the use of SHAP values for feature explanations 24 – 26 .”

“89,90 To this end, collaboration across fi elds between pharmacometricians, informaticians, clinicians, clinical phar-macologists, and disease and domain experts is imperative for understanding and deciphering model results and predictions, as well as to ensure the correct implementation of AI into…”

“Anchors, on the other hand, do not maintain invariance across various architectures [57].”

“Some authors speak of conditional VI as an “improvement” over marginal measures (Aas et al., 2021; Frye et al., 2020; Kumar et al., 2020), perhaps because the former is generally more complex or closer to nature.”

“Moreover, while SHAP values have been frequently used by the machine learning community in recent years, their adoption is not unanimous, and some limitations on their use as explanatory tools have been pointed out 37 .”

“However, the computational complexity of Shapley Value grows factorially with respect to the number of participating players [20].”

“This is an especially compelling reason to prefer the core over the Shapley value: since the stakeholders involved with machine learning are often people, it is imperative to employ a solution concept that is consistent with their behavior and intuition (Bhatt et al. 2019) (Kumar et al. 2020).”

“…for its detailed contribution analysis, struggles with computational efficiency, especially outside of tree-based models [31], and also grapples with the issue of lack of precision mostly due to their restrictive hypothesis of local linearity and feature independence as with LIME [26], [32].”

“A number of other issues have been identified in the literature (Hooker et al., 2019; Kindermans et al., 2019; Adebayo et al., 2020; Kumar et al., 2020; Sixt et al., 2020).”

“Shapley Values have been explored in various tasks with human decision makers, a topic about which there is much debate (Kumar et al., 2020).”

“These could include discrete features in the input ( textual highlights ; Lei et al., 2016; see Fig.”

“, some ML models tend to leverage out-of-distribution artifacts and thus provide nonsensical explanations [63], which must be avoided.”

“However, ambiguities regarding the underlying assumptions of the method [Kumar et al., 2020] and the recent proliferation of mutually incompatible implementations [Sundararajan and Najmi, 2019; Merrick and Taly, 2020] have complicated this picture.”

“Various methods have been suggested to measure the feature importance in recent years [62–64].”

“However, it was shown that the SV is limited when explaining complex decision systems, and feature interactions , i.e. the joint contributions of features’ groups, are required to understand such processes [20, 29, 48, 49, 61, 66, 78, 81, 82, 86].”

“Although the Shapley value provides a principled framework in game theory, one critical issue is that the economic notion of the Shapley axioms is not intuitively applicable to the attribution problem [Kumar et al., 2020, Rozemberczki et al., 2022].”

“However, the fair division axioms used in the Shapley value have not been statistically examined, and it raises a fundamental question about the appropriateness of these axioms in machine learning problems (Kumar et al., 2020; Sim et al., 2022; Rozemberczki et al., 2022).”

“SHAP values have desired mathematical properties as a solution to game theory problems [50].”

“LIME, Shapley Values (SHAP) [76] and permutation feature significance [85] are a few popular XAI approaches independent of the model.”

“We note however, that concerns about using Shapley value have been raised by Kumar et al. (2020). Mixing and matching the components of xt and xb presents some problems.”

“Understanding feature importance is essential to comprehending machine learning models’ internal mechanisms [25].”

“On the one hand, Kumar et al. [77] questioned the suitability of Shapley values to explain model predictions to a human.”

“See also Mase, Owen and Seiler (2022) and Kumar et al. (2020).”

“In contrast, a number of authors have reported pitfalls with the use of SHAP and Shapley values as a measure of feature importance [105, 60, 95, 74, 37, 104, 78, 2, 101, 59, 20].”

“We remark that SHAP-scores have attracted the attention of the ML community and have found several applications and extensions (Rathi, 2019; Fidel et al., 2020; Bertossi et al., 2020; Merrick and Taly, 2020; Takeishi and Kawahara, 2020; Covert and Lee, 2021; Kumar et al., 2020).”

“It has been noted that this requires the DNN model to be evaluated in an region of the input space for which it has not received any training data and for which its accuracy was never evaluated (Sundararajan et al., 2017; Kumar et al., 2020).”

“However, this versatility comes at the cost of interpretability as explainable artificial intelligence (XAI) methods still face major challenges (Kumar et al., 2020; Molnar et al., 2021).”

“This quote was phrased only by one respondent, and indicates that SHAP-stories can address some limitations of SHAP, namely that SHAP attributes significance to all features [2] and does not have an established protocol for interpretation [37].”

“There are also critical aspects of SVs in this context as for example studied in [27].”

“Limitations of the SV due to correlations and interactions have been widely studied by Slack et al. (2020), Sundararajan and Najmi (2020), and Kumar et al. (2020, 2021).”

“ible that the resulting scores are counterintuitive. On the other hand, some of these approaches, such as SHAP, come with some nice theoretical properties (although in practice they might be violated [50, 57]). ‹ Model simplification comes with the immediate advantage and flexibility of allowing to approximate an opaque…”

“While SHAP is popular and intuitive, it suffers from drawbacks such as the emergence of mathematical problems, further complexities that arise to solve these mathematical problems and the ill-suited game-theoretic framework for quantifying the feature importance [63].”

“The results from [122] 978 suggest that the mathematical correctness that underpins 979 SHAP is not sufficient alone; it should also be aligned with 980 the specific use case and the human-centric understanding of 981 SHAP’s quality of explanations.”

“The authors of SHAP recommend using interventional distributions if we want the explanation to be “true to the model”, and observational distributions if we want the explanation to be “true to the data” [14].”

“This is a common point of criticism, as the model needs to some extent extrapolate away from the original data manifold (Hooker and Mentch, 2019; Kumar et al., 2020).”

“There also exist a lot of exciting papers focusing on the designs of SHAP/TreeSHAP implementations [32, 27, 12, 33], as well as the theoretical justification of SHAP values [11, 17, 30, 1, 7, 23, 5].”

“…analysis, it struggles with computational efficiency and, like LIME, may generate explanations that omit relevant features or create unrealistic scenarios, raising concerns about reliability (Hooshyar & Yang, 2024b; I. E. Kumar et al., 2020; Slack et al., 2020; Van den Broeck et al., 2022).”

“Forlocalexplanations,thetwomaintypesare marginal and conditional Shapley values 1 , and there is an ongoing debate about when to use them (Chen et al. 2020; Kumar et al. 2020; Chen et al. 2022).”

“About the methodological foundations, they seem to be not fully understood by many users, potentially leading to misuses of the tool [43].”

“While post-hoc methods are shown to be effective for explaining the decisions of complex black-box models, recent research has argued that these explanations can be misleading, and can complicate model understanding even more [21], [22].”

“The SHAP value can be used to represent each feature’s marginal contribution value and measure the feature’s importance [63].”

“There are indeed significant amount of academic papers criticizing the blind use of model-agnostic explanation methods [8, 13, 14] (to name a few).”

“Other criticisms of using methods based on Shapley values include defining “games" in the context of a learning task and the inability of these methods to capture the indirect influences of features on the target label [40].”

“, 2020; Littlefield et al., 2021; Christodoulou et al., 2019; Gravesteijn et al., 2020; Nusinovici et al., 2020; Lynam et al., 2020). Sophisticated methods can only improve upon linear main-effects models by capturing more nuanced non-linearities and interactions. Almost by definition, these effects are of smaller siz…”

“However, interpreting these contributions as counterfactual statements may not be a sensible way to summarize information [11] sinceφ j is the average marginal contribution of feature j .”

“Furthermore, the Shapley value intuitively aligns more with what is important in an additive setting, as mentioned in previous works [20].”

“Both have their benefits and drawbacks depending on the concrete application (Chen et al. 2020; Kumar et al. 2020; Miroshnikov, Kotsiopoulos, and Kannan 2021).”

“At last, we would like to highlight that although SHAP is popular, as an explanation tool it contains several limitations, specifically Kumar et al. [48] have shown that SHAP violates several expected properties of explanations from a human-centric perspective.”

“21,22 Therefore, it is still essential to consider explainability results in the context of prior similar studies and overall established medical knowledge.”

“Shapley Value [25, 34, 35] originated from cooperative game theory and is the best way to distribute benefits fairly by considering the contributions of various agents.”

“In particular, this line of research sheds light on the limitations of some technical solutions for aiding user understanding, or worse, on their potential for deception [21, 23, 40].”

“Nonetheless, it has several problems in the measurement of feature importance (34).”

“Note that by restricting our scope in this manner, we potentially exclude much of the important post-hoc work done by researchers [e.g. 31, 72, 112].”

“1 Dropping individual features can also lead to out-of-distribution samples, further limiting the effectiveness of meth-ods and metrics that rely on simulating feature removal (Kumar et al., 2020; Sundararajan and Najmi, 2020).”