Problems with Shapley-value-based explanations as feature importance measures

“Many recently proposed “local” methods (Ribeiro et al., 2016; Lundberg & Lee, 2017; Lundberg et al., 2018) define a value function vf,x : 2 → R that depends on a specific data instance x to explain how each feature contributes to the output of the function on this instance.”

“Since the notion of the sum of two games is not especially meaningful, the Additivity axiom has been described by game theorists as “mathematically convenient” and “not nearly so innocent as the other two” Osborne and Rubinstein [1994]. The choice to constrain the value to be unique in this way has implications for wh…”

“Instead, Miller (2019) argues that humans are selective about explanations: certain contrasts are more meaningful than others.”

“This perspective is closer in spirit to the “humangrounded metrics” that Doshi-Velez & Kim (2017) describe in comparison with the “functionally-grounded evaluation” of the previous section.”

“TreeSHAP (Lundberg et al., 2018), an efficient algorithm for calculating SHAP values on additive tree-based models such as random forests and gradient boosting machines, can estimate EXS̄ |XS [f(xS ,XS̄)] by observing what proportion”

“Slack et al. (2020) demonstrate how to exploit this sensitivity by devising models which illegally discriminate on some protected feature for in-distribution samples, but exhibit different behavior on the out-of-distribution samples used by KernelSHAP so as to simulate “fairness” in the resulting explanations.”

“A variety of methods to assign feature influence using the Shapley value have recently been developed (Lipovetsky & Conklin, 2001; Štrumbelj & Kononenko, 2014; Lundberg et al., 2018; Datta et al., 2016b; Merrick & Taly, 2019; Frye et al., 2019; Aas et al., 2019).”

“Shapley-value-based explanations are primarily used for purposes of normative evaluation: deciding whether a model’s behavior is acceptable (Bhatt et al., 2020).”

“Kaur et al. (2020) conducted a contextual inquiry and survey of data scientists to observe their interpretation of interpretability tools including the SHAP Python package.”

“Ustun et al. (2019) further specify a way to model this problem by searching for changes within characteristics which are actually mutable; they call this the “actionable recourse” problem (with a corresponding analysis by Venkatasubramanian & Alfano (2020)).”

“We use the framework set out by Selbst and Barocas [2018], who argue that there are three general motivations behind the call for explanations in AI.”

“He attributes this insight to work by Lipton (1990).”

“Kruskal (1987) and Lipovetsky & Conklin (2001) proposed using the Shapley value to analyze global feature importance in linear regression by using the value function vf (S) to represent theR(2) of a linear model f built on predictors S, to decompose the variance explained additively between the features. Owen & Prieur…”

“If we are willing to consider intervention distributions (Section 2.2.1), then the framework provided by Merrick & Taly (2019) provides a slightly different contrastive explanation: in their setting, the Shapley value assignment can be thought of as a set of answers to the question, “Why f(x)…”

“ight be able to alter their own situation to approximate q. This setup has been formalized as the “counterfactual explanation” problem byWachter et al.(2017) (with an analysis of hidden assumptions byBarocas et al. (2020)).Ustun et al.(2019) further specify a way to model this problem by searching for changes within c…”

“This setup has been formalized as the “counterfactual explanation” problem by Wachter et al. (2017) (with an analysis of hidden assumptions by Barocas et al.”

“Method vf,x(S) v̂f,x(S) KernelSHAP, Shapley sampling values EXS̄ |XS [f(xS ,XS̄)] ED[f(xS ,XS̄)] QII, FAE, Interventional TreeSHAP ED[f(xS ,XS̄)] ED[f(xS ,XS̄)] Conditional TreeSHAP, Frye et al. (2019), Aas et al. (2019) EXS̄ |XS [f(xS ,XS̄)] EXS̄ |XS [f(xS ,XS̄)]”

“1), then the framework provided by Merrick and Taly [2019] provides a slightly different contrastive explanation: in their setting, the Shapley value assignment can be thought of as a set of answers to the question, “Why f(x) rather than f(r)?”, where r is chosen from the reference distribution.”

“Slack et al. (2020) demonstrate how to exploit this sensitivity by devising models which illegally discriminate on some protected feature for in-distribution samples, but exhibit different behavior on the out-of-distribution samples used by KernelSHAP so as to simulate “fairness” in the resulting explanations.”

“The change from unconditional to conditional permutation changes the respective interpretation method [78,7], or, in worst case, can break it [57,113,62].”

“More recently there has been a lot of work on the tradeoffs of variants of the original S HAP explanations, e.g., Sundararajan and Najmi (2020), Kumar et al. (2020), Janzing, Minorics, and Bloebaum (2020), Merrick and Taly (2020), and Aas, Jullum, and Løland (2019).”

“Having two popular types of Shapley value explanations has been cited as a weakness [22], but this issue is not unique to Shapley values; it is encountered by a large number of model explanation methods [26], and it is fundamental to understanding feature importance with correlated or statistically…”

“Figure 13 shows more examples of MNIST explanations using the combination of the conditional distribution and Shapley value (i.e., LossSHAP).”