Problems with Shapley-value-based explanations as feature importance measures

“Many recently proposed “local” methods (Ribeiro et al., 2016; Lundberg & Lee, 2017; Lundberg et al., 2018) define a value function vf,x : 2 → R that depends on a specific data instance x to explain how each feature contributes to the output of the function on this instance.”

“Since the notion of the sum of two games is not especially meaningful, the Additivity axiom has been described by game theorists as “mathematically convenient” and “not nearly so innocent as the other two” Osborne and Rubinstein [1994]. The choice to constrain the value to be unique in this way has implications for wh…”

“Instead, Miller (2019) argues that humans are selective about explanations: certain contrasts are more meaningful than others.”

“This perspective is closer in spirit to the “humangrounded metrics” that Doshi-Velez & Kim (2017) describe in comparison with the “functionally-grounded evaluation” of the previous section.”

“TreeSHAP (Lundberg et al., 2018), an efficient algorithm for calculating SHAP values on additive tree-based models such as random forests and gradient boosting machines, can estimate EXS̄ |XS [f(xS ,XS̄)] by observing what proportion”

“…prediction errors from interpretation errors on unlabeled data, and requires to manage out-of-distribution artifacts impacting the produced interpretations, which is also the case of models architecture and hyperparameters choice (Dombrowski et al., 2019; Kumar et al., 2020; Slack et al., 2020).”

“In the machine learning context, arbitrariness of Shapley values due to interactions and correlations has been discussed in Kumar et al. (2020), Slack et al. (2020), Sundararajan and Najmi (2020), and possible solutions have been proposed in Zhang et al. (2021), Kumar et al. (2021), Harris et al.…”

“For Shapley values, this is captured in the recent discussion on interventional as compared to observational Shapley values [23, 28, 48].”

“Furthermore, an explainability technique represents a form of model in its own right, and the process of modeling a model brings its own difficulties (see e.g., Rudin (2019); Kumar et al. (2020)).”

“Many explanatory techniques are less accurate in the presence of correlated input features (Altmann et al., 2019; Kumar et al., 2020) as post hoc explanation methods do not often account for dependencies between features.”

“Like many other explanation methods, TIME may perform out-of-distribution perturbations (Kumar et al. 2020), leading to an inflation of importance scores for correlated features.”

“However, in many application domains, the SV accounting solely for individual entities such as features or data points does not suffice (Kumar et al., 2020; 2021; Slack et al., 2020; Sundararajan Proceedings of the 41 st International Conference on Machine Learning , Vienna, Austria.”

“The counterfactual reasoning use case is introduced by Kumar et al. [25], where the authors discuss why SHAP feature importance scores [29] do not explicitly attempt to provide guidance for the use case.”

“Kumar et al. (2020) point out some difficulties in formulating a variable importance problem as a game suitable for use in Shapley value.”

“There are some reasons, including out-of distribution sampling during Shapley value approximation and not taking into account feature correlation, why Shapley values should be used with caution for black-box model interpretability [68].”

“But this could be misleading because the perturbations change the nature of the instance, and can even create Out-of-Distribution (OOD) points [40].”

“We also note that the use of Shapley value in other contexts, such as explaining a prediction is criticized by many researchers in different ways [9]; it was pointed out that additivity axiom need not hold in such applications of Shapley value.”

“The second, and most common, general paradigm is to rank features based on MI [38], or Shapley values [54], before selecting a predetermined number of the best performing features.”

“However, recent work by Kumar et al. (Kumar et al. 2020) exposes some mathematical issues with SHAP and concludes that this framework is ill-suited as a general solution to quantifying feature importance.”

“)e function v has the following definitions: if S is a permutation (coalition) of all factors; then v(S) describes the total impact of all factors in coalition S [35].”

“We test this approach across a suite of ML models based on decision trees, connectionism, and kernels, as a possible solution to investigate the processes in a pilot pit lake in the Athabasca Oil Sands (AOS) region of Western Canada.”

“, 2010] or local and instance-based methods such as Shapley values [Kumar et al., 2020], are reported in the supplement (section I).”

“We note that SHAP values contain inherent flaws (37) in approximating the impact of variables on predictions, and certainly do not aim to estimate causal effects.”

“The authors in [19] show that mathematical problems arise when Shapley values are used for feature importance and that the mitigating solutions necessarily induce further issues.”

“Nevertheless, it should be known that the optimum case for feature attribution is when the features that are being perturbed are independ - ent at the outset (Kumar et al. 2020; Chen et al. 2020), and for the model, the additive model is the best (Kumar et al. 2020).”

“This weakness can lead to recourse recommendations that are disconnected from real-world actions [2, 17], like recommending to an individual that they should “lower their age“ to receive a different outcome.”

“Kumar et al. (2020) argue that many of the above methods have problems that make Shapley values an awkward fit for the problem of machine learning interpretability.”

“This requires CSE to be interpretable to some extent (or at least explainable through the use of LIME 37 or SHAP 38,39 ).”

“7 can have different contributions for different predictions, which shows that interpreting Shapley values requires caution [Kumar et al., 2020].”

“The main drawback is that we assume feature independence (only for the calculation of v ( S )) which can produce out-of-distribution samples.”

“Oppositely, Kumar et al. (2020) discuss the problems of using Shapley value for model explanation.”

“There are limitations to the Shapley method which have been discussed elsewhere (Kumar et al., 2020), and the notion that the human-interpretable Shapley model suf fi ciently explains our model suggests that a simpler model may be adequate to begin with, even if the simpler model is harder to…”

“SHAP assumes that all states are independent, which they might not be in most robotics applications [31].”

“Moreover, Shapley values should be able to explain their re-sults in a way that aligns with human-centric goals of explainability [63].”

“In ML, limitations of feature attribution scores have been discussed in Wright et al. (2016), Sundararajan and Najmi (2020), and Kumar et al. (2020, 2021) among others.”

“It is what Kumar et al. (2020) call a conditional method because it requires specification of a conditional distribution on the features.”

“…a plausibility assessment is to use explanations such as feature importance with axiomatic properties for Correctness and Completeness such as SHAP, although the actual value of such properties is part of an ongoing discussion (Kumar et al. 2020; Merrick and Taly 2020; Sundararajan and Najmi 2020).”

“…and personhood” (i.e., the ability to make a decision or select amongst a set of necessary alternatives), (2) may not appropriately educate stakeholders on how to form a critical, nuanced interpretation of findings, and (3) does not facilitate debate on whether such findings are justified (64–66).”

“Important criticisms of Shapley values as feature importance measures have been raised elsewhere (Kumar et al., 2020).”

“Among XAI methods, the Shapley Values or the SHapley Additive exPlanations (SHAP) Framework has been recently employed also in the corporate finance and banking literature (Kumar et al., 2020; Fryer et al., 2021; Bitetto et al., 2023; Shalit, 2023; Basher and Sadorsky, 2025).”

“26,27 In our study, results from the Shapley feature importance analysis for TabNet did not exactly mirror the TabNet model-specific feature importance results; race/ethnicity as a feature was comparatively de-emphasized, whereas visual acuity and certain RNFL features were more important in the…”

“Calculating the Shapley value is costly, particularly when dealing with a large number of agents (Kumar et al. 2020).”

“…it is exactly the other way around since lower numbers represent exposure closer to the original one. the challenge of interpreting SHAP values, which may not always align with human expectations [19], potentially lacking contrastive-ness [27], and it can be susceptible to adversarial attacks [46].”

“Calculating SHAP values satisfies key fairness properties while accounting for all possible feature interactions [58].”

“…it is exactly the other way around since lower numbers represent exposure closer to the original one. the challenge of interpreting SHAP values, which may not always align with human expectations [19], potentially lacking contrastive-ness [27], and it can be susceptible to adversarial attacks [46].”

“However, for instance [13] recently pointed out several mathematical and human-centric issues with the use of Shapley value for XAI purposes.”

“…been shown to have limitations and challenges, in terms of the assumptions of feature independence, that of feature linearity and output, data distribution, and several others 28 , they do appear to be a mathematically more objective measure than the natural language rationale provided by LLMs.”

“The Shapley value is a metric in game theory to evaluate the relative contribution of each input predictor feature to the overall prediction [24].”

“Kumar et al. (2020) and Hancox-Li & Kumar (2021), meanwhile, investigate SHAP (Lundberg & Lee, 2017), and find problems from both technical and philosophical perspectives.”

“In terms of baseline values of Shapley values, most studies (Covert et al., 2020a; Merrick and Taly, 2020; Sundararajan and Najmi, 2020; Kumar et al., 2020) Unlike previous studies, we rethink and formulate baseline values from the perspective of game-theoretic causality.”

“However, Kumar et al. (2020) argue that these explanations are not always easy to understand and do not match human expectations, i.e., we usually want explanations that make sense to us as humans.”

“The main advantage of interventional algorithms is that they are computationally less expensive, although, in contrast, they require the assumption of feature independence and use samples outside the distribution, which may result in unrealistic observations [33].”

“However, these approaches may be insufficiently reliable, robust, or hard to interpret themselves, which has motivated the study of inherently explainable methods [2, 10, 17, 28].”

“This may be explained by the fact that perturbation-based frameworks often generate perturbed examples that lie out of the data distribution [136].”

“The Shapley value [23, 24, 25], used to assess party contributions in federated learning [26, 27, 28], has significant computational costs, rendering it unsuitable for guiding feature splitting.”

“Investigating further linkages between AI and the semantic web or statistics, which would allow for using a variety of well-researched techniques, is another promising option to overcome robustness difficulties (Kumar et al. 2020 ).”

“Data scientists often have trouble understanding the exact relationship between the two models (Kaur et al. 2020) or explaining this relationship to stakeholders (Passi and Jackson 2018), which is ultimately undermining the value of both models and the whole enterprise (Kumar et al. 2020).”

“We use sample mean for Shapley value to avoid massive computations (Shapley for short), and drop the weights in Shapley value (Shapley-w for short) due to the intuition that not all permutations should exist in natural language [12, 21].”

“Some previous work have brought Shapley value into deep learning explanation methods [18, 5, 14].”

“[38] has shown that different explanation methods often do not agree with each other.”

“…1992; Smets, 2005); in addition, we may also look for the element of the core maximising the entropy (Abellán & Moral, 2003; Jaffray, 1995) or establish procedures for assigning relevance degrees to the different features in machine learning problems (Kumar et al., 2020; Lundberg & Lee, 2017).”

“A few works have suggested that people may overlook the properties of the Shapley value and have incorrect expectations of it [33, 18].”

“This could potentially result in unrealistic observations [93].”

“In the inference stage, these stakeholders can interact with the model by providing different sets of predictors and examining how predictions, performance, and predictor importance change accordingly.”

“However, it should be noted that such algorithms are not fully verified (Kumar et al. 2020; Rudin 2019); besides, they are approximate approaches.”

“This statement is generic, providing little practical guidance to developers as to what evidence would demonstrate compliance [38, 46, 54, 74, 95].”

“This method uses conditional expectations to represent the imp function and has gained popularity in machine learning (Broeck et al. (2022) [19]), despite some authors having raised questions about its explainability (Kumar et al. (2020) [7]).”

“Recently, the Shapley value is employed to explain the deep learning model and has presented excellent results [16], [17].”

“Kumar et al. (2020) present several qualms about the use of Shapley value in any of its forms for variable importance.”

“Weights are assigned to individual features highlighting their contribution (importance) towards model prediction, e.g. logistic regression (Thomas et al., 2008), saliency maps (Kindermans et al., 2019) and Shapley explanations (Kumar et al., 2020).”

“This is not rock solid yet, as both LIME and Shapley values have been criticized (Garreau and von Luxburg 2020; Kumar et al. 2020; Merrick and Taly 2019).”

“Kumar et al. (Kumar et al., 2020b) pointed to certain mathematical shortcomings of S HAP (including the question of addressing causality) and the fact that Shapley values represent only a summary of a game.”

“Interventional algorithms, while computationally more efficient, presume feature independence and rely on samples from outside the distribution, potentially leading to unrealistic observations [60].”

“Second, SHAP assumes feature independence, which is often violated in educational data where features tend to be interrelated, potentially leading to misleading interpretations [134].”

“We investigate the performance of each of EAI techniques in identifying local impacts of ranking indicators and demonstrate that Shapley Value [20] is a reliable explanation technique for this matter.”

“Many papers discuss failure modes of explanation algorithms (Adebayo et al., 2018; Kumar et al., 2020; Molnar et al., 2020; Bordt et al., 2022; Geirhos et al., 2024; Bove et al., 2024; Huang & Marques-Silva, 2024; Tomaszewska & Biecek, 2024).”

“…some widely used explainability scores, believed to be theoretically mature and robust, may behave counterintuitively in certain situations (Ignatiev, Narodyt-ska, and Marques-Silva, 2019b; Ignatiev, 2020; Camburu et al., 2019; Slack et al., 2020; Kumar et al., 2020; Huang and Marques-Silva, 2023).”

“However, it results in losing the locality of explanation because it attempts to explain the increment from any point in the domain, as suggested in [21] regarding SV.”