Problems with Shapley-value-based explanations as feature importance measures

“Many recently proposed “local” methods (Ribeiro et al., 2016; Lundberg & Lee, 2017; Lundberg et al., 2018) define a value function vf,x : 2 → R that depends on a specific data instance x to explain how each feature contributes to the output of the function on this instance.”

“Since the notion of the sum of two games is not especially meaningful, the Additivity axiom has been described by game theorists as “mathematically convenient” and “not nearly so innocent as the other two” Osborne and Rubinstein [1994]. The choice to constrain the value to be unique in this way has implications for wh…”

“Instead, Miller (2019) argues that humans are selective about explanations: certain contrasts are more meaningful than others.”

“This perspective is closer in spirit to the “humangrounded metrics” that Doshi-Velez & Kim (2017) describe in comparison with the “functionally-grounded evaluation” of the previous section.”

“TreeSHAP (Lundberg et al., 2018), an efficient algorithm for calculating SHAP values on additive tree-based models such as random forests and gradient boosting machines, can estimate EXS̄ |XS [f(xS ,XS̄)] by observing what proportion”

“However, has been pointed out [96, 97] that the existing definition of Shapley values in explainability, as used in SHAP [54], can produce misleading information regarding feature importance.”

“Time-series explainers can produce different importance patterns under different background sets because Shapley-based masking depends on the reference distribution [35, 41].”

“5E), we acknowledge that SHAP can have limitations in the presence of highly correlated features 38 , which are ubiquitous in genomics.”

“…to gain a high-level intuition of feature importances, but are not guaranteed to reliably attribute the correct importance to every feature 45,46,48 . are positive correlations between the Shannon entropy of the NPI active vs. non-active status distributions over time and the 95 % boundaries…”

“On top of that, the success of an explainable ML model will rely on the human comprehension of model behavior by less technical audiences (Kumar et al., 2020).”

“Firstly, these are known to disobey the “data manifold hypothesis” (Fefferman et al., 2016): Kumar et al. (Kumar et al., 2020) and Sundararajan et al. (Sundararajan and Najmi, 2020) showed these methods often operate outside the data’s true distribution.”

“Unfortunately, these approaches have to contend with the reliability issue that post hoc methods suffer from in the static settings [23].”

“Finally, better explainability methods need to be developed to address some recent critiques of Shapley-value-based explanation methods [55, 56].”

“Nevertheless, over the years researchers identified issues with the results obtained with the tool SHAP [77,120,96,1].”

“The impact of correlated data is examined by comparing the importance”

“Moreover, Kumar et al. (2020) pointed out that Shapley values do not satisfy human-centric goals of explainability, and Jethani et al. (2023) added that all the afore-mentioned methods can lead to false conclusions as they are “class-dependent”.”

“For variable importance assessments, in addition to LOCO methods within our framework, Shapley value-based measures are commonly used [24, 25, 26, 27].”

“Researchers have consistently shown (Laugel et al., 2019; Kumar et al., 2020; Slack et al., 2020; Alvarez-Melis and Jaakkola, 2018; Zhou et al., 2022) that these methods are unreliable, often disagreeing with each other (Krishna et al., 2022) and failing to reproduce model ground truth when it is…”

“This finding is unsurprising; it has been consistently shown in any class of model where one has the ground truth interpretation (Kumar et al. 2020), so long as the features are correlated – as predictors tend to be in real-world applications.”

“There are also critical aspects of SVs in this context as for example studied in [27].”

“…although these methods provide some interesting properties, they are sometimes based on some computational mechanisms (e.g., exact Shapley value computation) or hypotheses (e.g., independence between features) that are difficult to achieve in practice, leading to biased explanations [1, 8].”

“Unstable explanations can arise from high‐dimensional feature interactions and/or stochastic sampling variance inherent in SHAP estimation processes (Kumar et al., 2020; Molnar, 2025; Rudin et al., 2022).”

“For further reading on SHAP and its computational strategies, see articles Christoph ( 2020 ), Covert et al. ( 2021 ) and Kumar et al. ( 2020 ).”

“While SHAP offers robust insights, the specialized nature of its theoretical foundation may present a learning curve for practitioners not familiar with game theory concepts [24].”

“Additionally, the interpretability of SHAP values can sometimes be obscured by interactions between features; while SHAP attempts to account for these interactions, the resulting explanations may still be challenging for non-experts to understand [22].”

“In particular, SHAP values are sensitive to high correlations among different features [48].”

“Obtaining a model’s all possible weights is beneficial for computing, e.g., feature importance , an effective explanation measure [5, 8] that indicates the statistical contribution of each feature to the underlying model.”

“The method of feature removal can significantly influence the resulting feature attribution values and their interpretation [12, 36, 37].”

“While post-hoc methods offer some interpretability, recent work [3], [4] has argued that these methods can lead to unreliable explanations, which could potentially have detrimental effects on their explainability.”

“However, when using Shapley value for responsibility attribution, agents which contributes more to the task may end up being blamed more, which may not align with intuitive notions of responsibility in collaborative settings (Kumar et al., 2020).”

“Note that here we focus on the problems of SVA methods for discovering the informational dependencies between features and the target, while others consider causal relationships [14,32] or the conceptual inadequacies of Shapley values for explanations [15,16,12].”

“Also, during learning between epochs, the saliency maps can change dramatically [20].”

“The paper [Bud93] decomposes the R 2 using a method similar to Shapley attribution but with different weights, and [CS91] decomposes any goodness-of-fit metric of a regression model using a method shown in [Stu92] to be equivalent to Shapley attribution.”

“argue that SHAP cannot be used to inform users’ actions [5].”

“In fact, humans inherently understand explanations contrastively (Kumar et al., 2020).”

“Three FI methods: PI, SHAP, and Gini Importance are used.”

“To summarize the aforementioned issue, the mean-field interactions consist of minimizing simultaneously costs of the following form”

“Seventh, SHAP attribution among correlated features is not uniquely identified [35]; the factor-profile labels should be interpreted as associated patterns.”

“Section X recasts the most-cited critiques of attribution methods [9], [10], [14], [15], [18] as mathematical problems tied to explicit modelling choices.”

“However, Kumar et al. [2020], Yan and Procaccia [2021] have raised concerns about the imposed linearity axiom, as its necessity remains unclear.”

“However, coalition-based attri-bution remains expensive for high-dimensional visual inputs and may dilute the importance of correlated regions [42].”

“Although these approaches aim to support end-users in understanding why the AI predicted a particular outcome, numerous empirical studies have highlighted their limitations [15, 16, 38, 46].”

“Notably, recent studies [20]-[27] demonstrate that SHAP can produce misleading or biased scores.”

“Recent studies [14]-[20] showed that existing SHAP can produce misleading or biased indicating scores.”

“…such as LIME and SHAP, assume additivity of multiple variables, there is concern that the predictive model itself is learning more complex patterns among explanatory variables but may be losing some amount of information [10] due to the assumption [11] when providing explanatory properties.”

“…and conditional distributions often give drastically different results and are each subject to serious shortcomings that a number of recent works have brought to light, e.g., Frye et al. (2020), Janzing et al. (2020), Kumar et al. (2020), Sundararajan and Najmi (2020), and Hooker et al. (2021).”

“Shapley-based explanations provide principled marginal contribution estimates [21], but they scale poorly as the number of agents increases [22].”

“As a post hoc, model-agnostic method, SHAP assumes independence among source variables in analytical models (Kumar et al .”

“Model-agnostic states that the explanation method does not require prior knowledge of the models’ internals [38], which was already recognized as a desirable metric in the very first works on explainable AI [32], and has since been established [18].”

“On the other hand, FlowX also struggles with unfaithfulness, as Shapley values may introduce unintended, mathematically formalizable properties in feature importance explanations [21].”

“…to improve classic post-hoc feature attribution methods, including TreeSHAP (Lundberg & Lee, 2017), Kernel SHAP (Lundberg, 2017), Extended Kernel SHAP (Aas et al., 2021), along with the extension of SHAP to various scenarios (Fryer et al., 2021; Van den Broeck et al., 2022; Kumar et al., 2020).”

“A et al., 2016), as these approaches are inexact and prone to approximation bias (Kumar et al., 2020; Van den Broeck et al., 2022).”

“Many papers have discussed how existing SVA meth-ods make unreasonable explanations [3, 11, 15, 37].”

“This can be a problem if someone tries to manipulate the model (13).”

“This methodology, while informative in numerous situations, is not without concerns [14]–[16].”

“Although the utilization of Shapley values for explaining DL models may sometimes lead to non-intuitive feature attributions [8], [9], [10], the ideal mathematical properties make it possible to fairly measure the contribution of each input in the model.”

“LIME inherently suffers from stability issues [31], as for SHAP [19] the attribution of feature importance through mathematically formalizable properties (i.e., local accuracy, missingness, and consistency) may not always align with users’ expectations for explanations [17].”

“Furthermore, Shapley values can be applied in various contexts, including assessing model fairness and detecting attribute-based unfairness or discrimination (Lundberg & Lee, 2017; Lundberg et al. 2020; Kumar et al., 2020).”

“61 Thus, we used the permutation feature importance provided in the rfpimp package in Python.”

“Further discussion and examples on the mathematical issues that arise from the estimation procedures used when applying Shapley values as feature importance measures can be found in [35].”

“We also found several works in which game theory tools are used to explain machine learning models [8]–[13].”

“…compliant with legal regulation such as the General Data Protection Regulation’s “right to an explanation”, others argue that Shapley values do not provide explanations that suit human-centric goals of explainability (Adler et al. 2018; Merrick and Taly 2019) as stressed in Kumar et al. (2020).”

“This is important not only for the stakeholders who receive explanations but also to engineers who use these methods to evaluate the model F [Kumar et al., 2020].”

“Kumar et al. (2020) and Janzing et al. (2020) discuss these differences in the context of satisfying axioms of additive variable attributions.”

“Based on the Shapley values in game theory, it can assign a contribution value to each feature for the model’s prediction results, thus helping us understand the importance of each feature in the model’s decision-making process [95].”

“Conversely, Rudin [61], Huang and Marques-Silva [62], Kumar et al. [63], and others note that model-agnostic methods like SHAP provide perspectives grounded in data perturbation to understand feature contributions.”

“This is a strong assumption in the case of a real-world application that is criticised in (Kumar et al., 2020).”

“…some widely used explainability scores, believed to be theoretically mature and robust, may behave counterintuitively in certain situations (Ignatiev, Narodyt-ska, and Marques-Silva, 2019b; Ignatiev, 2020; Camburu et al., 2019; Slack et al., 2020; Kumar et al., 2020; Huang and Marques-Silva, 2023).”

“Kumar et al. (2020) highlight several mathematical problems with SHAP explanations and show that they do not suit human-centric goals of explainability.”

“This approach was adopted by Lipovetsky and Conklin (2001), Israeli (2007), and Huettner and Sunder (2012) to decompose the R2 of a linear model.”

“7 can have different contributions for different predictions, which shows that interpreting Shapley values requires caution [Kumar et al., 2020].”

“• Effect of feature product on local explanations [20].”

“As such, our work tries to promote substantive discussions and accurate definitions of explainability, as previously advocated, for example, by Lipton [28] and Kumar et al. [29].”

“A mean marginal contribution answers whether a feature helps on average, not whether, in some coalition, adding it turns an acceptable model into an unacceptable one, and Shapley-value-based scores have known limitations as feature-importance measures (Kumar et al., 2020).”

“Kumar et al. (2020) A specialised variant of SHAP, called TreeSHAP, was also employed.”