Problems with Shapley-value-based explanations as feature importance measures

“Many recently proposed “local” methods (Ribeiro et al., 2016; Lundberg & Lee, 2017; Lundberg et al., 2018) define a value function vf,x : 2 → R that depends on a specific data instance x to explain how each feature contributes to the output of the function on this instance.”

“Since the notion of the sum of two games is not especially meaningful, the Additivity axiom has been described by game theorists as “mathematically convenient” and “not nearly so innocent as the other two” Osborne and Rubinstein [1994]. The choice to constrain the value to be unique in this way has implications for wh…”

“Instead, Miller (2019) argues that humans are selective about explanations: certain contrasts are more meaningful than others.”

“This perspective is closer in spirit to the “humangrounded metrics” that Doshi-Velez & Kim (2017) describe in comparison with the “functionally-grounded evaluation” of the previous section.”

“TreeSHAP (Lundberg et al., 2018), an efficient algorithm for calculating SHAP values on additive tree-based models such as random forests and gradient boosting machines, can estimate EXS̄ |XS [f(xS ,XS̄)] by observing what proportion”

“Shapley-value attributions depend on the reference distribution and can be unstable across correlated features [28]; adversarially constructed models can produce misleading SHAP and LIME explanations [29]; and post-hoc explanation of black-box models has been criticized as a weaker guarantee than…”

“It also enables the possibility of computing feature importance [24, 30], which quantifies each feature’s statistical contribution to the trained model.”

“In practice, too much randomization may yield different stories from one run to another [4, 5].”

“We separately report SHAP attribution direction (Spearman correlation between feature value and attribution) and partial dependence, since Shapley-value attributions are known to encode which features drove a prediction without reliably encoding which way they pushed it [21].”

“Recent work has shown that post-hoc attributions can be highly sensitive to model initialization and training data, raising concerns about their reliability for guiding compound optimization [33].”

“SHAP is a representative model-agnostic method that estimates feature contributions based on Shapley values (Kumar et al., 2020).”

“Kumar et al. [21] make this argument for Shapley-value feature importance, Janzing et al. [17] for the choice of value function, which they cast as a causal problem, and Jacovi and Goldberg [16] for the definition and evaluation of faithfulness.”

“These findings indicate that the model uses both physicochemical information and learned transport-related scores, but they do not establish biological causality or identify the transporter responsible for an individual prediction [41].”

“Kumar et al. argue that mathematical properties inherited from game theory do not automatically make Shapley values valid measures of feature importance for a learned function [6].”

“Kumar et al. [32] show that Shapley-based importance behaves counterintuitively under dependence and redundancy in features.”

“[18], [19] Moreover, recent studies have raised concerns about the reliability of SHAP explanations, particularly in the context of anomaly detection, where the model’s sensitivity to input changes may not accurately reflect its internal reasoning [20], [21].”

“Moreover, the attributions can be unstable, changing with choices external to the model and data, such as LIME’s perturbation scheme or SHAP’s reference distribution [10, 11, 12, 13].”

“Dependence among features can change Shapley-based attribution, and feature-importance summaries can be misleading when they are treated as intrinsic properties of the data-generating process (Aas et al., 2021; Kumar et al., 2020).”

“However, their reliance on quantitative attributions often fails to provide sufficient semantic interpretability in practical settings, limiting their usefulness for practitioner decision-making [17, 25].”

“SHAP values provide model-level association structure rather than mechanistic or causal evidence, and their interpretation in correlated feature spaces requires caution [30, 38] (Figure 3).”

“Crucially, we conceptualise SHAP strictly as a model-faithful diagnostic rather than a causal quantity, acknowledging fundamental critiques of SHAP’s causal validity [4, 37, 39, 40].”

“However, Kumar et al. [11] point out that some further assumptions, such as the feature independence, are needed to use this framework for interpretation.”

“Kumar et al. (Kumar et al., 2020) further argued that Shapley-based feature importance has nontrivial mathematical and semantic limits, especially when feature dependence and causal interpretation matter.”

“SHAP improves model interpretability by quantifying variable contributions and revealing heterogeneous effects, but its explanations mainly indicate predictive associations rather than causal relationships, especially when explanatory variables are interdependent [35–37].”

“This concern is consistent with Kumar, Venkatasubramanian, Scheidegger, and Friedler (2020) who argue that Shapley-value-based feature importance may not fully satisfy human-centered explanation goals.”

“However, the algorithm has also been frequently criticized [Kumar et al., 2020]: Similar to counterfactual explanations, SHAP is sensitive to off-manifold behavior [Frye et al., 2020, Taufiq et al., 2023] and can be manipulated by an adversary [Slack et al., 2020, Hwang et al., 2025].”

“More specifically, it applies the most commonly used solution concept for non-cooperative games (91), i.e., Nash equilibrium (NE) is a situation in a game where no player can improve their outcome by unilaterally changing their strategy, assuming all other players’ strategies remain the same.”

“Post-hoc XAI methods, especially in a model-agnostic setting, have been criticized for their unreliability in the absence of ground-truth information [25], [26].”

“Provenance plays an important role in ML by providing detailed insights into how models are trained, validated, and deployed [3, 16, 31, 37, 45, 46, 49, 56, 58].”

“We also acknowledge existing critiques of SHAP (Huang & Marques-Silva, 2024; Kumar et al., 2020; Slack et al., 2020; Fryer et al., 2021; Biradar et al., 2024); our goal is not to defend its axiomatic foundations, but to provide a way to compute provably exact values for a widely used explanation…”

“• Generalising, if VIF ( 𝑥 𝑖 ) > 𝜃 , where 𝜃 ∈ [ 5 , 10 ] is a diagnostic threshold, then the variance of attribution values for 𝑥 𝑖 across bootstrap resamples satisfies: for some model-dependent constant 𝑐 > 0 .”

“Wright et al. (2016); Kumar et al. (2020, 2021) agree on the limited capabil - ity of the SV to capture synergy effects between features.”

“…such as LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017) are fundamentally correlational (Janzing et al., 2020): their marginal imputations disrupt the joint distribution, producing explanations based on causally impossible counterfactuals (Kumar et al., 2020; Frye et al., 2020).”

“Although the tree‐based SHAP interpreter is emerging as a de facto method for explaining the outputs of machine learning models (Feng et al., 2024; Merrick & Taly, 2020), interpretation still requires caution and domain expertise (I. E. Kumar et al., 2020).”

“The integration of XGBoost with SHAP enables not only accurate phishing detection, but also a deep understanding of the factors that influence the prediction [19] – [21].”

“SHAP, an additive feature attribution model based on cooperative game theory, measures the marginal contribution and importance of each feature [67,68].”

“The problem is especially acute for correlated features, where SHAP must distribute credit among features that carry overlapping information [Kumar et al., 2020, Chen et al., 2020].”

“Yet impossibility results [5], conflicting opera-tionalizations [60], and axiom-level critiques [32, 33, 65] highlight that axioms constrain but context matters, motivating domain-specific axiomatizations.”

“One area of research studies how to improve sampling strategies so that they work better and have less variation [10], [19].”

“Feature importance, however artfully com‑ puted, is not a causal pathway [24, 25].”

“7), supporting the complexity and specificity of their geochemical fingerprints (Kumar et al. 2020).”

“The authors [18] showed that SHAP baseline selection critically affects attribution magnitudes, with poorly chosen baselines producing biologically implausible explanations.”

“Kumar et al. [8] demonstrated that SHAP consistently provided more faithful explanations across healthcare datasets.”

“However, computing the exact Shapley value requires evaluating all possible coalitions, causing exponential complexity with increasing agent numbers [11].”

“…a single score cannot disambiguate the nature of a feature’s contribution, conflating direct influence with importance derived from complex non-linearities or interactions [5]. deriving this information directly from the model’s geometry via the Hessian, offering an efficient, model-native measure.”

“There are also limitations to using SHAP feature importance scores for ML model interpretability, as suggested by Kumar et al. [168].”

“The fidelity of the simplified risk scores to the underlying XGBoost model depends on SHAP’s additive feature attribution, which may not always fully capture complex nonlinearities or nuanced higher-order interactions [16].”

“There also exist works identifying issues with Shapley values and SHAP scores [19, 30, 27, 24, 35, 32], and proposing solutions to mitigate these drawbacks.”

“While these attributions methods can be applied to sequential models, many assume feature independence [43, 39].”

“…state–transition prediction task based on a recently proposed set–sequence model (6) and trained on extensive panel data, AICO detects time-varying macroeconomic variables such as mortgage rates, as well as features summarizing past state transitions, as significant predictors of borrower behavior.”

“While SHAP is theoretically grounded, some scholars have noted pitfalls using it when used for clinical studies—specifically, the lack of alignment with mechanistic goals of human interpretation [23].”

“Different approaches to compute an evaluation function v have been proposed in the literature (see, for instance, [7, 5, 12] for a discussion on this topic).”

“Additionally, Kumar et al. [48] observed that feature importance evaluation methods based on the Shapley value estimation may risk underestimating the importance of relevant features, resulting in all such features being assigned lower importance values.”

“To address out-of-distribution baselines generated by random sampling or general pretrained language model Kumar et al. [2020], we pretrain BERT-base on these five datasets to ensure in-distribution sampling and evaluate in-distribution baselines (i.e. condition-in and condition-in-uw ) over BERT…”

“Since school choice influences learning, thinking, and future success, this challenge is crucial [2].”

“SHAP is applicable across various models, including tree-based [15] and neural network models [16], enabling researchers to identify key predictors and analyze model behavior.”

“A feature importance tool that focuses on a single model may give the illusion of no racial bias even though the underlying data is actually highly racially biased (Chen et al., 2020; Kumar et al., 2020).”

“…> A2 [25] GRA method in NCS [32] A1 > A4 > A3 > A5 > A2 [28] NC Cosine measure [33] A1 > A4 > A3 > A5 > A2 [29] IGNCSCA [58] A1 > A3 > A4 > A5 > A2 [54] NC-CODAS [58] A1 > A3 > A4 > A5 > A2 [54] makes it possible to focus and resource on improving these crucial variables parameters, guaranteeing a…”

“Another feature-driven method is the Neighbor Shapley Value(CONE-SHAP), which assesses the importance of each concept by analyzing the physical and semantic neighbors of concepts[3, 26–28,”

“We used the SHAP framework [10] to analyze the importance of features.”

“Some argue that this is illogical as features not explicitly used by the model have the possibility of receiving non-zero allocation [Kumar et al., 2020].”

“out several limitations for each of the different distribution choices (Kumar et al., 2020; Fryer et al., 2021), causing the Shapley values to be seen as unstable and as such, unfit for safety-critical domains such as healthcare.”

“However, these tools can introduce their own biases [15,2].”

“[17] showed that mathematical problems arise when SHAP is used for feature importance in tree models.”

“[20], [21].”

“As these indices are derived from cooperative game theory, as noted in [21] and [37], they comprise a coalition of players and a characteristic function.”

“Explanation stability was assessed using three complementary metrics computed across ten independent perturbation runs per instance, illustrated in Figure 3, drawing on recent work on interpretability robustness [27], [28], [54].”

“AlthoughexplainableAIframeworks,suchasSHapley AdditiveexPlanations(SHAP)[27],havebeguntoimprove modeltransparencybyquantifyingfeaturecontributions[28], the reliability of these explanations is highly sensitive to the quality and size of the input data [29], [30], [31].”

“However, the standard SHAP method can produce unstable or even misleading feature importance because it relies on knowledge of the underlying probability distribution, which is often unavailable or estimated from a limiteddataset[58],[59].”

“There are some studies such as (Ribeiro et al., 2016) and (Kumar et al., 2020) that highlight the strengths and weaknesses of these explanatory methods in different fields.”

“While these methods are widely utilized, they present significant limitations in root cause analysis, as they do not explicitly incorporate causal reasoning [11].”

“Although the utilization of Shapley values for explaining DL models may sometimes lead to non-intuitive feature attributions [8], [9], [10], the ideal mathematical properties make it possible to fairly measure the contribution of each input in the model.”