Problems with Shapley-value-based explanations as feature importance measures

“Many recently proposed “local” methods (Ribeiro et al., 2016; Lundberg & Lee, 2017; Lundberg et al., 2018) define a value function vf,x : 2 → R that depends on a specific data instance x to explain how each feature contributes to the output of the function on this instance.”

“Since the notion of the sum of two games is not especially meaningful, the Additivity axiom has been described by game theorists as “mathematically convenient” and “not nearly so innocent as the other two” Osborne and Rubinstein [1994]. The choice to constrain the value to be unique in this way has implications for wh…”

“Instead, Miller (2019) argues that humans are selective about explanations: certain contrasts are more meaningful than others.”

“This perspective is closer in spirit to the “humangrounded metrics” that Doshi-Velez & Kim (2017) describe in comparison with the “functionally-grounded evaluation” of the previous section.”

“TreeSHAP (Lundberg et al., 2018), an efficient algorithm for calculating SHAP values on additive tree-based models such as random forests and gradient boosting machines, can estimate EXS̄ |XS [f(xS ,XS̄)] by observing what proportion”

“But, the methods estimating value function by conditional or interventional expectation may produce unreliable attributions [12].”

“Both approaches come with their respective caveats [Kumar et al., 2020].”

“In the extreme case of identical features, the result Shapley values is effectively averaged over each feature, thus resulting in artificially lowered feature im-portances [46].”

“…methods available to generate rank orderings of predictor importance (Lundberg & Lee, 2017) based on concerns that have been raised in the literature about the plausibility of the assumptions needed for such rankings to be interpretable (Kumar, Venkatasubramanian, Scheidegger, & Friedler, 2020).”

“Recent work has pointed out that mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity [49].”

“We also acknowledge existing critiques of SHAP [70, 57, 99, 49, 25]; our goal is not to defend its axiomatic foundations but to analyze the complexity of a widely adopted explanation method.”

“The research by Kumar et al. (2020), presented at the International Conference on Machine Learning, further quantifies this limitation of SHAP explanations.”

“Shapley values have known limitations, especially the assumption that features are not correlated [31] which is untrue for EMBER features.”

“Furthermore, they have been shown to suffer from various issues, such as a vulnerability to correlated features (Kumar et al., 2020), problems of locality (Laugel et al., 2018b) or general lack of trustworthiness (Slack et al., 2020), making their returned explanations possibly unreliable and…”

“Motivated by a series of well-documented failures of machine learning (ML) models being deployed in real-world systems with harmful biases and costly inaccurate predictions [2, 3, 4], and the discovery that post-hoc blackbox model explainers are not always truthful to the underlying model [5], there has been significa…”

“Also, during learning between epochs, the saliency maps can change dramatically [20].”

“For example, Kumar et al. [23] question the justification of the additivity constraint imposed by SHAP.”

“Furthermore, as con fi rmed by a recent paper by (Kumar et al., 2020), problems arise from the estimation procedures used for speci fi c post-hoc explainability techniques.”

“Although SHAP values have demonstrated great success to explain machine learning algorithms, Kumar et al. (2020) show that both observational SHAP and interventional SHAP have quite strong limitations, while the HFD is an interesting alternative to overcome these problems.”

“Epistemic uncertainty • variance from data sampling (different D yield different mean SHAP values) • measures sensitivity to training data composition 3.”

“The negative effects of interventional value functions for the Shapley values in explainability and feature selection are examined by Kumar et al. [45].”

“Our experimental results show that our pipeline can predict heart failure more accurately than traditional survival analysis models and is comparable to other state-of-the-art machine learning models, while providing intelligibility.”

“…most methods focus on the local neighborhood of an input sample, and rely on assumptions such as linearity (e.g. gradient-based methods) or independence of the input features (e.g. approximation of Shapley values (Štrumbelj & Kononenko, 2011; 2014; Lundberg & Lee, 2017; Kumar et al., 2020)).”

“Shapley values for feature importance [36] based on their inconsistency across distributions and the lack of a normative human evaluation for the values [32].”

“One of the reasons such methods are vulnerable to adversarial attacks is that the data sampled from input pertubation often are irrelevant, and the model is forced to explain input samples it has never seen before [11].”

“However, problems with Shapley-value-based explanations as feature importance are still left [11].”

“However, when the data is perturbed and has a large number of features in deep learning model, they become unsuitable [15], [18].”

“To facilitate result interpretation, a Bee Swarm plot was used, showing simultaneously the impact of each variable on all dataset instances, allowing for identification of critical thresholds, nonlinear behaviors, and potential interactions between variables [27–31].”

“Kumar et al. (2020) dealt also with correlated variables concluding that “ … Shapley values are not a natural solu-tion to the human-centric goals of explainability … ” .”

“SHAP overestimates irrelevant features over relevant features under binary networks [62] and incorrectly identifies important features in out-of-distribution scenarios [63].”

“4.4 H3: Neither perceived teleology nor quality ratings are affected by autonomous vs. human driver status We manipulated across conditions whether participants were told the blue car was an autonomous vehicle or was driven by a human driver (independent variable ‘AV’ in Study 1).”

“This may be explained by the fact that perturbation-based frameworks often generate perturbed examples that lie out of the data distribution [115].”

“A number of authors have reported pitfalls with the use of SHAP and Shapley values as a measure of feature importance [73, 42, 66, 52, 27, 72, 55, 2, 70, 41, 14].”

“We also remark that —while in this section we focus on the connection of counterfactual explanations with Shapley-values based explanation because of its popularity in the XAI field as well as in broader machine learning community — our theoretical results can be generalised to other game-theoretic solution concepts b…”

“Although providing additional model interpretability, of relevance for cognitive modelling, Shapley values cannot provide causal evidence (feature X causes a better prediction of outcome Y) but only weaker correlation patterns [41].”

“…also implicitly rely on the unconditional expected value of the label as a base case, which is hard for the human decision-maker to interpret or conceptualize [Kumar et al. 2020 Humans “build models to capture key features of the world, to envisage possible scenarios and to generate explanations”…”

“Sixth, we made use of SHAP value method to generate insights into our predictions but acknowledge that there are issues regarding interpretability (Kumar et al. 2020; Weerts, van Ipenburg, and Pechenizkiy 2019).”

“This did not prevent some approaches such as KernelSHAP to be extensively used, 1 https://archive.ics.uci.edu/ml/datasets/adult in spite of the issues mentioned in this paper, or in other studies (Kumar et al., 2020).”

“46,47 Although current research has demonstrated the potential of SHAP to enhance model transparency, a major translational gap remains: how to convert interpretability outputs into actionable clinical guidance for RTS decision making.”

“Also Merrick and Taly (2020) and Kumar et al. (2020) consider this when investigating the axioms of SHAP values.”

“However, studies have shown that post-hoc explanations may not always faithfully reflect the decision-making process of the underlying model, leading to misleading or incomplete interpretations (Kumar et al., 2020; Bilodeau et al., 2024).”

“We select the SHAP approach as several recent studies have shown that SHAP values can achieve better results for assessing feature importance in machine learning models than other metrics for various problems of interest [28, 55, 13, 24].”

“TreeSHAP spreads the contribution among the set of correlated resulting in less sparse feature attribution [10, 11, 28].”

“…may not be interpretable if the data is incorrect, the model acts on incorrect assumptions, has inappropriate biases, or the incorrect set of variables is used as inputs [103], which is why checking and validating the prediction ability of the models is crucial before understanding their structure.”

“Considering the complexity of many real-world systems, it is unlikely that the independence assumption of [5] in the global explanation methods holds in general, hence it can lead to spurious explanations [14].”

“For example, in Kumar et al., 12 the authors highlight the mathematical limitation of different approximation techniques such as SHAP or LIME, human-centric issues, and the dif fi culty with causal reasoning.”

“We excluded artificial neural networks (ANN) from the consideration, because it is difficult to perform important feature selection in such cases: the only working option is the Shapley values method with SHAP algorithm, but this has limitations [42].”

“While decision trees can be ensembled leading to better performance than single trees, like in the random survival forest algorithm by Ishwaran et al, this makes them considerably less interpretable [25-27].”

“We also hope to explore whether our extension can be applied to similar explainable AI methods like SHAP where neighborhood sampling is also employed and the same out-of-distribution problem is observed (Lundberg and Lee 2017; Slack et al. 2019; Kumar et al. 2020).”

“They use a Deep Neural Network to make a classification of the teams and Shapley values for feature importance.”

“Although certain attribution methods, such as Shapley value , have been shown grounded in relatively sound theoretical foundations, they are not without limitations [55], [138], [139].”

“Additionally, research has demonstrated that some explainability metrics, despite being considered theoretically sound and robust, can exhibit unexpected behavior under certain circumstances [7, 16, 17, 19, 24, 31].”

“We begin by proving the lower bound.”

“Recent studies have suggested issues with implementing Shapley values for feature importance measures [39], but the technique is a common tool in the literature, implemented here to attempt to provide insight into our transition probability predictions models.”

“The widespread use of these approaches has exposed their limitations, including their great computational complexity (Kumar et al., 2020).”

“However, the reliability of explanations provided by XAI approaches remains a critical challenge [8], [9], [10], [11].”

“See also Kumar et al. (2020).”

“Unlike additive attribution methods, which allocate importance scores across features but are often hard for humans to interpret (Kumar et al., 2020) or lack actionability (Bilodeau et al., 2024), sufficient and contrastive reasons provide discrete, condition-like explanations that directly answer…”

“However, recent studies have revealed that such techniques can yield inconsistent or misleading insights, especially in complex manufacturing contexts, due to their sensitivity to model structure and input representation (Huang & Marques-Silva, 2024 ; Hwang et al., 2025; Kumar et al., 2020 ).”

“However, several key limitations of most non-symbolic methods of XAI has been identified in recent years [Ignatiev et al. , 2019b; Izza et al. , 2020; Kumar et al. , 2020; Ignatiev, 2020; Izza et al. , 2022; Marques-Silva and Huang, 2024; Zhang et al. , 2024].”

“Other works connect degraded attribution quality to complexity more indirectly: Kumar et al. (2020) provide examples where Shapley-value attri-butions misrepresent feature contributions in the presence of intricate dependencies, Fokkema et al. (2023) prove that attribution methods offering…”

“Other representative methods leverage the Shapley Value (Shapley, 1953), but its computational complexity grows factorially with the number of agents (Kumar et al., 2020), handicapping efficient training in practice.”

“Approximation methods—Monte Carlo sampling, regression-based estimators—sidestep the exponential cost by evaluating only a subset of coalitions, but introduce estimation errors that grow with dimensionality (Kumar et al., 2020).”

“Knowing SHAP’s limitation [39], the features were ranked using the mean SHAP value of all data with three hyperparameter-tuned classifiers.”

“There are generally two approaches to handling removed features in feature perturbation for general set functions, as discussed in Chen et al. (2020) and Kumar et al. (2020).”

“Due to its open-source nature and ease of use, SHAP is commonly used, although the interviewees are aware of the criticisms levied at this method, e.g., by Slack et al. (2020); Kumar et al. (2020), and thus sceptical of its performance and reliability.”

“Along with the computational demands, we are also faced with the subjectivity involved in selecting neighborhoods and other parameters specific to these techniques [85,86].”

“Many XAI methods are designed to explain the behavior of black-box machine learning methods as a post-processing step [43, 53]; however, these methods are prone to approximation bias [2, 34, 51, 60].”

“Incidentally, notice that —borrowing the terminology of [6]— this goes in the direction of having a more “conditional” type of estimation compared to the “interventional” approach of Anchors, whose neighbourhood generation breaks correlations of the data-generating distribution.”

“Although Kumar et al. (2020) mention that some contrastive interpretations of standard values are unlocked via properly setting out-of-coalitions feature values, there is no obvious way to use the ϕ ( u i ) ’s to formulate contrastive statements of the type “the feature that is most responsible to…”

“Kumar et al. [46] argue that SHAP [56] has several human-centred issues, including non-contrastive and non-actionable explanations, and that most people do not have a correct mental model of Shapley values.”

“We will show that TERC can achieve this even in the presence of perfectly redundant state variables (Frye et al., 2020; Kumar et al., 2020).”

“The best model for each mechanical parameter and stainless steel grade is chosen according to the values of MSE and R ; these appear in bold in Table 3.”

“For each query point, the total deviation of the prediction from the mean is equal to the sum of the Shapley values for all features [11], [12].”

“Some popular heuristics ( e.g. , data valuation (Ghorbani & Zou, 2019), coresets (Borsos et al., 2020a)) fall short from a variety of angles (Basu et al., 2021; Kumar et al., 2020; Toneva et al., 2019; Sener & Savarese, 2018).”

“GE supports understanding how a model makes decisions based on a holistic view of its features, e.g., [1], [3], while LE explains why the model makes a specific prediction, e.g., [4].”

“Such algorithms are not fully verified [24, 25]; besides, they are approximate approaches.”

“Indeed, many Shapley values can be defined given the assumptions made on the model, the data, and the features excluded from the coalitions (see for instance Strumbelj and Kononenko (2010), Redell (2019), Kumar et al. (2020), Sundararajan and Najmi (2020), and Aas et al. (2021)).”

“Despite the solid theoretical foundation, there is concern [Kumar et al., 2020] about its suitability for explanations.”

“This finding is unsurprising; it has been consistently shown in any class of model where one has the ground truth interpretation (Kumar et al. 2020), so long as the features are correlated – as predictors tend to be in real-world applications.”

“Post-hoc methods have a number of shortcomings given our desired objectives: First, it is unclear whether post-hoc explanations indeed reflect the black-box model’s true ”reasoning” [Kumar et al., 2020, Rudin, 2018].”

“Recent studies (Dabkowski and Gal 2017; Merrick and Taly 2020; Kumar et al. 2020; Binder et al. 2016; Shriku-mar, Greenside, and Kundaje 2017; Frye et al. 2020; Tan 2023) have provided experiential advice on choosing base-line values, while falling short of theoretical illumination.”