Enhanced integrated gradients: improving interpretability of deep learning models using splicing codes as a case study

“[105] propose enhanced integrated gradients and demonstrate that nonlinear paths can improve interpretability on”

“It heavily depends on the baseline choice and can be expensive since it has to compute the gradients multiple times [28].”

“Incorporation of established bioinformatic methods and databases with ML and DL frameworks [128,196] Generation of interpretable DL models [197–199] A major challenge for the application of ML and particularly DL to biological sequences is the representation of nucleotides or amino acid sequences…”

“Further, Jha et al. (2020) use KNNs and auto-encoders to learn latent paths for RNAs.”

“Jha et al. proposed the enhanced integrated gradients (EIG) method [169], which identified significant splicing code features in the liver and was adapted for identifying common transcriptional signatures of cancers [170].”

“As DeepLIFT is not implementation invariant, Sundararajan et al. [136] introduced integrated gradients (IG) , which was later enhanced by Jha et al. [66].”

“This problem can be alleviated with Integrated Gradients (40,41) or DeepLIFT, which propagates activation differences relative to a selected reference back to the input, reducing the computational overhead of obtaining accurate gradients (42).”

“The combination of both red and blue pixels indicates those regions of interest that are regarded as highly important for the model in assigning a class label to the entire image [27,28].”

“‘Backpropagation’, which determines the effect of each artificial neuron on the output, can identify regions of high importance in the input sequence; examples of backpropagation that have been explored to explain splicing models include enhanced integrated gradient (EIG) and Shapley values (in Deep SHAP) 109 .”

“• EIG (Jha et al., 2020).”

“This point was considered by Jha et al. (2020). They train an autoencoder whose encoding layers are extracted from the original model to be explained, and they obtain a generally nonstraight path as the image of a straight line path in the latent space defined by their auto-encoder.”

“Consistent with the findings of Jha et al. [17], we speculate that this is due to IG’s linear interpolation path, as interpolated inputs can be out-of-distribution.”

“Following our previous work (Jha et al., 2020), we used the median of normal samples as the baseline (see Methods section ”Interpretation of tumor classification models” for details).”

“The most common first step is to use an attribution method, e.g. gradient-based saliency maps (Sundararajan et al. 2017, Jha et al. 2020, Majdandzic et al. 2023), or Shapley-based methods such as DeepSHAP (Lundberg and Lee 2017, Shrikumar et al. 2017).”

“The integrated gradient along the i th dimension of inputs x and baseline x ′ is defined as shown in equation 21, where ∂ F ( x ) ∂ x i is the gradient of F ( x ) along the i th dimension [47].”

“Unsurprisingly, a large number of these feature attribution methods have been proposed, but we have seen three main classes of feature attribution methods: gradient-based [1, 2, 3, 4], path-based [5, 6], and counterfactual- or perturbation-based [7, 8, 9].”

“Even though there are still technical challenges in applying AS profiles to clinical areas, active research on AS is of great importance, which may lead to novel ideas for pertinent applications in clinical and pharmaceutical fields and offer further opportunities to resolve the complex patterns of AS.”

“Enhanced integrated gradients (EIGs) for biology introduced nonlinear paths, group-specific baselines, and statistical significance tests, outperforming DeepSHAP in identifying biologically relevant features [ 96].”

“22 Surrogate Models and Rule Extraction (Global Interpretability) An interpretable model can be trained to provide a high-level view of model behavior and decision boundaries.”

“While the method of Integrated Gradients has been used in several genomics studies to identify regulators of splicing in important genome regions for a distinct purpose [53], the baseline selection of integrated gradients for DNA sequencing and epigenetics modifications remains an open question.”

“In addition, the baseline is unclear in certain fields, such as genomics domain (Jha et al., 2020).”

“Integrated gradients have been used to analyze RNA splicing models (82).”

“Recently, deep neural networks (DNNs), predominantly based on convolutional neural networks (CNNs) or convolutional-recurrent network hybrids, have emerged as a promising alternative, in most cases, improving prediction performance on held-out test data 13–19 .”

“To assess the signi fi cance of these attributions, we followed the procedure in 37 and conducted a two-sided t-test comparing the k-mer attribution scores of each token and a random set of tokens, adjusting for multiple comparisons using Benjamini-Hochberg procedure with a signi fi cance threshold…”

“Integrating Integrated gradient with a self-refined segment editing model [16] for object tracking can enhance the model's transparency and provide interpretable insights into its tracking decisions.”

“[ 90] For CNN-based models, a saliency map method has been proposed for identifying what part of a sample (image, or text) was identified as an important feature by the model.”

“Integrated gradients have been extended in several works [26, 27, 28].”

“Since its introduction, IG has continuously evolved (Ghorbani, Abid, and Zou 2019; Jha et al. 2020; Yang, Wang, and Bilgic 2023; Pourdarbani et al. 2023; Yang, Wang, and Bilgic 2023), language models’ explanation (En-guehard 2023), and electronic health (Duell et al. 2023).”

“While other attribution methods like Integrated Gradients or LRP [14] can assign importance scores to model components, the direct, interventional nature of patching is widely considered a more rigorous and reliable test of a component's true functional role”

“The idea of using a k NN algorithm to avoid computing gradients on out of distribution data points has also been used in Enhanced Integrated Gradients Jha et al. (2020).”

“Thus ∇ 𝑓 ( 𝛾 ( 𝛼 )) = ∇ 𝑓 ( 𝑇𝛾 ( 𝛼 )) = 𝑇 ∇ 𝑓 ( 𝛾 ( 𝛼 )) .”

“Among many approaches, path-based methods (Sundararajan et al., 2017; Jha et al., 2020; Kapishnikov et al., 2021; Pan et al., 2021; Zaher et al., 2024; Zhuo & Ge, 2024) are particularly attractive because they connect attributions to principled requirements such as completeness and sensitivity…”

“Gradient-based methods depend on the choice of baseline, can produce erroneous results if the network output saturates in the early stages of integration, and often involve high computational costs-especially for Integrated Gradients [46] [47].”

“The interpretability plays a pivotal role in both model optimization and application [29].”

“Additionally, δ transitions the path x 0 x out-of-distribution [Jha et al., 2020, Adebayo et al., 2020], i.e. through intermediary images not representative of training data, leading to noise and artefacts in attributions.”

“IG [59] 0.3303 0.6775 0.3472 GIG [25] 0.3319 0.6781 0.3462 IG 2 [67] 0.2030 0.6010 0.3980 AGI [44] 0.4033 0.7290 0.3257 EIG [23] 0.3316 0.6688 0.3373 MIG [64] 0.3356 0.6751 0.3395 SPI [22] 0.1980 0.6105 0.4125 DiffIG 0.5711 0.7878 0.2166”

“IG has been proven to satisfy two fundamental axioms: sensitivity (the attribution is non-zero when the predictions are different for baseline) and implementation invariance (the attributions are always identical for two functionally equivalent networks) (Jha et al. 2020).”

“Additional considerations include baseline selection, where dinucleotide-shuffled references outperform zero baselines [58], and integration path choice [59].”

“Larger gradient changes suggest a more significant influence of the corresponding feature on the model output, indicating a more critical role in the learning process (Jha et al. 2020; Chan et al., 2024).”

“Several studies have recently reported integrated gradients (IG) and Shapley additive explanations (SHAP), included in explainable artificial intelligence methods [24,25], to interpret prediction models and to explore important features for prediction [6,26,27].”

“These include in silico mutagenesis [2] and gradient-based methods [10, 11].”

“Attribution methods have been applied to biological sequences to investigate features associated with predicted mRNA abundance [11], RNA splicing [12] and RNA protein-coding potential [13].”

“IG is a widely used and theoretically grounded attribution method for explaining predictions of di ff erentiable models [36, 37].”

“Methods such as SHAP, LIME, Integrated Gradients, and DeepLIFT can therefore be used to rank influential features and generate hypotheses about resistance-associated mutations, motifs, genes, or regulatory regions [125–127].”

“IG’s proven success on biological sequence models provides robust validation for high-confidence interaction calls 81.”

“Post-hoc feature attribution methods such as DeepLIFT [28] or Enhanced Integrated Gradients [15] are used to identify biologically important sequence features [20].”

“We quanti fi ed protein-level importance in the pretrained ProLM masked-regression setting using IG 75 .”

“[ 82] for tissue-specific splicing patterns in mice and Enhanced Integrated Gradients [ 83] for tissue-specific regulator identification, laid the conceptual foundations for context-aware variant effect prediction.”

“Variables with positive contributions increased the probability of predicting income reporting discontinuities, whereas negative contributions favored compliance predictions [36].”

“…systematically evaluated based on their discriminative importance for caste identity rather than expression magnitude alone, providing an interpretable ranking of key regulatory candidates and effectively complementing conventional differential expression and functional enrichment analyses [50,51].”

“…approaches such as integrated gradient calculation and game theory-based approaches such as Shapley Additive exPlanations (SHAP) are now applied to networks trained on genomics and epigenomic datasets (Lundberg and Lee 2017, Sundararajan et al. 2017, Jha et al. 2020, Majdandzic et al. 2023).”

“Despite the progress made in designing interpretable-by-design deep nets [21, 34, 42], post-hoc interpretation techniques are the most common strategy to study what the model learns (via input perturbations [40, 61] or attribution-based methods [28, 49, 52]).”

“…algorithms, such as LIME (Ribeiro, Singh, and Guestrin 2016), integrated gradients (Sundararajan, Taly, and Yan 2017; Miglani et al. 2020; Jha et al. 2020; Sanyal and Ren 2021; Enguehard 2023), and model-dependent meth-ods, such as LRP(Voita, Sennrich, and Titov 2021), VMASK (Chen and Ji…”

“In [4, 6, 11] methods, based on partial derivatives in ANN sensitivity analysis were proposed to calculate variable importance.”