Signal or Noise? Evaluating Commonly Used Attribution Methods for Explaining Deep Neural Networks in Electrocardiogram Classification
Open accessAbstract
Aims: Attribution-based explainability methods are widely used in electrocardiogram (ECG) analysis to interpret predictions from 'black-box' deep neural networks (DNNs). To be useful in clinical applications, attribution methods must produce explanations that are both clear and reflective of the model's inner workings. This study evaluates 12 attribution methods in DNN-based ECG classification. Methods and results: We analysed 12 attribution methods using a dataset of 873 710 median beat ECGs spanning nine diagnostic classes. Methods were applied to convolutional neural network-based models trained for ECG classification. Performance was evaluated across four experiments: inter-method similarity, self-consistency, dependence on model weights, and ability to identify features important for model inference. All task models achieved an area under the receiver operating curve above 0.95. Attribution methods demonstrated low correlation and high variability across inter-method comparisons. Self-consistency across random model initializations was moderate for most methods (mean correlation 0.41-0.65). Randomizing model weights led to rapid loss of correlation, although some methods did not converge to zero. Perturbation of input data revealed differences in how well attribution methods identified features relevant to model performance. Conclusion: Attribution methods demonstrated limited reliability, instability across model variants and incomplete dependence on learned parameters, constraining their utility in high-stakes settings such as healthcare. These findings suggest that attribution techniques should be used cautiously and supported by task-specific sanity checks. Approaches grounded in rigorous validation, inherently interpretable modelling or counterfactual explanations may better support clinically meaningful insight.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work evaluating attribution methods for ECG classification has emphasized the need for sanity checks, rigorous validation, and caution when using post-hoc explanations in high-stakes clinical settings.
“Recent work evaluating attribution methods for ECG classification has emphasized the need for sanity checks, rigorous validation, and caution when using post-hoc explanations in high-stakes clinical settings [50].”
From Deep Learning Applied to 12-Lead ECGs for Detection of Structural, Metabolic and Systemic Disease (2026) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · see citing works
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · see citing works
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019) · Nature Medicine · doi:10.1038/s41591-018-0268-3 · Cited by 3,148 · see citing works
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · see citing works
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 2× · see citing works
“4 This ambiguity limits clinical interpretability.”
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · see citing works
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811 · see citing works
Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging
Arun, N., Gaw, N., Singh, P., Chang, K., Aggarwal, M., Chen, B., Hoebel, K., Gupta, S., Patel, J., Gidwani, M., et al. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021200267 · Cited by 342 · see citing works
Attia, Z. I., Kapa, S., Yao, X., López-Jiménez, F. A., Mohan, T. L., Pellikka, P. A., Carter, R. E., Shah, N. D., Friedman, P. A., & Noseworthy, P. A. (2019) · Journal of Cardiovascular Electrophysiology · doi:10.1111/jce.13889 · Cited by 195 · see citing works
A review of evaluation approaches for explainable AI with applications in cardiology
Salih, A. M., Galazzo, I. B., Gkontra, P., Rauseo, E., Lee, A. M., Lekadir, K., Radeva, P. I., Petersen, S. E., & Menegaz, G. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10852-w · Cited by 77 · see citing works
Automatic Triage of 12‐Lead ECGs Using Deep Convolutional Neural Networks
van de Leur, R. R., Blom, L. J., Gavves, E., Hof, I. E., van der Heijden, J. F., Clappers, N., Doevendans, P. A., Hassink, R. J., & van Es, R. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.015138 · Cited by 67 · see citing works
Wouters, P. C., van de Leur, R. R., Vessies, M. B., van Stipdonk, A. M. W., Ghossein, M. A., Hassink, R. J., Doevendans, P. A., van der Harst, P., Maass, A. H., Prinzen, F. W., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac617 · Cited by 58 · see citing works
Goettling, M., Hammer, A., Malberg, H., & Schmidt, M. U. (2024) · Scientific Reports · doi:10.1038/s41598-024-63656-x · Cited by 51 · see citing works
Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
Wagner, P., Mehari, T., Haverkamp, W., & Strodthoff, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108525 · Cited by 48 · see citing works
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · MethodsX · doi:10.1016/j.mex.2023.102009 · Cited by 33 · see citing works
Hempel, P., Ribeiro, A. H., Vollmer, M., Bender, T., Dörr, M., Krefting, D., & Spicher, N. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01428-7 · Cited by 31 · see citing works
Avula, V. R., Wu, K. C., & Carrick, R. T. (2023) · JACC Advances · doi:10.1016/j.jacadv.2023.100686 · Cited by 28 · see citing works
A novel XAI framework for explainable AI-ECG using generative counterfactual XAI (GCX)
Jang, J.-H., Jo, Y., Kang, S., Son, J. M., Lee, H. S., Kwon, J., & Lee, M. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-08080-5 · Cited by 23 · see citing works
Riek, N. T., Gokhale, T., Martin‐Gill, C., Kraevsky-Phillips, K., Zègre‐Hemsey, J. K., Saba, S., Callaway, C. W., Akçakaya, M., & Al‐Zaiti, S. S. (2024) · Journal of Electrocardiology · doi:10.1016/j.jelectrocard.2024.153792 · Cited by 11 · see citing works
Suh, J., Kim, J., Kwon, S., Jung, E., Ahn, H., Lee, K.-Y., Choi, E., & Rhee, W. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109088 · Cited by 9 · see citing works
Evaluating gradient-based explanation methods for neural network ECG analysis using heatmaps
Storås, A. M., Maeland, S., Isaksen, J. L., Hicks, S. A., Thambawita, V. L., Graff, C., Hammer, H. L., Halvorsen, P., Riegler, M. A., & Kanters, J. K. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae280 · Cited by 5 · see citing works
Interpretable ECG classification via a query-based latent space traversal (qLST)Preprint
Vessies, M. B., Vadgama, S., van de Leur, R. R., Doevendans, P. A., Hassink, R. J., Bekkers, E. J., & van Es, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07386 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [1]Somani S, Russak AJ, Richter F, Zhao S, Vaid A, Chaudhry F, et al Deep learning and the electrocardiogram: review of the current state-of-the-art. EP Eur 2021; 23: 1179– 1191. PMC8350862 33564873
- [9]Adebayo J, Gilmer J, Muelly M, Goodfellow I, Hardt M, Kim B. Sanity checks for saliency maps. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, eds. Advances in Neural Information Processing Systems. Vol. 31. Montreal: NeurIPS; 2018. p1–11.
- [10]Konate S, Lebrat L, Cruz RS, Smith E, Bradley A, Fookes C, et al A comparison of saliency methods for deep learning explainability. In: 2021 Digital Image Computing: Techniques and Applications (DICTA). Gold Coast: IEEE; 2021. p 1– 8.
- [23]Lundberg SM, Lee SI. A unified approach to interpreting model predictions. In: Guyon I, Luxburg UV, Bengio S, eds. Advances in Neural Information Processing Systems. Vol. 30. Long Beach: NeurIPS; 2017. p 1– 10.
- [25]Rong Y, Leemann T, Borisov V, Kasneci G, Kasneci E. A consistent and efficient evaluation strategy for attribution methods. Proc Mach Learn Res 2022; 162: 1– 26.
- [31]Sethi S, Chen D, Statchen T, Burkhart MC, Bhandari N, Ramadan B, et al ProtoECGNet: case-based interpretable deep learning for multi-label ECG classification with contrastive learning. Proc Mach Learn Res 2025; 298: 1– 30. PMC12700622 41394314
Cited by
Sorted by citations
Deep Learning Applied to 12-Lead ECGs for Detection of Structural, Metabolic and Systemic Disease
Shaulian, S., Zeltser, R., & Makaryus, A. N. (2026) · Diagnostics · doi:10.3390/diagnostics16172834
Liu, E., Liu, Y., Wang, Z., Zhou, H., & Wang, X. F. (2026) · Physiological Measurement · doi:10.1088/1361-6579/ae6750
From signal to bedside: the missing link between ECG-AI explainability and clinical trust
Bhardwaj, A., Vasudev, I., & Jayaraman, P. (2026) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztag092
Sato, E., Izumi, Y., Saito, Y., Saijo, Y., Kitai, T., Hayashida, A., Murata, A., Kaneko, T., Sakamoto, A., Sakamoto, T., et al. (2026) · NEJM AI · doi:10.1056/aioa2600258
Yeo, T. X. J., Wang, L. R., Cheong, M. Y. Y., Xu, X., Shanmugam, H., Hew, Y. Y., Chiang, P. W. E., Ang, X. W., Balwan, R., Latiff, S. T. B. A., et al. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103560
Ali, M. N. B., & Mahmoud, M. (2026) · medRxiv · doi:10.64898/2026.10.17.26365795
- Authors
- Arends, B. K. O., van Amsterdam, W. A. C., van der Harst, P., van Smeden, M., van Es, R., & van de Leur, R. R.
- Affiliations
- University Medical Center Utrecht (NL)
- Department of Data Science and Biostatistics, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University , Utrecht ,
- Department of Cardiology, University Medical Center Utrecht , Internal ref E03.511, Heidelberglaan 100, 3584 CX Utrecht ,
- ORCID
- Venue
- European Heart Journal - Digital Health
- Publisher
- Oxford University Press (OUP)
- Published
- 2026-03-04
- Volume
- 7
- Issue
- 3
- Pages
- ztag038-ztag038
- ISSN
- 2634-3916 (ISSN-L)
- PMC
- PMC12980500
- OpenAlex
- W7134913262
- Cited by
- 6
- References
- 26
- Citations by year
- 2026: 6
- FWCI
- 20.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- European Heart Journal - Digital Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- Dutch Research Council (628.011.213)
- Netherlands Organisation for Health Research and Development (104021004)
- Dutch Heart Foundation (2019B011, 2022TKI2322)
- Health∼Holland
- Top Sector Life Sciences & Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › ECG Monitoring and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- attribution methods, explainable AI, ECG classification, deep neural network, convolutional neural networks, model interpretability, self-consistency, input perturbation, sanity checks, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
