Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 18 of the 30 sources its claims cite were checked for support.
Abstract
The integration of artificial intelligence (AI) in medical imaging has revolutionized diagnostic capabilities, yet the black-box nature of deep learning models poses significant challenges for clinical adoption. Current explainable AI (XAI) approaches, including SHAP, LIME, and Grad-CAM, predominantly focus on post hoc explanations that may inadvertently undermine clinical decision-making by providing misleading confidence in AI outputs. This paper presents a systematic review and meta-analysis of 67 studies (covering 23 radiology, 19 pathology, and 25 ophthalmology applications) evaluating XAI fidelity, stability, and performance trade-offs across medical imaging modalities. Our meta-analysis of 847 initially identified studies reveals that LIME achieves superior fidelity (0.81, 95% CI: 0.78-0.84) compared to SHAP (0.38, 95% CI: 0.35-0.41) and Grad-CAM (0.54, 95% CI: 0.51-0.57) across all modalities. Post hoc explanations demonstrated poor stability under noise perturbation, with SHAP showing 53% degradation in ophthalmology applications (ρ = 0.42 at 10% noise) compared to 11% in radiology (ρ = 0.89). We demonstrate a consistent 5-7% AUC performance penalty for interpretable models but identify modality-specific stability patterns suggesting that tailored XAI approaches are necessary. Based on these empirical findings, we propose a comprehensive three-pillar accountability framework that prioritizes transparency in model development, interpretability in architecture design, and a cautious deployment of post hoc explanations with explicit uncertainty quantification. This approach offers a pathway toward genuinely accountable AI systems that enhance rather than compromise clinical decision-making quality and patient safety.
Overview
CoChat Summary
TL;DR
This systematic review and meta-analysis of 67 medical imaging studies finds that post hoc XAI methods have low fidelity and instability, with LIME fidelity 0.81 vs SHAP 0.38 and Grad-CAM 0.54, and interpretable models incur a 5–7% AUC penalty. The authors propose a three-pillar accountability framework prioritizing transparency and interpretability, treating post hoc explainability as supplementary.
Key findings
- Across 67 studies (23 radiology, 19 pathology, 25 ophthalmology), LIME achieved superior fidelity (0.81, 95% CI: 0.78–0.84) compared with SHAP (0.38, 95% CI: 0.35–0.41) and Grad-CAM (0.54, 95% CI: 0.51–0.57).
- Stability under 10% noise was modality-specific: SHAP performed best in radiology (ρ = 0.89, 95% CI: 0.85–0.92), Grad-CAM in pathology (ρ = 0.82, 95% CI: 0.78–0.85), and SHAP poorly in ophthalmology (ρ = 0.42, 95% CI: 0.37–0.47); the abstract reports 53% degradation in ophthalmology vs 11% in radiology.
- Interpretable models showed a consistent AUC penalty: radiology −0.063 (95% CI: −0.089 to −0.037), pathology −0.058 (95% CI: −0.081 to −0.035), and ophthalmology −0.071 (95% CI: −0.098 to −0.044), all p < 0.001.
- The proposed three-pillar framework distinguishes transparency, interpretability, and explainability, with risk-stratified deployment: high-risk applications require interpretable models, 95% explainability coverage, and mandatory uncertainty quantification.
Methods
- PRISMA systematic review and meta-analysis of peer-reviewed studies published January 2019–December 2024 across PubMed/MEDLINE, IEEE Xplore, ACM Digital Library, and Web of Science; 847 papers identified, 142 met inclusion criteria, and 67 provided quantitative data for meta-analysis (κ = 0.89).
- Fidelity was assessed using the causal-based methodology of Alvarez-Melis & Jaakkola, measuring correlation between XAI-predicted feature importance and actual model behavior via occlusion; stability used Ghorbani et al.’s Gaussian noise perturbation protocol (0–30% of maximum image intensity) with Spearman rank correlation.
- Meta-analysis used inverse variance-weighted aggregation and random-effects models due to I² > 75%; performance trade-off analysis included 31 studies, with ROC curves reconstructed from sensitivity/specificity and AUC 95% CIs estimated by bootstrapping (n = 10,000).
Limitations
- Stated: English-language restriction may introduce language bias; significant heterogeneity (I² > 75%) limited direct comparisons and reduced precision; publication bias toward positive results may have inflated fidelity scores; cross-sectional analysis cannot establish causality; limited standardization of fidelity protocols; retrospective studies may not reflect real-world performance; the framework may be difficult in resource-constrained settings, and performance trade-offs may not be acceptable for all clinical applications.
- Stated: evolving fidelity/clinical utility metrics may not capture all relevant XAI performance aspects; scope limited to three imaging modalities; temporal limitations may not reflect rapidly evolving XAI methods.
- Evident from the text: the three-pillar framework and implementation protocol are conceptual recommendations derived from the review/meta-analysis, not prospectively validated in this paper; the meta-analysis relies on published study-level data rather than individual patient data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
9 Valid · 6 Not valid · 3 Not checked · 60 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“A critical yet often overlooked limitation of current XAI approaches is their failure to adequately communicate model uncertainty alongside explanations [35].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 4. Current State of XAI in Medical Imaging > 4.3. Model Uncertainty and Clinical Decision Confidence
What the sources say
- Could not verifycites [35] Singh, 2025, Deep learning-based uncertainty quantification for quality assurance in hepatobiliary imaging-based techniques, Oncotarget sourceExperimental — accuracy not yet measured
The Introduction’s clinical implementation section recommends reporting uncertainty metrics alongside imaging findings, but does not assess whether current XAI approaches fail to communicate uncertainty with explanations. Searches for XAI, explainability, and explanations alongside uncertainty found no passage making that claim. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“More critically, recent studies reveal that clinicians’ trust in AI recommendations is fundamentally linked to their ability to understand the underlying reasoning processes [9,10,11].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 1. Introduction
What the sources say
- Not checked — no full text of the sourcecites [9] Cabitza, 2017, Unintended consequences of machine learning in medicine, JAMA source
- Could not verifycites [10] Patel, 2019, Human–machine partnership with artificial intelligence for chest radiograph diagnosis, NPJ Digit. Med. sourceExperimental — accuracy not yet measured
The Introduction describes radiologists checking low-confidence AI outputs, and the Discussion reports improved diagnostic accuracy from combining radiologists with AI. Neither section addresses whether clinicians’ trust in AI recommendations depends on understanding the AI’s reasoning (searched for trust, explainability, interpretability, and related terms). (LLM, conf=0.75)
- Partly supportedcites [11] Lundberg, S.M., and Lee, S.I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst., 30. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says that correctly interpreting a model’s output engenders appropriate user trust, supporting the general link between interpretability and trust. It does not specifically address clinicians’ trust in AI recommendations, and the human-explanation experiment in §5.2 (p. 8) did not test clinicians or measure trust. (LLM, conf=0.86)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“More alarmingly, recent research indicates that explanations can reduce diagnostic accuracy by fostering overconfidence in incorrect AI predictions [16,20,21,22,23].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 1. Introduction
What the sources say
- Partly supportedcites [16] Ghassemi, 2021, The false hope of current approaches to explainable artificial intelligence in health care, Lancet Digit. Health sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says explainability techniques can hamper people’s ability to detect serious model mistakes and can unreasonably increase confidence in an algorithmic decision. It does not report that explanations reduced diagnostic accuracy, or establish overconfidence in incorrect AI predictions as the mechanism for such a reduction. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [20] Garapati, S., Swain, S.K., and Krishna, M.V. (2024, January 19–21). A Survey on Brain Tumor Segmentation Using Deep Learning Techniques. Proceedings of the 2024 2nd International Conference on Signal Processing, Communication, Power and Embedded System (SCOPES), Odisha, India. source
- Not checked — no full text of the sourcecites [21] Ukwuoma, C.C., Cai, D., Eziefuna, E.O., Oluwasanmi, A., Abdi, S.F., Muoka, G.W., Thomas, D., and Sarpong, K. (2025). Enhancing histopathological medical image classification for Early cancer diagnosis using deep learning and explainable AI–LIME & SHAP. Biomed. Signal Process. Control, 100. source
- Could not verifycites [22] Page, 2021, The PRISMA 2020 statement: An updated guideline for reporting systematic reviews, BMJ sourceExperimental — accuracy not yet measured
The Abstract discusses machine learning for identifying evidence in systematic reviews, and the Discussion addresses reporting guidelines; neither addresses diagnostic accuracy or overconfidence in incorrect AI predictions. Searches for the claim’s wording and for diagnostic accuracy, AI explanations, and overconfidence found no relevant passage. No page numbers were provided in our copy. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [23] Suh, 2020, Development and validation of an explainable artificial intelligence-based decision-supporting tool for prostate biopsy, BJU Int. source
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Explanation fidelity was evaluated using the established causal-based methodology by Alvarez-Melis & Jaakkola [14].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 2. Methods > 2.5. Fidelity Assessment Methodology
What the sources say
- Could not verifycites [14] Alvarez-Melis, D., and Jaakkola, T.S. (2018). On the robustness of interpretability methods. arXiv. sourceExperimental — accuracy not yet measured
Section 2, p. 2 describes a local Lipschitz measure for evaluating the robustness of explanations, not a causal-based method for evaluating fidelity. Searches for causal-based methodology and explanation fidelity found no passage in our copy establishing the claimed method. (LLM, conf=0.76)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While post hoc methods may highlight important features or attention regions, they rarely convey the model’s confidence boundaries or uncertainty estimates crucial for clinical decision-making [36].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 4. Current State of XAI in Medical Imaging > 4.3. Model Uncertainty and Clinical Decision Confidence
What the sources say
- Partly supportedcites [36] Kompa, 2021, Second opinion needed: Communicating uncertainty in medical machine learning, NPJ Digit. Med. sourceExperimental — accuracy not yet measured
The Introduction says uncertainty is not routinely quantified or communicated in medical ML, and “What is uncertainty quantification?” says most recent medical ML systems neglect predictive uncertainty. The paper also explains why uncertainty matters for clinical decisions. It does not establish the narrower claim that post hoc feature- or attention-based methods rarely convey confidence boundaries or uncertainty estimates. (LLM, conf=0.83)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
35 more claims · 2 with their cited sources checked
Create a free account to see all 78 claims and every source verdict.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 20
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By contrast, feature- or concept-based explanations appeared only in systems incorporating conventional predictive components amenable to attribution, such as saliency maps, concept bottleneck representations, or SHAP analyses over structured variables [31,32,33].”
From Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review (2026) · cites this paper as [31]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > Abstract
“LIME achieves superior fidelity (0.81, 95% CI: 0.78–0.84)”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 4. Current State of XAI in Medical Imaging > 4.2. Fundamental Limitations of Post Hoc Explanations
“Low Fidelity: Our meta-analysis confirms that LIME and SHAP approximations achieve only 30–40% fidelity to actual model behavior in medical imaging tasks.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > References
“7. US Food and Drug Administration Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices 2022”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > References
“(accessed on 12 January 2021)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 2. Methods > 2.2. Inclusion and Exclusion Criteria
“Studies focusing exclusively on image segmentation without classification components.”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 2. Methods > 2.7. Performance Trade-Off Analysis
“From the 67 studies meeting the inclusion criteria, 31 provided complete data for interpretability–performance trade-off analysis. We excluded 36 studies due to insufficient reporting of both interpretable and black-box model results (n = 19), the use of proprietary datasets preventing comparison validity (n = 8), focus solely on segmentation rather than classification tasks (n = 6), or the implementation of hybrid approaches that could not be clearly categorized (n = 3).”
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“…has been the rapid development of explainable AI (XAI) techniques, with methods like SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and Grad-CAM (Gradient-weighted Class Activation Mapping) becoming standard tools in medical AI research [11,12].”
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 1×
“These approaches promise to illuminate the decision-making processes of complex neural networks through post hoc analysis, offering feature importance rankings, saliency maps for visual interpretation, and local approximations of model behavior [13,14], as shown in Table 1.”
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“However, this remarkable capability comes with a fundamental trade-off: as models become more powerful, they simultaneously become more opaque, creating what researchers term the interpretability–performance paradox [5].”
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Cited by
Beyond single systems: how multi-agent ai is reshaping ethics in radiology
Salehi, S., Singh, Y., Habibi, P., & Erickson, B. J. (2025) · Bioengineering · doi:10.3390/bioengineering12101100 · Cited by 21
Self-citationMartinho, D., Sobreiro, P., Domingues, A., Martinho, F. I. F., & Nogueira, N. F. R. (2026) · Healthcare · doi:10.3390/healthcare14030287 · Cited by 22
Panayides, A. S., Chen, H., Filipović, N., Geroski, T., Hou, J., Lekadir, K., Marias, K., Matsopoulos, G. K., Papanastasiou, G., Sarder, P., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3649496 · Cited by 20
Enhancing the reliability of Alzheimer's disease prediction in MRI images
Islam, J., Furqon, E. N., Farady, I., Alex, J. S. R., Shih, C., Kuo, C.-C., & Lin, C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111111 · Cited by 11
Borra, S., Dey, N., Fong, S., Sherratt, R. S., & Shi, F. (2026) · Cancers · doi:10.3390/cancers18091361 · Cited by 9
Accuracy of artificial intelligence applications in periodontics: a thematic narrative review
Azhari, A. (2026) · Frontiers in Dental Medicine · doi:10.3389/fdmed.2026.1729825 · Cited by 5
Cherukuri, S., & Prasad, S. D. V. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1699960 · Cited by 10
Haupt, M., & Maurer, M. H. (2026) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2026.100098 · Cited by 8
Use of Automation Technologies and Data Mining in Speech Recognition for Autism
Mao, R., & Zhu, Y. (2026) · Brain and Behavior · doi:10.1002/brb3.71229 · Cited by 4
“(Mathew et al. 2025 ; Singh et al. 2025 ).”
Yang, Y., He, H., Yu, C., & Nia, P. S. (2026) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezag072 · Cited by 5
Liu, H., Meng, T., & Qie, R. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106245 · Cited by 6
Explainable Multi-Modal Deep Learning Framework for Enhancing Cyber-Physical Systems Security
Tumkur, S. D., Iyer, J., & Kurra, S. B. (2025) · IEEE International Conference on Cyber Security and Cloud Computing (CSCloud) · doi:10.1109/cscloud66326.2025.00015 · Cited by 3
Ali, M. B. G., & Smiti, A. (2026) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2026.102730 · Cited by 4
Ibrahim, Z. A., & Al-Sambawee, M. A. J. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-00956-3 · Cited by 3
Annamalai, A., Dhanes, V., Jayalakshmi, L., Rajeshkumar, S., & Ravi, S. (2026) · Cancer Treatment and Research Communications · doi:10.1016/j.ctarc.2026.101193 · Cited by 7
Chen, K., Xue, M., Meng, X., Li, G., Wang, X., Long, Y., & Li, H. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000004443 · Cited by 3
Machine Learning for Radiomics in Oncology: Challenges, Limitations, and Future Directions
Missaoui, R., Saadaoui, W., Coco, M. D., Helali, A., Leo, M., & Carcagnì, P. (2026) · Sensors · doi:10.3390/s26144619 · Cited by 2
“Probing techniques can reveal what biological information is encoded [62], but they cannot show how that information drives predictions or whether decision-relevant features reflect biology or pretraining artifacts [83,84].”
Song, J., Xu, Y., Zhu, Z., Zhou, J., He, Y., Chen, X. S., Liang, Y., Chen, W., Liu, Q., & Li, J. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1849346 · Cited by 1
“These fine-grained uncertainty signals also provide a natural basis for explaining model predictions, thereby addressing the interpretability requirements emphasized in recent surgical AI guidelines (41).”
Nasiri, A. M., Saleh, M. F. M., Alshammari, M., Alahmari, A. D., Alkharras, R., & Almohareb, F. I. (2026) · Hemoglobin · doi:10.1080/03630269.2026.2653733 · Cited by 2
“Other feature attribution methods, like Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), can be employed to quantify the contribution of specific clinical variables in predictive models [24].”
Mozolewski, M., Bayrak, B., Bach, K., & Nalepa, G. J. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10711-9 · Cited by 4
Optimizing Radiographic Diagnosis Through Signal-Balanced Convolutional Models
Neemuchwala, S., Ali, R. H., Abbas, Q., Khan, T. A., Shahnaz, A., & Ahmed, I. (2026) · Journal of Imaging · doi:10.3390/jimaging12030108 · Cited by 2
“Gradient-based methods such as Grad-CAM and its variants generate visual heatmaps that highlight image regions contributing most significantly to a model’s prediction [17,18].”
Traditional machine learning in biomedical image analysis: before you go too deep
Сергеевна, Ч. Е., Сергеевич, В. Н., Витальевич, С. М., Tyurin‐Kuzmin, P. A., & Kulebyakin, K. Y. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1695230 · Cited by 3
Artificial intelligence in biomarker discovery for diseases: diagnostic and therapeutic prospects
Dinç, R., & Ardıç, N. (2026) · Signal Transduction and Targeted Therapy · doi:10.1038/s41392-026-02946-4 · Cited by 1
TRAX: A framework for quantifying trustworthiness of explainable AI in medical image classification
Pisarcik, A., Hlavata, R., Kamencay, P., & Hudec, R. (2026) · Array · doi:10.1016/j.array.2026.101234
Fortaleza, J. A. G., Cabuhat, K. S. P., Argüello-Ramos, E. D., Cruz, J. T. P., Matamis, J. G., Mamaat, J. E. R., Libres, A. C., Leon, C. S., & Nuevo, J. J. M. (2026) · Frontiers in Microbiology · doi:10.3389/fmicb.2026.1892814
Biomarkers in axial spondyloarthritis diagnosis: from clinical signs to multi-omics integration
Xin, P., Xu, Z., Pan, Y., Zhang, M., & Wang, R. (2026) · Frontiers in Immunology · doi:10.3389/fimmu.2026.1929862
UbiQVision: Spatial Dempster–Shafer fusion of XAI attributions for reliable deep vision ensembles
Dubey, A., Anžel, A., İlgen, B., & Hattab, G. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.101000
“This empowers safer decision-making even when models encounter challenging or novel imaging inputs [79, 34].”
Wu, H., Yang, Y., Wang, L., Su, H., & Li, X. (2026) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70580
Diamantidis, D. E., Tsakaldimis, G., Smyrlis, N. V., Chousein, C., Kafalis, C., Foutzitzi, K., Deligeorgiou, E., Panagiotopoulos, N., Lailisidis, S., Georgellis, C., et al. (2026) · Current Urology · doi:10.1097/cu9.0000000000000367
An interpretable machine learning framework for pediatric appendicitis and peritonitis diagnosis
Dyuthishree, A., Shenoy, B., Sampathila, N., Ujwal, P., Prabhu, O., & Chadaga, K. (2026) · Healthcare Analytics · doi:10.1016/j.health.2026.100483
Explainable deep learning models for kidney stone classification using axial CT scans
Burki, M. Z., Naseem, R., Wajahat, A., Rizwan, M., & Muhammad, N. (2026) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-026-00856-7
“While gradient-based methods like Grad-CAM Selvaraju et al. ( 2020 ) are widely applied, standard saliency maps can sometimes highlight non-salient features and require independent clinical validation Singh et al. (2025).”
Joshi, P. (2026) · Spectroscopy Journal · doi:10.3390/spectroscj4030014
“These tools have advanced substantially, but they explain the model rather than expose an interpretable representation, and recent works caution that their visualizations should be read as illustrative of coherence rather than as verified causality [28,29].”
Lu, X., Zhao, G., Tang, Y., Li, F., & Cheng, M. M. H. (2026) · Current Proteomics · doi:10.1016/j.curpro.2026.100139 · Cited by 1
Vaseegar, S. S., Kumar, Y., V, Parthasarathy, S., et al. (2026) · doi:10.1109/icisc69558.2026.11681120
“Singh et al. evaluated the performance of different post-hoc explainability techniques, including Grad-CAM, LIME, and SHAP, and found major discrepancies in the accuracy of explanation localization across different model topologies [18].”
Zhu, Z., Wang, Z., Dong, C., et al. (2026) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2026.113193
Osman, H. A. (2026) · African Journal of Laboratory Medicine · doi:10.4102/ajlm.v15i1.3130
“57,58,60,61,62,63 In haematology, the application of explainable AI can enhance interpretability by clarifying how specific cellular features or molecular variants contribute to diagnostic outputs, thereby supporting clinician oversight rather than autonomous decision-making.”
Singh, A. K., Pandey, G., Kolipaka, T., & Srivastava, S. (2026) · BioNanoScience · doi:10.1007/s12668-026-02733-z · Cited by 2
“However, it is important to note that most XAI approaches are inherently post hoc and unlikely to reflect the true internal workings of an AI model accurately [82].”
Arun, A. P., Sreenivasan, N., & Sarode, M. (2026) · Food Engineering Reviews · doi:10.1007/s12393-026-09453-w · Cited by 4
Beyond the algorithm: nursing as a core foundation for ethical and equitable AI in healthcare
Sun, Y., Li, Y. D., Liu, Y., & Yu, S. (2026) · Cognition Technology & Work · doi:10.1007/s10111-026-00886-1
Yan, G., & Fan, C. (2026) · Clinical Anatomy · doi:10.1002/ca.70176 · Cited by 1
Explainable YOLOv11 for PCOS Detection from Ultrasound Images Using Grad-CAM
Purnomo, A. S. A., & Gunawan, P. H. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620608
Mao, H.-L., Wang, X.-Y., Wan, L., et al. (2026) · Diagnostics · doi:10.3390/diagnostics16131988
“More broadly, explainable AI approaches are increasingly important in determining whether a biomarker model relies on clinically meaningful signals rather than spurious associations [41].”
Whewell, J., Peters, R., & Kulon, J. (2026) · Technologies · doi:10.3390/technologies14070391 · Cited by 5
Li, L., Wang, J. B., Zhou, K., Li, Q., Wu, H., Song, H., & Fang, X. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1862470
“At the same time, post-hoc explanations remain limited by concerns regarding stability, faithfulness, and suitability for time-resolved clinical event detection (29).”
Li, S., Wang, L., Zhao, H., Yu, L., Yan, X. D., Han, L., Zhang, J., & Jiang, Y. X. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1859961 · Cited by 1
Fang, R., Xu, R., Wu, Z., & Tong, Y. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1845156
“Explainable artificial intelligence (XAI) can enhance diagnostic accuracy and doctors’ trust in applications based on UWF-CFP (70).”
Peña, A. D., Clariza, J., & Aballiar-Vista, M. A. (2026) · Transactions of the Institute of Aviation · doi:10.2478/tar-2026-0009
“Accountability frameworks further argue that transparency, interpretability, and explicit UQ are prerequisites for safe decision support instead of opaque automation [33].”
Burlacu, A., Olariu, M., Geman, O., Iftene, A., Goroftei, E. R. B., & Brinza, C. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1847389
Balancing Accuracy and Interpretability in Explainable Ai for Breast Disease Diagnosis
Abdigaliyev, Z., Shayakhmetov, A., Shpakovskiy, R., Kenbay, I., Omirgaliyev, R., & Makatov, A. (2026) · IEEE International Conference on Smart Information Systems and Technologies (SIST) · doi:10.1109/sist61674.2026.11596172
“A meta-analysis of 847 studies found that LIME achieved higher average accuracy (0.81, 95% CI: 0.78–0.84) than SHAP (0.38, 95% CI: 0.35–0.41) and Grad-CAM (0.54, 95% CI: 0.51–0.57) across medical imaging tasks, indicating that SHAP explanations may distort model logic in 55–68% of cases [7].”
InfluentialExplainable Deep Learning for Pediatric Brain Tumor Segmentation Using Attention U-Net
Aitkazin, M., Riza, T., Bolysbek, M., Lemma, A. A., Mengist, N. B., & Aukazhiyeva, Z. (2026) · IEEE International Conference on Smart Information Systems and Technologies (SIST) · doi:10.1109/sist61674.2026.11595967
“…reviews of explainable artificial intelligence (XAI) in oncology and medical imaging have further emphasized that post hoc methods such as Grad-CAM, SHAP, and LIME are increasingly used to improve transparency, but their clinical value depends on careful evaluation and responsible use [10, 11].”
Imaging and AI in tertiary prevention of lung cancer: Narrative review and clinical perspectives
Curci, C. M., Morabito, A., Pagliaro, R., Perrotta, F., Ottaiano, A., Santone, A., D’Agnano, V., Mormile, R., D'Errico, A. G., Bianco, A., et al. (2026) · Multidisciplinary Respiratory Medicine · doi:10.5826/mrm.2026.1089 · Cited by 1
Aravinda, C. V., Shaffi, N., Viswan, V., Al-Rahbi, A., Choy, K. W., Bouchareb, Y., & Sirasanagandla, S. R. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1807189
“SHAP (SHapley Additive exPlanations) methods have demonstrated superior fidelity in medical imaging applications, achieving 0.81 fidelity scores compared to 0.54 for Grad-CAM across multiple modalities (Singh et al., 2025).”
InfluentialYang, Y., Huang, D., Xie, Y., Qiu, L., Yan, J., & Liu, Z. (2026) · Journal of Biophotonics · doi:10.1002/jbio.70280
Yu, C., Jiang, L., Long, L., & Yu, H. (2026) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2026.1761811 · Cited by 1
Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review
Collaco, B. G., Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Wood, N. G., Gopala, N., Raman, R., Hester, E. O., & Forte, A. J. (2026) · Bioengineering · doi:10.3390/bioengineering13050513 · Cited by 3
“In conventional clinical machine learning, explainability has often centered on post hoc feature-attribution methods applied to static predictive models trained on imaging or structured electronic health record data [31,36].”
Exploring LIME and Its Variant to Explain Breast Cancer Image Classification in Neural Networks
Maity, S., Ummesalma, M., & Parihar, R. S. (2026) · International Conference on Artificial Intelligence and Machine Learning Applications Theme: Healthcare and Internet of Things (AIMLA) · doi:10.1109/aimla67915.2026.11522369
“The medical AI shows that both methods are effective in increasing transparency, but disparities in fidelity and stability have been noted across modalities [2, 6].”
Li, C.-C., Zhou, Z., Ji, Y., et al. (2026) · World Chinese Journal of Digestology · doi:10.11569/wcjd.v34.i3.183
Liu, C., Zhang, H., Jia, Z., et al. (2026) · BMC Medical Imaging · doi:10.1186/s12880-026-02253-y
“This characteristic is especially useful in medical imaging studies, offering a viable route to convert “non-semantic imaging features” into “clinically interpretable indicators,” thereby potentially advancing the clinical implementation of models [31, 32].”
Ha, J., Park, H., Shinn, J., & Kim, H. (2026) · Journal of Korean Medical Association · doi:10.5124/jkma.26.0013
“These findings are consistent with previous studies reporting that, although medical experts acknowledge potential benefits of medical AI, they repeatedly identify limited explainability and ambiguous responsibility as major barriers to acceptance [11–13].”
Van, S.-H., Tran, T.-K., Tran, H.-K., & Van, D.-N. (2026) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom69009.2026.11360872
Embedded Shift Evolution in Explainable Deep Learning Systems
Ziweritin, S., Gunasekaran, A., Iram, S., & Chen, M. (2026) · Studies in big data · doi:10.1007/978-3-032-21009-8_12 · Cited by 1
Current Research Trends for Early Screening of Lung Cancer Using AI
Shah, S., & Thanki, R. (2026) · Signals and communication technology · doi:10.1007/978-3-032-29615-3_4
Malik, I., & Agarwal, B. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-25809-0_1
Samsumar, L. D., Imran, B., Efendi, M. M., Wahyudi, E., Basri, H., Junianto, M. B. S., Supardiantob, S., & Yani, A. (2025) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis68501.2025.11326976
“…improve interpretability have largely relied on post-hoc methods such as Grad-CAM or attention rollout, which often suffer from low fidelity and do not guarantee alignment with medical biomarkers, as their explanations may not faithfully reflect the model’s actual decision-making mechanisms [15].”
Jiang, Y. (2025) · ITM Web of Conferences · doi:10.1051/itmconf/20258004001
Metadata
- Authors
- Singh, Y., Hathaway, Q. A., Keishing, V., Salehi, S., Wei, Y., Horvat, N., Vera-Garcia, D. V., Choudhary, A., Mula Kh, A., Quaia, E., & Andersen, J. B.
- Affiliations
- Mayo Clinic (US)
- University of Pennsylvania (US)
- Sheikh Shakhbout Medical City (SSMC), Abu Dhabi P.O. Box 11001, United Arab Emirates
- University of Padua (IT)
- University of Copenhagen (DK)
- Corresponding author
- Singh, Yashbir
- ORCID
- Venue
- Bioengineering
- Publisher
- MDPI AG
- Published
- 2025-08-15
- Volume
- 12
- Issue
- 8
- Pages
- 879
- ISSN
- 2306-5354 (ISSN-L)
- PMC
- PMC12383817
- OpenAlex
- W4413340328
- Cited by
- 72
- References
- 37
- Citations by year
- 2026: 63 · 2025: 9
- FWCI
- 6.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Bioengineering (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- explainable AI, diagnostic imaging, post-hoc explanation, SHAP, lime, Grad-CAM, noise perturbation, uncertainty quantification, interpretability, accountability framework, radiology, pathology, ophthalmology, clinical decision-making, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.