Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 18 of the 30 sources its claims cite were checked for support.

Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability

Open access

Abstract

The integration of artificial intelligence (AI) in medical imaging has revolutionized diagnostic capabilities, yet the black-box nature of deep learning models poses significant challenges for clinical adoption. Current explainable AI (XAI) approaches, including SHAP, LIME, and Grad-CAM, predominantly focus on post hoc explanations that may inadvertently undermine clinical decision-making by providing misleading confidence in AI outputs. This paper presents a systematic review and meta-analysis of 67 studies (covering 23 radiology, 19 pathology, and 25 ophthalmology applications) evaluating XAI fidelity, stability, and performance trade-offs across medical imaging modalities. Our meta-analysis of 847 initially identified studies reveals that LIME achieves superior fidelity (0.81, 95% CI: 0.78-0.84) compared to SHAP (0.38, 95% CI: 0.35-0.41) and Grad-CAM (0.54, 95% CI: 0.51-0.57) across all modalities. Post hoc explanations demonstrated poor stability under noise perturbation, with SHAP showing 53% degradation in ophthalmology applications (ρ = 0.42 at 10% noise) compared to 11% in radiology (ρ = 0.89). We demonstrate a consistent 5-7% AUC performance penalty for interpretable models but identify modality-specific stability patterns suggesting that tailored XAI approaches are necessary. Based on these empirical findings, we propose a comprehensive three-pillar accountability framework that prioritizes transparency in model development, interpretability in architecture design, and a cautious deployment of post hoc explanations with explicit uncertainty quantification. This approach offers a pathway toward genuinely accountable AI systems that enhance rather than compromise clinical decision-making quality and patient safety.

Overview

CoChat Summary

TL;DR

This systematic review and meta-analysis of 67 medical imaging studies finds that post hoc XAI methods have low fidelity and instability, with LIME fidelity 0.81 vs SHAP 0.38 and Grad-CAM 0.54, and interpretable models incur a 5–7% AUC penalty. The authors propose a three-pillar accountability framework prioritizing transparency and interpretability, treating post hoc explainability as supplementary.

Key findings

  • Across 67 studies (23 radiology, 19 pathology, 25 ophthalmology), LIME achieved superior fidelity (0.81, 95% CI: 0.78–0.84) compared with SHAP (0.38, 95% CI: 0.35–0.41) and Grad-CAM (0.54, 95% CI: 0.51–0.57).
  • Stability under 10% noise was modality-specific: SHAP performed best in radiology (ρ = 0.89, 95% CI: 0.85–0.92), Grad-CAM in pathology (ρ = 0.82, 95% CI: 0.78–0.85), and SHAP poorly in ophthalmology (ρ = 0.42, 95% CI: 0.37–0.47); the abstract reports 53% degradation in ophthalmology vs 11% in radiology.
  • Interpretable models showed a consistent AUC penalty: radiology −0.063 (95% CI: −0.089 to −0.037), pathology −0.058 (95% CI: −0.081 to −0.035), and ophthalmology −0.071 (95% CI: −0.098 to −0.044), all p < 0.001.
  • The proposed three-pillar framework distinguishes transparency, interpretability, and explainability, with risk-stratified deployment: high-risk applications require interpretable models, 95% explainability coverage, and mandatory uncertainty quantification.

Methods

  • PRISMA systematic review and meta-analysis of peer-reviewed studies published January 2019–December 2024 across PubMed/MEDLINE, IEEE Xplore, ACM Digital Library, and Web of Science; 847 papers identified, 142 met inclusion criteria, and 67 provided quantitative data for meta-analysis (κ = 0.89).
  • Fidelity was assessed using the causal-based methodology of Alvarez-Melis & Jaakkola, measuring correlation between XAI-predicted feature importance and actual model behavior via occlusion; stability used Ghorbani et al.’s Gaussian noise perturbation protocol (0–30% of maximum image intensity) with Spearman rank correlation.
  • Meta-analysis used inverse variance-weighted aggregation and random-effects models due to I² > 75%; performance trade-off analysis included 31 studies, with ROC curves reconstructed from sensitivity/specificity and AUC 95% CIs estimated by bootstrapping (n = 10,000).

Limitations

  • Stated: English-language restriction may introduce language bias; significant heterogeneity (I² > 75%) limited direct comparisons and reduced precision; publication bias toward positive results may have inflated fidelity scores; cross-sectional analysis cannot establish causality; limited standardization of fidelity protocols; retrospective studies may not reflect real-world performance; the framework may be difficult in resource-constrained settings, and performance trade-offs may not be acceptable for all clinical applications.
  • Stated: evolving fidelity/clinical utility metrics may not capture all relevant XAI performance aspects; scope limited to three imaging modalities; temporal limitations may not reflect rapidly evolving XAI methods.
  • Evident from the text: the three-pillar framework and implementation protocol are conceptual recommendations derived from the review/meta-analysis, not prospectively validated in this paper; the meta-analysis relies on published study-level data rather than individual patient data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/bioengineering12080879

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 PRISMA 2020 flow diagram showing systematic review methodology for explainable AI in medical imaging. The flow diagram illustrates the identification, screening, eligibility assessment, and inclusion of studies from initial database searches through final analysis.
Figure 1 PRISMA 2020 flow diagram showing systematic review methodology for explainable AI in medical imaging. The flow diagram illustrates the identification, screening, eligibility assessment, and inclusion of studies from initial database searches through final analysis.CC BY · PubMed Central
Figure 2 Multi-modal assessment of explainable AI methods: fidelity, stability, and performance trade-offs in medical imaging applications. This comprehensive analysis evaluates three prominent explainable AI (XAI) methods—SHAP, LIME, and Grad-CAM—across medical imaging modalities based on a systematic review of 67 studies. (A) Fidelity performance Across modalities, (B) Stability under 10% Noise (p-value), (C) Interpretability performance Trade-off.
Figure 2 Multi-modal assessment of explainable AI methods: fidelity, stability, and performance trade-offs in medical imaging applications. This comprehensive analysis evaluates three prominent explainable AI (XAI) methods—SHAP, LIME, and Grad-CAM—across medical imaging modalities based on a systematic review of 67 studies. (A) Fidelity performance Across modalities, (B) Stability under 10% Noise (p-value), (C) Interpretability performance Trade-off.CC BY · PubMed Central
Figure 3 Overview of dominant explainable AI (XAI) approaches in medical imaging. Four primary XAI methodologies used in medical imaging applications, categorized by their explanation mechanisms. Attribution methods (SHAP and LIME) generate feature importance scores to identify image regions that strongly influence model predictions.
Figure 3 Overview of dominant explainable AI (XAI) approaches in medical imaging. Four primary XAI methodologies used in medical imaging applications, categorized by their explanation mechanisms. Attribution methods (SHAP and LIME) generate feature importance scores to identify image regions that strongly influence model predictions.CC BY · PubMed Central
Figure 4 A comprehensive three-pillar framework for medical AI accountability with risk-stratified implementation guidelines. This framework illustrates a holistic approach to medical AI accountability through three interconnected pillars, namely transparency (blue), interpretability (green), and explainability (orange). The decision tree demonstrates how accountability measures should be scaled according to clinical risk levels, with high-risk applications (red) requiring the most stringent oversight including real-time monitoring and ethics review, medium-risk applications (orange) needing moderate measures, and low-risk applications (green) requiring basic documentation. The bottom panel contrasts traditional XAI approaches—which focus primarily on technical interpretability—with the proposed comprehensive framework that emphasizes clinical orientation, multi-stakeholder engagement, and built-in regulatory compliance. The evolution arrow indicates the necessary transition from algorithm-centric explanations to patient-centered accountability systems in medical AI deployment.
Figure 4 A comprehensive three-pillar framework for medical AI accountability with risk-stratified implementation guidelines. This framework illustrates a holistic approach to medical AI accountability through three interconnected pillars, namely transparency (blue), interpretability (green), and explainability (orange). The decision tree demonstrates how accountability measures should be scaled according to clinical risk levels, with high-risk applications (red) requiring the most stringent oversight including real-time monitoring and ethics review, medium-risk applications (orange) needing moderate measures, and low-risk applications (green) requiring basic documentation. The bottom panel contrasts traditional XAI approaches—which focus primarily on technical interpretability—with the proposed comprehensive framework that emphasizes clinical orientation, multi-stakeholder engagement, and built-in regulatory compliance. The evolution arrow indicates the necessary transition from algorithm-centric explanations to patient-centered accountability systems in medical AI deployment.CC BY · PubMed Central

Verification

Claims the paper makes

9 Valid · 6 Not valid · 3 Not checked · 60 Own results and interpretations

35 more claims · 2 with their cited sources checked

Create a free account to see all 78 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 24
  • Probably not a substantial claim. 20

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 37 references

Cited by

See all 72

Metadata

Authors
Singh, Y., Hathaway, Q. A., Keishing, V., Salehi, S., Wei, Y., Horvat, N., Vera-Garcia, D. V., Choudhary, A., Mula Kh, A., Quaia, E., & Andersen, J. B.
Affiliations
  • Mayo Clinic (US)
  • University of Pennsylvania (US)
  • Sheikh Shakhbout Medical City (SSMC), Abu Dhabi P.O. Box 11001, United Arab Emirates
  • University of Padua (IT)
  • University of Copenhagen (DK)
Corresponding author
Singh, Yashbir
Venue
Bioengineering
Publisher
MDPI AG
Published
2025-08-15
Volume
12
Issue
8
Pages
879
ISSN
2306-5354 (ISSN-L)
OpenAlex
W4413340328
Cited by
72
References
37
Citations by year
2026: 63 · 2025: 9
FWCI
6.87
Type
journal article
Language
en
Open access
Locations
  • Bioengineering (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Research at the University of Copenhagen (University of Copenhagen) (submittedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
explainable AI, diagnostic imaging, post-hoc explanation, SHAP, lime, Grad-CAM, noise perturbation, uncertainty quantification, interpretability, accountability framework, radiology, pathology, ophthalmology, clinical decision-making, model transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability. https://app.cochat.ai/papers/10.3390/bioengineering12080879
The paper
Singh, Y., Hathaway, Q. A., Keishing, V., Salehi, S., Wei, Y., Horvat, N., Vera-Garcia, D. V., Choudhary, A., Mula Kh, A., Quaia, E., & Andersen, J. B. (2025). Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability. Bioengineering. https://doi.org/10.3390/bioengineering12080879

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.