Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 18 of the 30 sources its claims cite were checked for support.
Abstract
The integration of artificial intelligence (AI) in medical imaging has revolutionized diagnostic capabilities, yet the black-box nature of deep learning models poses significant challenges for clinical adoption. Current explainable AI (XAI) approaches, including SHAP, LIME, and Grad-CAM, predominantly focus on post hoc explanations that may inadvertently undermine clinical decision-making by providing misleading confidence in AI outputs. This paper presents a systematic review and meta-analysis of 67 studies (covering 23 radiology, 19 pathology, and 25 ophthalmology applications) evaluating XAI fidelity, stability, and performance trade-offs across medical imaging modalities. Our meta-analysis of 847 initially identified studies reveals that LIME achieves superior fidelity (0.81, 95% CI: 0.78-0.84) compared to SHAP (0.38, 95% CI: 0.35-0.41) and Grad-CAM (0.54, 95% CI: 0.51-0.57) across all modalities. Post hoc explanations demonstrated poor stability under noise perturbation, with SHAP showing 53% degradation in ophthalmology applications (ρ = 0.42 at 10% noise) compared to 11% in radiology (ρ = 0.89). We demonstrate a consistent 5-7% AUC performance penalty for interpretable models but identify modality-specific stability patterns suggesting that tailored XAI approaches are necessary. Based on these empirical findings, we propose a comprehensive three-pillar accountability framework that prioritizes transparency in model development, interpretability in architecture design, and a cautious deployment of post hoc explanations with explicit uncertainty quantification. This approach offers a pathway toward genuinely accountable AI systems that enhance rather than compromise clinical decision-making quality and patient safety.
Overview
CoChat Summary
TL;DR
This systematic review and meta-analysis of 67 medical imaging studies finds that post hoc XAI methods have low fidelity and instability, with LIME fidelity 0.81 vs SHAP 0.38 and Grad-CAM 0.54, and interpretable models incur a 5–7% AUC penalty. The authors propose a three-pillar accountability framework prioritizing transparency and interpretability, treating post hoc explainability as supplementary.
Key findings
- Across 67 studies (23 radiology, 19 pathology, 25 ophthalmology), LIME achieved superior fidelity (0.81, 95% CI: 0.78–0.84) compared with SHAP (0.38, 95% CI: 0.35–0.41) and Grad-CAM (0.54, 95% CI: 0.51–0.57).
- Stability under 10% noise was modality-specific: SHAP performed best in radiology (ρ = 0.89, 95% CI: 0.85–0.92), Grad-CAM in pathology (ρ = 0.82, 95% CI: 0.78–0.85), and SHAP poorly in ophthalmology (ρ = 0.42, 95% CI: 0.37–0.47); the abstract reports 53% degradation in ophthalmology vs 11% in radiology.
- Interpretable models showed a consistent AUC penalty: radiology −0.063 (95% CI: −0.089 to −0.037), pathology −0.058 (95% CI: −0.081 to −0.035), and ophthalmology −0.071 (95% CI: −0.098 to −0.044), all p < 0.001.
- The proposed three-pillar framework distinguishes transparency, interpretability, and explainability, with risk-stratified deployment: high-risk applications require interpretable models, 95% explainability coverage, and mandatory uncertainty quantification.
Methods
- PRISMA systematic review and meta-analysis of peer-reviewed studies published January 2019–December 2024 across PubMed/MEDLINE, IEEE Xplore, ACM Digital Library, and Web of Science; 847 papers identified, 142 met inclusion criteria, and 67 provided quantitative data for meta-analysis (κ = 0.89).
- Fidelity was assessed using the causal-based methodology of Alvarez-Melis & Jaakkola, measuring correlation between XAI-predicted feature importance and actual model behavior via occlusion; stability used Ghorbani et al.’s Gaussian noise perturbation protocol (0–30% of maximum image intensity) with Spearman rank correlation.
- Meta-analysis used inverse variance-weighted aggregation and random-effects models due to I² > 75%; performance trade-off analysis included 31 studies, with ROC curves reconstructed from sensitivity/specificity and AUC 95% CIs estimated by bootstrapping (n = 10,000).
Limitations
- Stated: English-language restriction may introduce language bias; significant heterogeneity (I² > 75%) limited direct comparisons and reduced precision; publication bias toward positive results may have inflated fidelity scores; cross-sectional analysis cannot establish causality; limited standardization of fidelity protocols; retrospective studies may not reflect real-world performance; the framework may be difficult in resource-constrained settings, and performance trade-offs may not be acceptable for all clinical applications.
- Stated: evolving fidelity/clinical utility metrics may not capture all relevant XAI performance aspects; scope limited to three imaging modalities; temporal limitations may not reflect rapidly evolving XAI methods.
- Evident from the text: the three-pillar framework and implementation protocol are conceptual recommendations derived from the review/meta-analysis, not prospectively validated in this paper; the meta-analysis relies on published study-level data rather than individual patient data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
9 Valid · 6 Not valid · 3 Not checked · 60 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“A critical yet often overlooked limitation of current XAI approaches is their failure to adequately communicate model uncertainty alongside explanations [35].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 4. Current State of XAI in Medical Imaging > 4.3. Model Uncertainty and Clinical Decision Confidence
What the sources say
- Could not verifycites [35] Singh, 2025, Deep learning-based uncertainty quantification for quality assurance in hepatobiliary imaging-based techniques, Oncotarget sourceExperimental — accuracy not yet measured
The Introduction’s clinical implementation section recommends reporting uncertainty metrics alongside imaging findings, but does not assess whether current XAI approaches fail to communicate uncertainty with explanations. Searches for XAI, explainability, and explanations alongside uncertainty found no passage making that claim. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“More critically, recent studies reveal that clinicians’ trust in AI recommendations is fundamentally linked to their ability to understand the underlying reasoning processes [9,10,11].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 1. Introduction
What the sources say
- Not checked — no full text of the sourcecites [9] Cabitza, 2017, Unintended consequences of machine learning in medicine, JAMA source
- Could not verifycites [10] Patel, 2019, Human–machine partnership with artificial intelligence for chest radiograph diagnosis, NPJ Digit. Med. sourceExperimental — accuracy not yet measured
The Introduction describes radiologists checking low-confidence AI outputs, and the Discussion reports improved diagnostic accuracy from combining radiologists with AI. Neither section addresses whether clinicians’ trust in AI recommendations depends on understanding the AI’s reasoning (searched for trust, explainability, interpretability, and related terms). (LLM, conf=0.75)
- Partly supportedcites [11] Lundberg, S.M., and Lee, S.I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst., 30. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says that correctly interpreting a model’s output engenders appropriate user trust, supporting the general link between interpretability and trust. It does not specifically address clinicians’ trust in AI recommendations, and the human-explanation experiment in §5.2 (p. 8) did not test clinicians or measure trust. (LLM, conf=0.86)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“More alarmingly, recent research indicates that explanations can reduce diagnostic accuracy by fostering overconfidence in incorrect AI predictions [16,20,21,22,23].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 1. Introduction
What the sources say
- Partly supportedcites [16] Ghassemi, 2021, The false hope of current approaches to explainable artificial intelligence in health care, Lancet Digit. Health sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says explainability techniques can hamper people’s ability to detect serious model mistakes and can unreasonably increase confidence in an algorithmic decision. It does not report that explanations reduced diagnostic accuracy, or establish overconfidence in incorrect AI predictions as the mechanism for such a reduction. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [20] Garapati, S., Swain, S.K., and Krishna, M.V. (2024, January 19–21). A Survey on Brain Tumor Segmentation Using Deep Learning Techniques. Proceedings of the 2024 2nd International Conference on Signal Processing, Communication, Power and Embedded System (SCOPES), Odisha, India. source
- Not checked — no full text of the sourcecites [21] Ukwuoma, C.C., Cai, D., Eziefuna, E.O., Oluwasanmi, A., Abdi, S.F., Muoka, G.W., Thomas, D., and Sarpong, K. (2025). Enhancing histopathological medical image classification for Early cancer diagnosis using deep learning and explainable AI–LIME & SHAP. Biomed. Signal Process. Control, 100. source
- Could not verifycites [22] Page, 2021, The PRISMA 2020 statement: An updated guideline for reporting systematic reviews, BMJ sourceExperimental — accuracy not yet measured
The Abstract discusses machine learning for identifying evidence in systematic reviews, and the Discussion addresses reporting guidelines; neither addresses diagnostic accuracy or overconfidence in incorrect AI predictions. Searches for the claim’s wording and for diagnostic accuracy, AI explanations, and overconfidence found no relevant passage. No page numbers were provided in our copy. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [23] Suh, 2020, Development and validation of an explainable artificial intelligence-based decision-supporting tool for prostate biopsy, BJU Int. source
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Explanation fidelity was evaluated using the established causal-based methodology by Alvarez-Melis & Jaakkola [14].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 2. Methods > 2.5. Fidelity Assessment Methodology
What the sources say
- Could not verifycites [14] Alvarez-Melis, D., and Jaakkola, T.S. (2018). On the robustness of interpretability methods. arXiv. sourceExperimental — accuracy not yet measured
Section 2, p. 2 describes a local Lipschitz measure for evaluating the robustness of explanations, not a causal-based method for evaluating fidelity. Searches for causal-based methodology and explanation fidelity found no passage in our copy establishing the claimed method. (LLM, conf=0.76)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While post hoc methods may highlight important features or attention regions, they rarely convey the model’s confidence boundaries or uncertainty estimates crucial for clinical decision-making [36].”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 4. Current State of XAI in Medical Imaging > 4.3. Model Uncertainty and Clinical Decision Confidence
What the sources say
- Partly supportedcites [36] Kompa, 2021, Second opinion needed: Communicating uncertainty in medical machine learning, NPJ Digit. Med. sourceExperimental — accuracy not yet measured
The Introduction says uncertainty is not routinely quantified or communicated in medical ML, and “What is uncertainty quantification?” says most recent medical ML systems neglect predictive uncertainty. The paper also explains why uncertainty matters for clinical decisions. It does not establish the narrower claim that post hoc feature- or attention-based methods rarely convey confidence boundaries or uncertainty estimates. (LLM, conf=0.83)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
35 more claims · 2 with their cited sources checked
Create a free account to see all 78 claims and every source verdict.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 20
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By contrast, feature- or concept-based explanations appeared only in systems incorporating conventional predictive components amenable to attribution, such as saliency maps, concept bottleneck representations, or SHAP analyses over structured variables [31,32,33].”
From Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review (2026) · cites this paper as [31]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > Abstract
“LIME achieves superior fidelity (0.81, 95% CI: 0.78–0.84)”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 4. Current State of XAI in Medical Imaging > 4.2. Fundamental Limitations of Post Hoc Explanations
“Low Fidelity: Our meta-analysis confirms that LIME and SHAP approximations achieve only 30–40% fidelity to actual model behavior in medical imaging tasks.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > References
“7. US Food and Drug Administration Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices 2022”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > References
“(accessed on 12 January 2021)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 2. Methods > 2.2. Inclusion and Exclusion Criteria
“Studies focusing exclusively on image segmentation without classification components.”
§ Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability > 2. Methods > 2.7. Performance Trade-Off Analysis
“From the 67 studies meeting the inclusion criteria, 31 provided complete data for interpretability–performance trade-off analysis. We excluded 36 studies due to insufficient reporting of both interpretable and black-box model results (n = 19), the use of proprietary datasets preventing comparison validity (n = 8), focus solely on segmentation rather than classification tasks (n = 6), or the implementation of hybrid approaches that could not be clearly categorized (n = 3).”
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“…has been the rapid development of explainable AI (XAI) techniques, with methods like SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and Grad-CAM (Gradient-weighted Class Activation Mapping) becoming standard tools in medical AI research [11,12].”
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 1×
“These approaches promise to illuminate the decision-making processes of complex neural networks through post hoc analysis, offering feature importance rankings, saliency maps for visual interpretation, and local approximations of model behavior [13,14], as shown in Table 1.”
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“However, this remarkable capability comes with a fundamental trade-off: as models become more powerful, they simultaneously become more opaque, creating what researchers term the interpretability–performance paradox [5].”
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111 · cited 1×
“A recent study by Abdar et al. [38] demonstrates that incorporating uncertainty quantification into XAI visualizations significantly improves clinical decision-making quality, with physicians making more appropriate referral decisions when shown confidence-adjusted explanations.”
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · cited 1×
“SHAP values guide pathologists in understanding AI-assisted histopathology diagnoses [32].”
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639 · cited 1×
“LIME explanations support ophthalmologists in retinal disease classification tasks [33].”
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 2×
“More alarmingly, recent research indicates that explanations can reduce diagnostic accuracy by fostering overconfidence in incorrect AI predictions [16,20–23].”
CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · cited 1×
“More critically, recent studies reveal that clinicians’ trust in AI recommendations is fundamentally linked to their ability to understand the underlying reasoning processes [9–11].”
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765 · cited 1×
“Explanation stability was evaluated following the perturbation protocol of Ghorbani et al. [15], introducing calibrated Gaussian noise at systematically increasing levels (0%, 5%, 10%, 15%, 20%, 25%, 30% of maximum image intensity) and measuring explanation consistency through Spearman rank…”
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · cited 1×
“While post hoc methods may highlight important features or attention regions, they rarely convey the model’s confidence boundaries or uncertainty estimates crucial for clinical decision-making [36].”
Viral pneumonia screening on chest X-rays using confidence-aware anomaly detection
Zhang, J., Xie, Y., Pang, G., Liao, Z., Verjans, J., Li, W., Sun, Z., He, J., Li, Y., Shen, C., et al. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3040950 · Cited by 389
Obuchowski (2005) · doi:10.2214/ajr.184.2.01840364 · Cited by 321 · cited 1×
“ROC curves were reconstructed from reported sensitivity and specificity values using the method of Obuchowski [27].”
On the Robustness of Interpretability Methods
Alvarez-Melis, D., & Jaakkola, T. S. (2018) · arXiv · doi:10.48550/arxiv.1806.08049 · Cited by 310
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300 · cited 1×
“A model predicting pneumonia with 51% probability should be treated fundamentally differently from one predicting the same condition with 95% probability, yet current XAI visualizations often present both scenarios with equal visual confidence [37].”
Human–machine partnership with artificial intelligence for chest radiograph diagnosis
Patel, B. N., Rosenberg, L., Willcox, G., Baltaxe, D., Lyons, M., Irvin, J., Rajpurkar, P., Amrhein, T., Gupta, R., Halabi, S., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0189-7 · Cited by 254
Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00106-0 · Cited by 229 · cited 1×
“In radiology, Grad-CAM visualizations help radiologists understand convolutional neural network focus areas in chest X-ray interpretation [31].”
Explanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171 · cited 1×
“Adversarial Vulnerability: Recent research demonstrates that explanations can be manipulated to appear reasonable while maintaining incorrect predictions [34].”
Interpretable Image Recognition with Hierarchical Prototypes
Hase, P., Chen, C., Li, O., & Rudin, C. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5265 · Cited by 123
Ukwuoma, C. C., Cai, D., Eziefuna, E. O., Oluwasanmi, A., Abdi, S. F., Muoka, G. W., Thomas, D., & Sarpong, K. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107014 · Cited by 83 · cited 1×
“Clinical Utility: This is the measurable improvement in patient outcomes, diagnostic accuracy, or clinical decision-making quality when using AI assistance compared to standard practice [21,23,24].”
An explainable deep-learning model reveals clinical clues in diabetic retinopathy through SHAP
Herrero-Tudela, M., Romero-Oraá, R., Hornero, R., Gutiérrez‐Tobal, G. C., Gálvez, M. I. L., & García, M. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107328 · Cited by 54 · cited 3×
“Explainability Trap: This is the phenomenon where post hoc explanations create an illusion of understanding without providing genuine insight into model behavior, potentially leading to decreased clinical performance and compromised patient safety [24,25].”
Suh, J., Yoo, S., Park, J., Cho, S. Y., Cho, M. C., Son, H., & Jeong, H. (2020) · BJU International · doi:10.1111/bju.15122 · Cited by 45 · cited 2×
“More alarmingly, recent research indicates that explanations can reduce diagnostic accuracy by fostering overconfidence in incorrect AI predictions [16,20–23].”
Alshahrani, M., Al-Jabbar, M., Senan, E. M., Ahmed, I. A., & Saif, J. A. M. (2023) · Diagnostics · doi:10.3390/diagnostics13172783 · Cited by 35 · cited 1×
“Interpretability–Performance Paradox: This is the observed trade-off between model complexity/accuracy and the ability to understand model decision-making processes, requiring careful balance in clinical applications [24,26].”
The role of saliency maps in enhancing ophthalmologists’ trust in artificial intelligence models
Wong, C. Y. T., Antaki, F., Woodward-Court, P., Ong, A. Y., & Keane, P. A. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100087 · Cited by 30 · cited 1×
“Explainability Trap: This is the phenomenon where post hoc explanations create an illusion of understanding without providing genuine insight into model behavior, potentially leading to decreased clinical performance and compromised patient safety [24,25].”
Artificial intelligence in European medicines regulation
Hines, P. A., Herold, R., Pinheiro, L., Frias, Z., & Arlett, P. (2023) · Nature Reviews Drug Discovery · doi:10.1038/d41573-022-00190-3 · Cited by 24
Singh, Y., Andersen, J. B., Hathaway, Q., Venkatesh, S. K., Gores, G. J., & Erickson, B. (2025) · Oncotarget · doi:10.18632/oncotarget.28709 · Cited by 8 · cited 1×
“A critical yet often overlooked limitation of current XAI approaches is their failure to adequately communicate model uncertainty alongside explanations [35].”
Self-citationPawar, L., Patil, S., Dhotre, S., & Sonawane, S. (2024) · 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap64064.2024.10710903 · Cited by 5
A Comparative Analysis of Non-gradient Methods of Class Activation Mapping
Chakraborty, M., Sardar, S., & Maulik, U. (2023) · Advances in Intelligent Systems and Computing · doi:10.1007/978-981-99-1472-2_16 · Cited by 2 · cited 1×
“Studies demonstrate that post hoc explanation methods often provide misleading information about model behavior, with local approximations achieving poor fidelity to actual decision boundaries [14–19].”
A Survey on Brain Tumor Segmentation Using Deep Learning Techniques
Garapati, S., Swain, S. K., & Vamsi Krishna, M. (2024) · 2024 2nd International Conference on Signal Processing, Communication, Power and Embedded System (SCOPES) · doi:10.1109/scopes64467.2024.10991128 · Cited by 1 · cited 1×
“More alarmingly, recent research indicates that explanations can reduce diagnostic accuracy by fostering overconfidence in incorrect AI predictions [16,20–23].”
Cited by
Beyond single systems: how multi-agent ai is reshaping ethics in radiology
Salehi, S., Singh, Y., Habibi, P., & Erickson, B. J. (2025) · Bioengineering · doi:10.3390/bioengineering12101100 · Cited by 21
Self-citationMartinho, D., Sobreiro, P., Domingues, A., Martinho, F. I. F., & Nogueira, N. F. R. (2026) · Healthcare · doi:10.3390/healthcare14030287 · Cited by 22
Panayides, A. S., Chen, H., Filipović, N., Geroski, T., Hou, J., Lekadir, K., Marias, K., Matsopoulos, G. K., Papanastasiou, G., Sarder, P., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3649496 · Cited by 20
Enhancing the reliability of Alzheimer's disease prediction in MRI images
Islam, J., Furqon, E. N., Farady, I., Alex, J. S. R., Shih, C., Kuo, C.-C., & Lin, C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111111 · Cited by 11
Borra, S., Dey, N., Fong, S., Sherratt, R. S., & Shi, F. (2026) · Cancers · doi:10.3390/cancers18091361 · Cited by 9
Metadata
- Authors
- Singh, Y., Hathaway, Q. A., Keishing, V., Salehi, S., Wei, Y., Horvat, N., Vera-Garcia, D. V., Choudhary, A., Mula Kh, A., Quaia, E., & Andersen, J. B.
- Affiliations
- Mayo Clinic (US)
- University of Pennsylvania (US)
- Sheikh Shakhbout Medical City (SSMC), Abu Dhabi P.O. Box 11001, United Arab Emirates
- University of Padua (IT)
- University of Copenhagen (DK)
- Corresponding author
- Singh, Yashbir
- ORCID
- Venue
- Bioengineering
- Publisher
- MDPI AG
- Published
- 2025-08-15
- Volume
- 12
- Issue
- 8
- Pages
- 879
- ISSN
- 2306-5354 (ISSN-L)
- PMC
- PMC12383817
- OpenAlex
- W4413340328
- Cited by
- 72
- References
- 37
- Citations by year
- 2026: 63 · 2025: 9
- FWCI
- 6.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Bioengineering (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- explainable AI, diagnostic imaging, post-hoc explanation, SHAP, lime, Grad-CAM, noise perturbation, uncertainty quantification, interpretability, accountability framework, radiology, pathology, ophthalmology, clinical decision-making, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.