The DOI matches its Crossref record; 62 of the 88 references it cites were found in a registry or an index; 21 of the 30 sources its claims cite were checked for support.
The false hope of current approaches to explainable artificial intelligence in health care
Open accessAbstract
The black-box nature of current artificial intelligence (AI) has caused some to question whether AI must be explainable to be used in high-stakes scenarios such as medicine. It has been argued that explainable AI will engender trust with the health-care workforce, provide transparency into the AI decision making process, and potentially mitigate various kinds of bias. In this Viewpoint, we argue that this argument represents a false hope for explainable AI and that current explainability methods are unlikely to achieve these goals for patient-level decision support. We provide an overview of current explainability techniques and highlight how various failure cases can cause problems for decision making for individual patients. In the absence of suitable explainability methods, we advocate for rigorous internal and external validation of AI models as a more direct means of achieving the goals often associated with explainability, and we caution against having explainability be a requirement for clinically deployed models.
Overview
CoChat Summary
## TL;DR
This Viewpoint argues that current explainable AI methods in health care are unlikely to provide reliable, patient-level justifications, build trust, or justify clinical deployment decisions. The authors instead recommend rigorous validation and using explanations mainly for developer/auditor tasks such as troubleshooting, model performance improvement, and algorithmic audit.
## Key findings
- The paper says current explainability techniques can produce broad descriptions of how a system works, but for individual decisions the explanations are “unreliable” or only “superficial,” and may be misleading.
- The authors argue that explanations cannot yet “provide reassurance that an individual decision is correct, increase trust among users, nor justify the acceptance of AI recommendations in clinical practice.”
- They note that explainability can still be useful for “model troubleshooting and systems audit,” including identifying common failure modes or biases.
- For clinical deployment, they urge that AI systems be justified by “reliable and experimentally confirmed performance,” not by explainability-based rationalisations.
## Methods
- This is a Viewpoint, not an empirical study; the paper gives a conceptual and argumentative overview of explainable AI in health care.
- It discusses current explainability approaches, including inherent explainability and post-hoc explainability, with medical imaging heat maps, LIME, SHAP, feature visualisation, and prototypical explanations as illustrative examples.
- The main analysis is a qualitative critique of how these methods can fail at the patient level, drawing on examples from medical imaging, language models, and prior literature on interpretability, automation bias, and model auditing.
## Limitations
- The paper does not report original data, a study sample, or quantitative evaluation of explainability methods; it is a perspective piece.
- It states that “precise technical definitions of explainability lack consensus,” which limits how uniformly the concept can be assessed.
- Evident from the text: the argument is largely theoretical and literature-based, so its claims about real-world clinical impact are not supported by new empirical testing in this paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Valid · 1 Not valid · 7 Not checked · 41 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although most discussions and policies call for normative evaluations, current techniques are only capable of descriptive accounts and it is our own intuition that often “serves as the unacknowledged bridge” between the two.”
What the sources say
- Supportedcites [36] Selbst, 2018, The intuitive appeal of explainable machines, Fordham Law Rev source
In “Interactive Approaches,” the paper says the techniques describe how a model operates but cannot determine whether its way of making decisions is normatively justified. This supports the distinction between descriptive accounts and normative evaluation. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“who show that even untrained networks can produce saliency maps that appear reassuring (appendix).”
What the sources say
- Supportedcites [20] Adebayo, 2018, Sanity checks for saliency maps, Adv Neural Inf Process Syst source
The Introduction (p. 2) describes comparing saliency maps from trained models with those from randomly initialized, untrained networks. The Cascading Randomization results (p. 5) say that visual inspection can fail to distinguish maps from networks with widely differing parameters, supporting the claim that such maps can appear reassuring. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“showed that common visual explanations remain unchanged even when precise modifications are made to the input that substantially alter the model's predictions (a process known as an adversarial attack), even when those attacks lead to incorrect model predictions (figure 2).”
What the sources say
- Supportedcites [23] Gu, 2019, Saliency methods for explaining adversarial attacks, arXiv source
Section 3 (p. 3) reports that, for all tested saliency methods except the authors’ enhanced GuidedBP, maps for clean images and adversarial versions were visually the same despite different predicted classes. The same section says adversarial perturbations can mislead classifications, matching the claim about incorrect predictions. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although they are popular for medical imaging models, they are well known to be problematic in the broader explainability literature.”
What the sources say
- Supportedcites [20] Adebayo, 2018, Sanity checks for saliency maps, Adv Neural Inf Process Syst source
The paper identifies problems with saliency heat maps in the broader explainability literature: Related Work (p. 3) describes their fragility, and the Conclusion (p. 10) warns that visually compelling maps can lack sensitivity to the model and data. It does not say every saliency method fails; Our Contributions (p. 2) distinguishes methods that passed its tests from those that failed. (LLM, conf=0.87)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“There is a great deal of evidence that humans tend to over-trust computer systems”
What the sources say
- Not checked — no full text of the sourcecites [37] Skitka, 1999, Does automation bias decision-making?, Int J Hum Comput Stud source
- Supportedcites [38] Lyell, 2017, Automation bias and verification complexity: a systematic review, J Am Med Inform Assoc source
The Introduction describes overreliance on imperfect automation and explains that users can rely on automated cues more heavily than other information. The Results report evidence of automation bias in a majority of the reviewed studies, supporting the broad claim that there is substantial evidence of this tendency. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [39] Howard, 2020, Are we trusting AI too much? source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
33 more claims · 6 with their cited sources checked
Create a free account to see all 68 claims and every source verdict.
- It may state two separate claims as one. 13
- Probably not a substantial claim. 20
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“These results align well with previous research by Ghassemi et al. and Rajkomar et al., which similarly found that increased AI transparency enhances clinician trust and promotes more effective integration of AI systems into clinical practice [27,28].”
From Enhancing Clinician Trust in AI Diagnostics: A Dynamic Framework for Confidence Calibration and Transparency (2025) · cites this paper as [27]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“That is to say, a medical doctor could rely on experts who thoroughly check and validate the system, and who are thus in a position to certify that the system works reliably, and even that it is periodically checked for sustained reliability across time (London, 2019 ; Topol, 2019 ; Ghassemi et al., 2021 ).”
From The Epistemic Cost of Opacity: How the Use of Artificial Intelligence Undermines the Knowledge of Medical Doctors in High-Stakes Contexts (2025) · cites this paper as Ghassemi et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies demonstrate that post hoc explanation methods often provide misleading information about model behavior, with local approximations achieving poor fidelity to actual decision boundaries [14,15,16,17,18,19].”
From Beyond Post hoc Explanations: A Comprehensive Framework for Accountable AI in Medical Imaging Through Transparency, Interpretability, and Explainability (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several studies have highlighted that individual XAI methods can produce inconsistent or conflicting explanations, which may lead to confusion or misinterpretation by clinicians20.”
From Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, explainability is a complex task that has challenges that need to be carefully addressed during AI development and evaluation to ensure that AI explanations are clinically meaningful and beneficial to end users.68”
From FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2025) · cites this paper as [68]
Experimental — accuracy not yet measured
12 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Research works printed but not linked: 1
- 2016, Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46, OJEU, 59, 294
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“68 Two recommendations for explainability are defined in the FUTURE-AI framework.”
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
Self-citationArtificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
“Techniques such as Shapley additive explanations (SHAP) and Integrated Gradients have also found utility in explaining machine-learning predictions and dataset shift across a variety of applications 326–328 .”
Method
Metadata
- Authors
- Ghassemi, M., Oakden-Rayner, L., & Beam, A. L.
- Affiliations
- Vector Institute (CA)
- Harvard University (US)
- Corresponding author
- Beam, Andrew L
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2021-11
- Volume
- 3
- Issue
- 11
- Pages
- e745-e750
- ISSN
- 2589-7500 (ISSN-L)
- OpenAlex
- W3209901185
- Cited by
- 1,724
- References
- 88
- Citations by year
- 2026: 603 · 2025: 458 · 2024: 257 · 2023: 249 · 2022: 153 · 2021: 4
- FWCI
- 11.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- The Lancet Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Heart, Lung, and Blood Institute (7K01HL141771-02)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- explainable AI, health care AI, black-box models, clinical decision support, trust in AI, algorithmic bias, model validation, external validation, clinical implementation of AI
- MeSH terms
- Artificial Intelligence, Communication, Decision Making, Delivery of Health Care, Diagnostic Imaging, Health Personnel, Humans, Models, Biological, Bias, Comprehension, Dissent and Disputes, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The false hope of current approaches to explainable artificial intelligence in health care. https://app.cochat.ai/papers/10.1016/s2589-7500(21)00208-9
- The paper
- Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021). The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health. https://doi.org/10.1016/s2589-7500(21)00208-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.