The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Believing in black boxes: machine learning for healthcare does not need explainability to be evidence-based
Open accessAbstract
Objective To examine the role of explainability in machine learning for healthcare (MLHC), and its necessity and significance with respect to effective and ethical MLHC application. Study design and setting This commentary engages with the growing and dynamic corpus of literature on the use of MLHC and artificial intelligence (AI) in medicine, which provide the context for a focused narrative review of arguments presented in favour of and opposition to explainability in MLHC. Results We find that concerns regarding explainability are not limited to MLHC, but rather extend to numerous well-validated treatment interventions as well as to human clinical judgment itself. We examine the role of evidence-based medicine in evaluating inexplicable treatments and technologies, and highlight the analogy between the concept of explainability in MLHC and the related concept of mechanistic reasoning in evidence-based medicine. Conclusion Ultimately, we conclude that the value of explainability in MLHC is not intrinsic, but is instead instrumental to achieving greater imperatives such as performance and trust. We caution against the uncompromising pursuit of explainability, and advocate instead for the development of robust empirical methods to successfully evaluate increasingly inexplicable algorithmic systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Technically speaking, similar to all other AI models, generative AI models in health care, particularly those based on deep learning, are often seen as “black boxes” [78].”
From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [78]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,804
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“In-deed, prominent case studies have demonstrated precisely how explainable (and not intentionally biased) models can nonetheless deliver biased results that are only apparent upon empirical analysis [ 57 , 58 ].”
BackgroundThe Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Yeh, M. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,836
Research works printed but not linked: 2
- Holzinger A, Biemann C, Pattichis CS, Kell DB. What do we need to build explainable AI systems for the medical domain?arXiv:171209923 [cs, stat]. Published online December 28, 2017. Accessed November 24, 2020. http://arxiv.org/abs/1712.09923 · arXiv
- Lipton, 2020, The mythos of model interpretability, arXiv:160603490 [cs, stat]
Cited by
Sorted by citations
Generative AI in medical practice: in-depth exploration of privacy and security challenges
Chen, Y., & Esmaeilzadeh, P. (2024) · Journal of Medical Internet Research · doi:10.2196/53008 · Cited by 353
“Technically speaking, similar to all other AI models, generative AI models in health care, particularly those based on deep learning, are often seen as “black boxes” [78].”
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
Medical deep learning—A systematic meta-review
Egger, J., Gsaxner, C., Pepe, A., Pomykala, K. L., Jonske, F., Kurz, M., Li, J., & Kleesiek, J. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106874 · Cited by 289
Rahman, A., Debnath, T., Kundu, D., Khan, M. S. I., Aishi, A. A., Sazzad, S., Sayduzzaman, M., & Band, S. S. (2024) · AIMS Public Health · doi:10.3934/publichealth.2024004 · Cited by 260
Allgaier, J., Mulansky, L., Draelos, R. L., & Pryss, R. C. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102616 · Cited by 201
Metadata
- Authors
- McCoy, L. G., Brenna, C. T. A., Chen, S. S., Vold, K., & Das, S.
- Affiliations
- University of Toronto (CA)
- Corresponding author
- McCoy, Liam G.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2022-02
- Volume
- 142
- Pages
- 252-257
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W3213880547
- Cited by
- 118
- References
- 64
- Citations by year
- 2026: 17 · 2025: 28 · 2024: 31 · 2023: 23 · 2022: 19
- FWCI
- 1.32
- Type
- journal article
- Language
- en
- Open access
- green
- philpapers.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- Funders
- Canadian Institutes of Health Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainability, machine learning in healthcare, artificial intelligence in healthcare, evidence-based medicine, mechanistic reasoning, algorithm evaluation, trust in algorithms, clinical judgment, ethical use of AI, black-box models
- MeSH terms
- Machine Learning, Artificial Intelligence, Delivery of Health Care, Humans, Technology, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
