The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Believing in black boxes: machine learning for healthcare does not need explainability to be evidence-based
Open accessAbstract
Objective To examine the role of explainability in machine learning for healthcare (MLHC), and its necessity and significance with respect to effective and ethical MLHC application. Study design and setting This commentary engages with the growing and dynamic corpus of literature on the use of MLHC and artificial intelligence (AI) in medicine, which provide the context for a focused narrative review of arguments presented in favour of and opposition to explainability in MLHC. Results We find that concerns regarding explainability are not limited to MLHC, but rather extend to numerous well-validated treatment interventions as well as to human clinical judgment itself. We examine the role of evidence-based medicine in evaluating inexplicable treatments and technologies, and highlight the analogy between the concept of explainability in MLHC and the related concept of mechanistic reasoning in evidence-based medicine. Conclusion Ultimately, we conclude that the value of explainability in MLHC is not intrinsic, but is instead instrumental to achieving greater imperatives such as performance and trust. We caution against the uncompromising pursuit of explainability, and advocate instead for the development of robust empirical methods to successfully evaluate increasingly inexplicable algorithmic systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Technically speaking, similar to all other AI models, generative AI models in health care, particularly those based on deep learning, are often seen as “black boxes” [78].”
From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [78]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,804
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“In-deed, prominent case studies have demonstrated precisely how explainable (and not intentionally biased) models can nonetheless deliver biased results that are only apparent upon empirical analysis [ 57 , 58 ].”
BackgroundThe Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Yeh, M. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,836
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Mortality and Morbidity in Patients Receiving Encainide, Flecainide, or Placebo
Echt, D. S., Liebson, P. R., Mitchell, L. B., Peters, R. W., Obias-Manno, D., Barker, A. H., Arensberg, D., Baker, A., Friedman, L., Greene, H. L., et al. (1991) · New England Journal of Medicine · doi:10.1056/nejm199103213241201 · Cited by 3,244
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
The mechanisms of action of metformin
Rena, G., Hardie, D. G., & Pearson, E. R. (2017) · Diabetologia · doi:10.1007/s00125-017-4342-z · Cited by 2,287
Ethics Guidelines for Trustworthy AI
Cannarsa, M. (2021) · The Cambridge Handbook of Lawyering in the Digital Age · doi:10.1017/9781108936040.022 · Cited by 2,095
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · cited 1×
“Sub-sequently, the pursuit of explainability has become an area of significant importance in contemporary MLHC research, emphasized by academics in computer science [ 4 , 8–12 ] medicine [12–14] , and bioethics [7] , as well as major regulatory bodies [15] and healthcare technology companies [16] .”
BackgroundArtificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801
Linguistic Features Identify Alzheimer’s Disease in Narrative Speech
Fraser, K. C., Meltzer, J. A., & Rudzicz, F. (2015) · Journal of Alzheimer's Disease · doi:10.3233/jad-150520 · Cited by 797
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633
Potential liability for physicians using artificial intelligence
Price, W. N., Gerke, S., & Cohen, I. G. (2019) · JAMA · doi:10.1001/jama.2019.15064 · Cited by 579
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553
Interpreting Causality in the Health Sciences
Russo, F., & Williamson, J. (2007) · International Studies in the Philosophy of Science · doi:10.1080/02698590701498084 · Cited by 549
Transparency in Algorithmic and Human Decision-Making: Is There a Double Standard?
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Philosophy & Technology · doi:10.1007/s13347-018-0330-6 · Cited by 420
Identifying Ethical Considerations for Machine Learning Healthcare Applications
Char, D. S., Abràmoff, M. D., & Feudtner, C. (2020) · The American Journal of Bioethics · doi:10.1080/15265161.2020.1819469 · Cited by 415
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411
Assessing risk, automating racism
Benjamin, R. (2019) · Science · doi:10.1126/science.aaz3873 · Cited by 378
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337
Translation of highly promising basic science research into clinical applications
Contopoulos-Ioannidis, D. G., Ntzani, E. E., & Ioannidis, J. P. A. (2003) · The American Journal of Medicine · doi:10.1016/s0002-9343(03)00013-5 · Cited by 323
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019) · arXiv · doi:10.48550/arxiv.1905.05134 · Cited by 318
Desai, R. J., Wang, S. V., Vaduganathan, M., Evers, T., & Schneeweiß, S. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.18962 · Cited by 315
Mechanisms underlying the weight loss effects of RYGB and SG: similar, yet different
Pucci, A. B. F., & Batterham, R. L. (2018) · Journal of Endocrinological Investigation · doi:10.1007/s40618-018-0892-2 · Cited by 230
The European artificial intelligence strategy: implications and challenges for digital health
Cohen, I. G., Evgeniou, T., Gerke, S., & Minssen, T. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30112-6 · Cited by 171
Mechanistic Evidence: Disambiguating the Russo–Williamson Thesis
Illari, P. M. (2011) · International Studies in the Philosophy of Science · doi:10.1080/02698595.2011.574856 · Cited by 171
The Routledge Companion to Philosophy of Medicine
Schües, C. (2016) · Routledge eBooks · doi:10.4324/9781315720739 · Cited by 164
Algorithmic Decision-Making and the Control Problem
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Minds and Machines · doi:10.1007/s11023-019-09513-7 · Cited by 145
Enslaving the Algorithm: From a “Right to an Explanation” to a “Right to Better Decisions”?
Edwards, L., & Veale, M. (2018) · IEEE Security & Privacy · doi:10.1109/msp.2018.2701152 · Cited by 141
Explainable AI for Healthcare: From Black Box to Interpretable Models
Adadi, A., & Berrada, M. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-981-15-0947-6_31 · Cited by 137
Toussaint, K., Yang, X. C., Zielinski, M. A., Reigle, K. L., Sacavage, S. D., Nagar, S., & Raffa, R. B. (2010) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1111/j.1365-2710.2009.01143.x · Cited by 135
Deep learning: definition and perspectives for thoracic imaging
Chassagnon, G., Vakalopolou, M., Nikos Paragios, A., & Revel, M. (2019) · European Radiology · doi:10.1007/s00330-019-06564-3 · Cited by 132
Gillon, R. (2015) · Journal of Medical Ethics · doi:10.1136/medethics-2014-102282 · Cited by 124
Ethical considerations about artificial intelligence for prognostication in intensive care
Beil, M., Proft, I., van Heerden, D., Sviri, S., & van Heerden, P. V. (2019) · Intensive Care Medicine Experimental · doi:10.1186/s40635-019-0286-6 · Cited by 124
Evidence-based mechanistic reasoning
Howick, J., Glasziou, P., & Aronson, J. K. (2010) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2010.100146 · Cited by 121
Nanayakkara, S., Fogarty, S., Tremeer, M., Ross, K. J., Richards, B., Bergmeir, C., Xu, S. Y., Stub, D. A., Smith, K. L., TACEY, M. A., et al. (2018) · PLoS Medicine · doi:10.1371/journal.pmed.1002709 · Cited by 121
The evidence that evidence-based medicine omits
Clarke, B., Gillies, D. A., Illari, P. K., Federica Russo, B., & Williamson, J. (2012) · Preventive Medicine · doi:10.1016/j.ypmed.2012.10.020 · Cited by 115
The new medical model: a renewed challenge for biomedicine
Fuller, J. (2017) · Canadian Medical Association Journal · doi:10.1503/cmaj.160627 · Cited by 113
Meske, C., & Bunde, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50334-5_4 · Cited by 105
Mechanisms responsible for excess weight loss after bariatric surgery
Ionut, V., & Bergman, R. N. (2011) · Journal of Diabetes Science and Technology · doi:10.1177/193229681100500536 · Cited by 105
Explainable Artificial Intelligence for Safe Intraoperative Decision Support
Gordon, L., Grantcharov, T., & Rudzicz, F. (2019) · JAMA Surgery · doi:10.1001/jamasurg.2019.2821 · Cited by 94
Nestor, B., McDermott, M. B. A., Boag, W., Berner, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00690 · Cited by 89
Explaining deep neural networks for knowledge discovery in electrocardiogram analysis
Hicks, S. A., Isaksen, J. L., Thambawita, V. L., Ghouse, J., Ahlberg, G., Linneberg, A. R., Grarup, N., Strümke, I., Ellervik, C., OLESEN, M. S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-90285-5 · Cited by 85
Mechanisms Underlying Type 2 Diabetes Remission After Metabolic Surgery
Pérez‐Pevida, B., Escalada, J., Miras, A. D., & Frühbeck, G. (2019) · Frontiers in Endocrinology · doi:10.3389/fendo.2019.00641 · Cited by 68
Ethics and Learning Health Care: The Essential roles of engagement, transparency, and accountability
KASS, N. E., & Faden, R. (2018) · Learning Health Systems · doi:10.1002/lrh2.10066 · Cited by 63
Explaining Machine Learning Decisions
Zerilli, J. (2022) · Philosophy of Science · doi:10.1017/psa.2021.13 · Cited by 57
What is mechanistic evidence, and why do we need it for evidence-based policy?
Marchionni, C., & Reijula, S. (2018) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2018.08.003 · Cited by 55
Enslaving the Algorithm: From a 'Right to an Explanation' to a 'Right to Better Decisions'?
Edwards, L., & Veale, M. (2018) · Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) · Cited by 53
The use of mechanistic evidence in drug approval
Aronson, J. K., Caze, A. L., Kelly, M. P., Parkkinen, V., & Williamson, J. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12960 · Cited by 50
Nardini, C., Annoni, M., & Schiavone, G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2012.01907.x · Cited by 17
LeBlang, T. R. (1995) · PubMed · Cited by 13
Lemoine, M. (2016) · HAL · doi:10.4324/9781315720739.ch27 · Cited by 7
Clinical Diagnosis—Is There Any Other Type?
Dhaliwal, G. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.3048 · Cited by 7
"Deep" Visual Patterns Are Informative to Practicing Radiologists in Mammograms in Diagnostic Tasks
Sevilla, J., & Hegdé, J. (2017) · Journal of Vision · doi:10.1167/17.10.90 · Cited by 6
Evidence, discovery and justification: the case of evidence‐based medicine
Gaeta, R., & Gentile, N. (2015) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12419 · Cited by 6
Hegdé, J., & Bart, E. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00670 · Cited by 5
Mayer, D. M. (2006) · Epilepsia · doi:10.1111/j.1528-1167.2006.00651.x · Cited by 3
Research works printed but not linked: 2
- Holzinger A, Biemann C, Pattichis CS, Kell DB. What do we need to build explainable AI systems for the medical domain?arXiv:171209923 [cs, stat]. Published online December 28, 2017. Accessed November 24, 2020. http://arxiv.org/abs/1712.09923 · arXiv
- Lipton, 2020, The mythos of model interpretability, arXiv:160603490 [cs, stat]
Cited by
Sorted by citations
Generative AI in medical practice: in-depth exploration of privacy and security challenges
Chen, Y., & Esmaeilzadeh, P. (2024) · Journal of Medical Internet Research · doi:10.2196/53008 · Cited by 353
“Technically speaking, similar to all other AI models, generative AI models in health care, particularly those based on deep learning, are often seen as “black boxes” [78].”
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
Medical deep learning—A systematic meta-review
Egger, J., Gsaxner, C., Pepe, A., Pomykala, K. L., Jonske, F., Kurz, M., Li, J., & Kleesiek, J. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106874 · Cited by 289
Rahman, A., Debnath, T., Kundu, D., Khan, M. S. I., Aishi, A. A., Sazzad, S., Sayduzzaman, M., & Band, S. S. (2024) · AIMS Public Health · doi:10.3934/publichealth.2024004 · Cited by 260
Allgaier, J., Mulansky, L., Draelos, R. L., & Pryss, R. C. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102616 · Cited by 201
Metadata
- Authors
- McCoy, L. G., Brenna, C. T. A., Chen, S. S., Vold, K., & Das, S.
- Affiliations
- University of Toronto (CA)
- Corresponding author
- McCoy, Liam G.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2022-02
- Volume
- 142
- Pages
- 252-257
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W3213880547
- Cited by
- 118
- References
- 64
- Citations by year
- 2026: 17 · 2025: 28 · 2024: 31 · 2023: 23 · 2022: 19
- FWCI
- 1.32
- Type
- journal article
- Language
- en
- Open access
- green
- philpapers.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- Funders
- Canadian Institutes of Health Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainability, machine learning in healthcare, artificial intelligence in healthcare, evidence-based medicine, mechanistic reasoning, algorithm evaluation, trust in algorithms, clinical judgment, ethical use of AI, black-box models
- MeSH terms
- Machine Learning, Artificial Intelligence, Delivery of Health Care, Humans, Technology, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
