Beware explanations from AI in health care
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
The benefits of explainable artificial intelligence are not what they appear
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An example includes a diagnostic algorithm that uses images of patients’ skin to detect skin cancer possibly failing in the summer when more people with more sun exposure have a different color tone, compared to the skin tones used in the training data [45].”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [45]
In the citing paper
“Post-hoc interpretability methods are only approximations and do not accurately depict the correct model behavior and therefore compromise trust in the explanations [5,110].”
From Transparency of deep neural networks for medical image analysis: A review of interpretability methods (2022) · cites this paper as [5]
In the citing paper
“Since post-hoc explanation methods are only approximations, they cannot ensure complete accountability [5].”
From Transparency of deep neural networks for medical image analysis: A review of interpretability methods (2022) · cites this paper as [5]
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215
Ethical and legal challenges of artificial intelligence-driven healthcare
Gerke, S., Minssen, T., & Cohen, G. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00012-5 · Cited by 1,170
Self-citationMarkus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Muehlematter, U. J., Daniore, P., & Vokinger, K. N. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30292-2 · Cited by 665
Intelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547
Research works printed but not linked: 2
- S. Wachter, B. Mittelstadt, C. Russell, Harv. J. Law Technol. 31, 841 (2018).
- R. M. Hamm, S. L. Smith, J. Fam. Pract. 47, 44 (1998).
Cited by
Ethical and regulatory challenges of AI technologies in healthcare: A narrative review
Mennella, C., Maniscalco, U., De Pietro, G., & Esposito, M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26297 · Cited by 1,229
Bias in artificial intelligence algorithms and recommendations for mitigation
Nazer, L. H., Zatarah, R., Waldrip, S., Ke, J. X. C., Moukheiber, M., Khanna, A. K., Hicklen, R. S., Moukheiber, L., Moukheiber, D., Ma, H., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000278 · Cited by 621
Opening the black box: the promise and limitations of explainable machine learning in cardiology
Petch, J., Di, S., & Nelson, W. (2022) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.09.004 · Cited by 552
Salahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508
Shaping the future of healthcare: ethical clinical challenges and pathways to trustworthy AI
Goktas, P., & Grzybowski, A. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14051605 · Cited by 336
“Key components include:”
Metadata
- Authors
- Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G.
- Affiliations
- University of Toronto (CA)
- Pennsylvania State University (US)
- INSEAD (FR)
- Harvard University (US)
- Venue
- Science
- Published
- 2021-07-16
- Volume
- 373
- Issue
- 6552
- Pages
- 284-286
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W3183015861
- Cited by
- 252
- References
- 14
- Citations by year
- 2026: 46 · 2025: 66 · 2024: 57 · 2023: 42 · 2022: 34 · 2021: 6
- FWCI
- 12.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- bepress Legal Repository (submittedVersion)
- eYLS (Yale Law School) (submittedVersion)
- Funders
- Novo Nordisk Fonden (NNF17SA0027784)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainable AI, health care, AI explanations, clinical decision support, AI limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Beware explanations from AI in health care. https://app.cochat.ai/papers/10.1126/science.abg1834
- The paper
- Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021). Beware explanations from AI in health care. Science. https://doi.org/10.1126/science.abg1834
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.