The DOI matches its Crossref record; 46 of the 49 references it cites were found in a registry or an index; 8 of the 49 sources its claims cite were checked for support.
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective
Open accessAbstract
Background: Explainability is one of the most heavily debated topics when it comes to the application of artificial intelligence (AI) in healthcare. Even though AI-driven systems have been shown to outperform humans in certain analytical tasks, the lack of explainability continues to spark criticism. Yet, explainability is not a purely technological issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration. This paper provides a comprehensive assessment of the role of explainability in medical AI and makes an ethical evaluation of what explainability means for the adoption of AI-driven tools into clinical practice. Methods: Taking AI-based clinical decision support systems as a case in point, we adopted a multidisciplinary approach to analyze the relevance of explainability for medical AI from the technological, legal, medical, and patient perspectives. Drawing on the findings of this conceptual analysis, we then conducted an ethical assessment using the “Principles of Biomedical Ethics” by Beauchamp and Childress (autonomy, beneficence, nonmaleficence, and justice) as an analytical framework to determine the need for explainability in medical AI. Results: Each of the domains highlights a different set of core considerations and values that are relevant for understanding the role of explainability in clinical practice. From the technological point of view, explainability has to be considered both in terms how it can be achieved and what is beneficial from a development perspective. When looking at the legal perspective we identified informed consent, certification and approval as medical devices, and liability as core touchpoints for explainability. Both the medical and patient perspectives emphasize the importance of considering the interplay between human actors and medical AI. We conclude that omitting explainability in clinical decision support systems poses a threat to core ethical values in medicine and may have detrimental consequences for individual and public health. Conclusions: To ensure that medical AI lives up to its promises, there is a need to sensitize developers, healthcare professionals, and legislators to the challenges and limitations of opaque algorithms in medical AI and to foster multidisciplinary collaboration moving forward.
Overview
CoChat Summary
TL;DR
This paper provides a multidisciplinary conceptual and ethical assessment of explainability in AI-based clinical decision support systems, covering technological, legal, medical, and patient perspectives. It concludes that omitting explainability threatens core ethical values in medicine and may have detrimental consequences for individual and public health.
Key findings
- From the technological perspective, explainability can be inherent to an algorithm or approximated by other methods; inherent explainability is generally more accurate, but there is often a trade-off with performance because inherently explainable models are usually traditional methods such as linear or logistic regression. Explainability also helps developers detect “Clever Hans” errors, such as the Mount Sinai x-ray model that relied on hardware-related metadata rather than clinically relevant image features.
- From the legal perspective, the paper identifies three core fields for explainability: informed consent, certification and approval as medical devices under FDA and MDR frameworks, and liability. It states that FDA and MDR currently require explainability only vaguely, and that GDPR’s implications remain debated.
- From the medical perspective, the paper distinguishes first-level explainability (understanding how the system arrives at conclusions in general) from second-level explainability (identifying which features were important for an individual prediction). Explainability can help clinicians resolve disagreement with AI, identify false positives and false negatives, and support trust and quality assurance.
- From the patient and ethical perspectives, “black-box medicine” is argued to conflict with patient-centered care and shared decision-making. Using Beauchamp and Childress’s four principles—autonomy, beneficence, nonmaleficence, and justice—the paper concludes that explainability is an ethical prerequisite for clinically applied AI, and that opaque systems may foster paternalism, defensive medicine, and unjust bias.
Methods
- The paper performs a conceptual analysis of pertinent literature on explainable AI from the technological, legal, medical, and patient perspectives, taking AI-based clinical decision support systems as the case in point.
- It then conducts an ethical assessment using the “Principles of Biomedical Ethics” by Beauchamp and Childress, structured around autonomy, beneficence, nonmaleficence, and justice.
- No empirical data collection, patient or clinician sample, or quantitative analysis is reported.
Limitations
- The paper states that the terminology of explainability is not well defined, that there exist only exploratory initial attempts to rank explainability methods quantitatively, that legal requirements remain vague or ambiguous, that liability questions lack clear answers, and that ethical consensus on disclosure of opaque AI has not yet emerged.
- The paper also states that further work is needed to sensitize developers, healthcare professionals, and legislators and to foster multidisciplinary collaboration.
- Evident from the text: this is a conceptual and normative analysis without empirical validation or systematic review methodology, and its legal discussion is explicitly framed from a Western legal point of view.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 5 Not valid · 34 Not checked · 65 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“While the work of Rudin et al. is important to raise attention to the shortcomings of approximating explainability methods, it is likely that some approximating methods, in contrast to the notion of [2], have value given the complex nature of explaining machine learning models.”
§ Explainability for artificial intelligence in healthcare: a multidisciplinary perspective > Results > The technological perspective
What the sources say
- Supportedcites [2] Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1:206–15. source
The citation is attached to Rudin’s critique of approximating explainability methods, not to the citing authors’ contrasting view that some methods have value. In “Key Issues with Explainable ML,” Rudin says approximate explanations cannot be perfectly faithful to a black-box model and may mislead users. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Clever Hans phenomena are also found in medicine.”
§ Explainability for artificial intelligence in healthcare: a multidisciplinary perspective > Results > The technological perspective
What the sources say
- Supportedcites [11] Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLOS Med. 2018;15:e1002683. source
The Discussion (page unavailable in our copy) reports that the more predictive a hospital system was of pneumonia, the more the model exploited that association, with poorer performance at external sites. This supports the broad claim that medical AI can learn misleading cues rather than the condition it is meant to detect, even though the paper does not use the term “Clever Hans.” (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Importantly, however, inherent explainability will, in general, be more accurate than methods that only approximate explainability [2].”
§ Explainability for artificial intelligence in healthcare: a multidisciplinary perspective > Results > The technological perspective
What the sources say
- Supportedcites [2] Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1:206–15. source
In “Explainable ML methods provide explanations that are not faithful to what the original model computes,” Rudin says approximations of black-box predictions cannot be perfectly faithful, whereas a fully faithful explanation would itself be the interpretable model. This supports the claim about the accuracy of explanations, rather than a general claim about predictive accuracy. (LLM, conf=0.89)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Since clinicians are no longer able to fully comprehend the inner workings and calculations of the decision aid they are not able to explain to the patient how certain outcomes or recommendations were derived [33].”
§ Explainability for artificial intelligence in healthcare: a multidisciplinary perspective > Results > The patient perspective
What the sources say
- Supportedcites [33] Bjerring JC, Busch J. Artificial intelligence and patient-centered decision-making. Philos Technol. 2020. https://doi.org/10.1007/s13347-019-00391-6. source
Section 5 (pp. 24–25) says black-box systems can leave practitioners without a sensible explanation of the information used to produce recommendations. It also says that, for recommendations based on black-box AI, neither practitioner nor patient can explain the grounds for the system’s performance, and explanations of why a particular recommendation was made may be unavailable. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“And, indeed, AI-powered systems have been shown to produce overall lower error rates than traditional methods [19–21].”
§ Explainability for artificial intelligence in healthcare: a multidisciplinary perspective > Results > The medical perspective
What the sources say
- Partly supportedcites [19] Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N. Can machine-learning improve cardiovascular risk prediction using routine clinical data? PLoS ONE. 2017;12:e0174944. sourceExperimental — accuracy not yet measured
The Prediction accuracy section (page unavailable) reports that every machine-learning model tested improved discrimination over the ACC/AHA cardiovascular-risk model. The Classification analysis section also reports more correctly predicted cases and non-cases. These findings support fewer prediction errors in this setting, but not the broader claim about AI-powered systems and traditional methods overall. (LLM, conf=0.84)
- Supportedcites [20] Kakadiaris IA, Vrigkas M, Yen AA, Kuznetsova T, Budoff M, Naghavi M. Machine learning outperforms ACC/AHA CVD risk calculator in MESA. J Am Heart Assoc. 2018;7:e009476. source
The Discussion reports that the machine-learning CVD risk calculator achieved significantly higher accuracy than the traditional ACC/AHA calculator using identical inputs. This supports the claim as an example of an AI-powered system producing fewer prediction errors; the comparison is specific to CVD risk prediction. (LLM, conf=0.90)
- Not checked yetcites [21] Liu T, Fan W, Wu C. A hybrid machine learning approach to cerebral stroke prediction based on imbalanced medical dataset. Artif Intell Med. 2019;101:101723–101723. source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
39 more claims · 1 with their cited sources checked
Create a free account to see all 111 claims and to verify the 41 references still open.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 47
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At present, there is no ethical and legal consensus regarding whether disclosing the application of a medical AI is required for informed consent in the United States, the European Union, or South Korea.”
From Patient perspectives on informed consent for medical AI: A web-based experiment (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The dependence on AI technology may also result in the lack of development of essential decision-making skills and clinical judgment [29].”
From Biomedical Ethical Aspects Towards the Implementation of Artificial Intelligence in Medical Education (2023) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Currently, there is no universally agreed-upon ethical or legal requirement to disclose the use of medical AI for informed consent in the United States, the European Union, or South Korea8,9.”
From Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classified as Software of Unknown Provenance (SOUP), these systems often do not disclose critical details about their data sources, development methodologies, or validation processes, making it challenging to assess and manage biases that may emerge at various stages of the model lifecycle8,20,21.”
From Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, there is no established ethical or legal requirement in the United States for obtaining informed consent for the use of medical AI in clinical practice.”
From Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability (2025) · cites this paper as [8]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainability for artificial intelligence in healthcare: a multidisciplinary perspective > Results > Ethical implications
“the “Principles of Biomedical Ethics” by Beauchamp and Childress [36, 37]”
§ Explainability for artificial intelligence in healthcare: a multidisciplinary perspective > References
“36. Beauchamp TL. Principles of biomedical ethics. Paperback May-2008. New York: Oxford University Press; 2008.”
Citations
Cites
Principles of Biomedical Ethics
(2012) · Thieves of Virtue · doi:10.7551/mitpress/9079.003.0009 · Cited by 13,453
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Decision aids for people facing health treatment or screening decisions
Stacey, D., Légaré, F., Lewis, K., Barry, M. J., Bennett, C. L., Eden, K. B., Holmes-Rovner, M., Llewellyn-Thomas, H., Lyddiatt, A., Thomson, R., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001431.pub5 · Cited by 6,685
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312
Shared decision making — The pinnacle of patient-centered care
Barry, M. J., & Edgman-Levitan, S. (2012) · New England Journal of Medicine · doi:10.1056/nejmp1109283 · Cited by 3,433
Raz, J. (1988) · doi:10.1093/0198248075.001.0001 · Cited by 2,895
A history and theory of informed consent
Kapp, M. B. (1986) · Journal of Legal Medicine · doi:10.1080/01947648609513478 · Cited by 2,331
Principles alone cannot guarantee ethical AI
Mittelstadt, B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0114-4 · Cited by 1,886
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719
Research works printed but not linked: 1
- Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academies Press (US); 2001. http://www.ncbi.nlm.nih.gov/books/NBK222274/. Accessed 21 May 2020.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
Deep learning in cancer diagnosis, prognosis and treatment selection
Tran, K. A., Kondrashova, O., Bradley, A., Williams, E. D., Pearson, J. V., & Waddell, N. (2021) · Genome Medicine · doi:10.1186/s13073-021-00968-x · Cited by 932
“assay noise or errors in recording) or systematic errors caused by bias within DL models themselves, which can result from the training data not being representative of the population where DL is later applied [27].”
BackgroundInfluentialAlbahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“67 It enables end users to interpret the AI model and outputs, understand the capacities and limitations of the AI tool, and intervene when necessary, such as to decide to use it or not.”
Metadata
- Authors
- the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I.
- Affiliations
- ETH Zurich (CH)
- Charité – Universitätsmedizin Berlin (DE)
- Birmingham City University (GB)
- Corresponding author
- Amann, Julia
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2020-11-30
- Volume
- 20
- Issue
- 1
- Pages
- 310-310
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC7706019
- OpenAlex
- W3109650690
- Cited by
- 2,203
- References
- 49
- Citations by year
- 2026: 699 · 2025: 555 · 2024: 407 · 2023: 306 · 2022: 168 · 2021: 67
- FWCI
- 53.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
- ETH Zürich Research Collection (cc-by)
- Funders
- Horizon 2020 Research and Innovation Programme (777107)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainability, artificial intelligence in healthcare, clinical decision support systems, biomedical ethics, informed consent, medical device regulation, liability, human-AI interaction, algorithmic opacity, multidisciplinary collaboration
- MeSH terms
- Artificial Intelligence, Delivery of Health Care, Health Facilities, Humans, Informed Consent, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. https://app.cochat.ai/papers/10.1186/s12911-020-01332-6
- The paper
- the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020). Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-020-01332-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.