Artificial intelligence in medicine and the disclosure of risks
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
This paper focuses on the use of ‘black box’ AI in medicine and asks whether the physician needs to disclose to patients that even the best AI comes with the risks of cyberattacks, systematic bias, and a particular type of mismatch between AI’s implicit assumptions and an individual patient’s background situation.Pace current clinical practice, I argue that, under certain circumstances, these risks do need to be disclosed. Otherwise, the physician either vitiates a patient’s informed consent or violates a more general obligation to warn him about potentially harmful consequences. To support this view, I argue, first, that the already widely accepted conditions in the evaluation of risks, i.e. the ‘nature’ and ‘likelihood’ of risks, speak in favour of disclosure and, second, that principled objections against the disclosure of these risks do not withstand scrutiny. Moreover, I also explain that these risks are exacerbated by pandemics like the COVID-19 crisis, which further emphasises their significance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In [15], the authors identified three types of security risks related to black-box AI: network attacks, system bias, and mismatch attacks.”
From Survey of Explainable AI Techniques in Healthcare (2023) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“M. Kiener argued that these so-called meta-risks (a risk that the risk-assessment is wrong) are normally not required to be disclosed, but in AI context, the disclosure decision shall be “personalized, i.e., specifically tailored to a patient's personal characteristics” (Kiener, 2021).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Kiener, 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The same applies to the explanations about AI algorithms, their inherent opacity, and risks about changing the accuracy of AI algorithms due to self-learning (risks of mismatch) (Kiener, 2021).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Kiener, 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Principles of Biomedical Ethics
(2012) · Thieves of Virtue · doi:10.7551/mitpress/9079.003.0009 · Cited by 13,453
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 2×
“This can be seen as a form of unintended discrimination and injustice, i.e. another instance of harming and wronging (Obermeyer et al. 2019; Popejoy and Fullerton 2016).”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Nguyen, A., Yosinski, J., & Clune, J. (2014) · arXiv · doi:10.48550/arxiv.1412.1897 · Cited by 3,258
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1×
“As Rajkomar and colleagues state: “routinely collected patient healthcare data are now approaching the genomic scale in volume and complexity” (Rajkomar et al. 2018, 1).”
BackgroundScalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242
Genomics is failing on diversity
Popejoy, A. B., & Fullerton, S. M. (2016) · Nature · doi:10.1038/538161a · Cited by 1,924
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“…depends on genetic test results is more likely to mischaracterize the risk of breast cancer for black patients than white patients” (Parikh et al. 2019, 2377) See also: (Carter et al. 2020; Challen et al. 2019; Char et al. 2018; Obermeyer et al. 2019; Popejoy and Fullerton 2016; Reddy et al. 2020).”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Research works printed but not linked: 2
- Parzeller M, Wenk M, Zedler B, Rothschild M (2007) Aufklärung und Einwilligung bei ärztlichen Eingriffen. Deutsches Ärzteblatt 104:576–586
- Samek WM, Grégoire V, Andrea H, Lars Kai M, Klaus-Robert R (eds) (2019) Explainable AI: interpreting, explaining and visualizing deep learning. Springer, New York
Cited by
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591
Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305
“Kiener [59] looked specifically at security in healthcare and identified three”
BackgroundKiseleva, A., Kotzinos, D., & De Hert, P. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.879603 · Cited by 278
Čartolovni, A., Tomičić, A., & Lazić Mosler, E. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104738 · Cited by 267
Coppola, F., Faggioni, L., Gabelloni, M., De Vietro, F., Mendola, V., Cattabriga, A., Cocozza, M. A., Vara, G., Piccinino, A., Lo Monaco, S., et al. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.710982 · Cited by 94
Metadata
- Authors
- Kiener, M.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Kiener, Maximilian
- ORCID
- Kiener, Maximilian
- Venue
- AI & SOCIETY
- Published
- 2020-10-22
- Volume
- 36
- Issue
- 3
- Pages
- 705-713
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- PMC
- PMC7580986
- OpenAlex
- W3094553643
- Cited by
- 97
- References
- 61
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 25 · 2023: 14 · 2022: 16 · 2021: 8
- FWCI
- 2.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI & Society (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- tub.dok (Hamburg University of Technology) (submittedVersion)
- Funders
- University of Oxford
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics and bioethics in healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- artificial intelligence in healthcare, black-box AI, risk disclosure, informed consent, physician-patient relationship, cyberattacks, algorithmic bias, duty to warn, COVID-19 pandemic, medical ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial intelligence in medicine and the disclosure of risks. https://app.cochat.ai/papers/10.1007/s00146-020-01085-w
- The paper
- Kiener, M. (2021). Artificial intelligence in medicine and the disclosure of risks. AI & SOCIETY. https://doi.org/10.1007/s00146-020-01085-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.