Artificial intelligence in medicine and the disclosure of risks
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
This paper focuses on the use of ‘black box’ AI in medicine and asks whether the physician needs to disclose to patients that even the best AI comes with the risks of cyberattacks, systematic bias, and a particular type of mismatch between AI’s implicit assumptions and an individual patient’s background situation.Pace current clinical practice, I argue that, under certain circumstances, these risks do need to be disclosed. Otherwise, the physician either vitiates a patient’s informed consent or violates a more general obligation to warn him about potentially harmful consequences. To support this view, I argue, first, that the already widely accepted conditions in the evaluation of risks, i.e. the ‘nature’ and ‘likelihood’ of risks, speak in favour of disclosure and, second, that principled objections against the disclosure of these risks do not withstand scrutiny. Moreover, I also explain that these risks are exacerbated by pandemics like the COVID-19 crisis, which further emphasises their significance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In [15], the authors identified three types of security risks related to black-box AI: network attacks, system bias, and mismatch attacks.”
From Survey of Explainable AI Techniques in Healthcare (2023) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“M. Kiener argued that these so-called meta-risks (a risk that the risk-assessment is wrong) are normally not required to be disclosed, but in AI context, the disclosure decision shall be “personalized, i.e., specifically tailored to a patient's personal characteristics” (Kiener, 2021).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Kiener, 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The same applies to the explanations about AI algorithms, their inherent opacity, and risks about changing the accuracy of AI algorithms due to self-learning (risks of mismatch) (Kiener, 2021).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Kiener, 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Principles of Biomedical Ethics
(2012) · Thieves of Virtue · doi:10.7551/mitpress/9079.003.0009 · Cited by 13,453
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 2×
“This can be seen as a form of unintended discrimination and injustice, i.e. another instance of harming and wronging (Obermeyer et al. 2019; Popejoy and Fullerton 2016).”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Nguyen, A., Yosinski, J., & Clune, J. (2014) · arXiv · doi:10.48550/arxiv.1412.1897 · Cited by 3,258
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1×
“As Rajkomar and colleagues state: “routinely collected patient healthcare data are now approaching the genomic scale in volume and complexity” (Rajkomar et al. 2018, 1).”
BackgroundScalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242
Genomics is failing on diversity
Popejoy, A. B., & Fullerton, S. M. (2016) · Nature · doi:10.1038/538161a · Cited by 1,924 · cited 2×
“This can be seen as a form of unintended discrimination and injustice, i.e. another instance of harming and wronging (Obermeyer et al. 2019; Popejoy and Fullerton 2016).”
BackgroundImplementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“…depends on genetic test results is more likely to mischaracterize the risk of breast cancer for black patients than white patients” (Parikh et al. 2019, 2377) See also: (Carter et al. 2020; Challen et al. 2019; Char et al. 2018; Obermeyer et al. 2019; Popejoy and Fullerton 2016; Reddy et al. 2020).”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
The second-person standpoint: morality, respect, and accountability
(2007) · Choice Reviews Online · doi:10.5860/choice.44-4978 · Cited by 1,666
Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235 · cited 1×
“…electronic health records (EHR), could considerably improve healthcare by personalising treatment to individual patients as well as significantly reduce costs by allocating medical resources more accurately (Bates et al. 2014; Jameson and Longo 2015; Krumholz 2014; Parikh et al. 2016, 2017).”
BackgroundAutomation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“First, psychological research has shown that people have an ‘auto-mation bias’, i.e. people display an unjustified reliance on machines over human decisions and are likely to think that AI is free from the frailties of human choice (Goddard et al. 2012).”
BackgroundPrecision Medicine — Personalized, Problematic, and Promising
Jameson, J. L., & Longo, D. L. (2015) · New England Journal of Medicine · doi:10.1056/nejmsb1503104 · Cited by 1,106
Adversarial attacks on medical machine learning
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019) · Science · doi:10.1126/science.aaw4399 · Cited by 1,088 · cited 3×
“Attacks on AI systems can undermine diagnostic accuracy, administer lethal drug doses, or sabotage critical moves in an operation (Finlayson et al. 2019; Hutson 2018; Kim et al. 2019; Sun et al. 2018).”
BackgroundA clinically applicable approach to continuous prediction of future acute kidney injury
Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., et al. (2019) · Nature · doi:10.1038/s41586-019-1390-1 · Cited by 1,086 · cited 1×
“…individual patients will die in the near future (Aczon et al. 2017), how likely it is that they will develop a severe form of specific diseases like pneumonia (Caruana et al. 2015), or how likely it is that they will suffer certain complications, such as acute kidney failure (Tomašev et al. 2019).”
BackgroundA governance model for the application of AI in health care
Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz192 · Cited by 979
Artificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956 · cited 1×
“…depends on genetic test results is more likely to mischaracterize the risk of breast cancer for black patients than white patients” (Parikh et al. 2019, 2377) See also: (Carter et al. 2020; Challen et al. 2019; Char et al. 2018; Obermeyer et al. 2019; Popejoy and Fullerton 2016; Reddy et al. 2020).”
BackgroundBlack-box Adversarial Attacks with Limited Queries and Information
Ilyas, A., Engstrom, L., Athalye, A., & Lin, J. (2018) · arXiv · doi:10.48550/arxiv.1804.08598 · Cited by 806
Artificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801
Addressing bias in artificial intelligence in health care
Parikh, R. B., Teeple, S., & Navathe, A. S. (2019) · JAMA · doi:10.1001/jama.2019.18058 · Cited by 764
Informed Consent: Legal Theory and Clinical Practice
Appelbaum, P. S., Lidz, C. W., & Meisel, A. (1987) · eScholarship@UMassChan · Cited by 733
Lallie, H. S., Shepherd, L. A., Nurse, J. R. C., Erola, A., Epiphaniou, G., Maple, C., & Bellekens, X. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102248 · Cited by 671
Krumholz, H. M. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0053 · Cited by 589 · cited 1×
“…electronic health records (EHR), could considerably improve healthcare by personalising treatment to individual patients as well as significantly reduce costs by allocating medical resources more accurately (Bates et al. 2014; Jameson and Longo 2015; Krumholz 2014; Parikh et al. 2016, 2017).”
Background(2016) · PubMed · Cited by 576
Coeckelbergh, M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00146-8 · Cited by 475 · cited 1×
“Yet, we often cannot understand why, on a certain occasion, the AI system made a particular decision, arrived at a particular diagnosis, or performed a particular move in an operation (Bathaee 2018; Carabantes 2019; Coeckelbergh 2020).”
BackgroundCarter, S. M., Rogers, W., Win, K. T., Frazer, H., Richards, B., & Houssami, N. (2020) · The Breast · doi:10.1016/j.breast.2019.10.001 · Cited by 312
(2018) · Harvard journal of law & technology · Cited by 295
Predicting COVID-19 Pneumonia Severity on Chest X-ray With Deep Learning
Cohen, J. P., Dao, L., Roth, K., Morrison, P., Bengio, Y., Abbasi, A. F., Shen, B., Mahsa, H. K., Ghassemi, M., Li, H., et al. (2020) · Cureus · doi:10.7759/cureus.9448 · Cited by 259
Clinical AI: opacity, accountability, responsibility and liability
Smith, H. (2021) · AI & SOCIETY · doi:10.1007/s00146-020-01019-6 · Cited by 245
Should Health Care Demand Interpretable Artificial Intelligence or Accept “Black Box” Medicine?
Wang, F., Kaushal, R., & Khullar, D. (2020) · Annals of Internal Medicine · doi:10.7326/m19-2548 · Cited by 244
Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., & Sun, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05942 · Cited by 244
Exploring the Causes of Comparative Optimism
Shepperd, J. A., Carroll, P., Grace, J., & Terry, M. (2002) · Psychologica Belgica · doi:10.5334/pb.986 · Cited by 213
There Is No Techno-Responsibility Gap
Tigard, D. W. (2021) · Philosophy & Technology · doi:10.1007/s13347-020-00414-7 · Cited by 192
Adversarial Reprogramming of Neural NetworksPreprint
Elsayed, G. F., Goodfellow, I., & Sohl‐Dickstein, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.11146 · Cited by 181
Autonomy, Informed Consent and Medical Law: A Relational Challenge
Maclean, A. R. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511576119 · Cited by 172
Black-box artificial intelligence: an epistemological and critical analysis
Carabantes, M. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00888-w · Cited by 170
Herring, J. (2022) · Oxford University Press eBooks · doi:10.1093/he/9780192856562.001.0001 · Cited by 169
Integrating Predictive Analytics Into High-Value Care: The Dawn of Precision Delivery
Parikh, R. B., Kakad, M., & Bates, D. W. (2016) · JAMA · doi:10.1001/jama.2015.19417 · Cited by 168
Artificial intelligence in cancer therapy
Ho, D. (2020) · Science · doi:10.1126/science.aaz3023 · Cited by 164 · cited 1×
“On average, AI is the better driver and in some domains of medical diagnosis (Bathaee 2018), drug development (Arshadi et al. 2020), and even the execution of treatment and surgery (Ficuciello et al. 2019; Ho 2020), AI already is—or soon promises to be—better than trained medical professionals.”
BackgroundInformed Consent: Legal Theory and Clinical Practice.
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-9-827_3 · Cited by 140
Autonomy in surgical robots and its meaningful human control
Ficuciello, F., Tamburrini, G., Arezzo, A., Villani, L., & Siciliano, B. (2019) · Paladyn, Journal of Behavioral Robotics · doi:10.1515/pjbr-2019-0002 · Cited by 92 · cited 1×
“On average, AI is the better driver and in some domains of medical diagnosis (Bathaee 2018), drug development (Arshadi et al. 2020), and even the execution of treatment and surgery (Ficuciello et al. 2019; Ho 2020), AI already is—or soon promises to be—better than trained medical professionals.”
BackgroundArgaw, S. T., Bempong, N.-E., Eshaya-Chauvin, B., & Flahault, A. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0724-5 · Cited by 91
DeepMalaria: Artificial Intelligence Driven Discovery of Potent Antiplasmodials
Keshavarzi Arshadi, A., Salem, M., Collins, J., Yuan, J. S., & Chakrabarti, D. (2020) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01526 · Cited by 77
McLean, S. A. M. (2009) · Routledge eBooks · doi:10.4324/9780203873199 · Cited by 68
Dynamic Mortality Risk Predictions in Pediatric Critical Care Using Recurrent Neural Networks
Aczon, M., Ledbetter, D., Ho, V. L., Gunny, A. M., Flynn, A., Williams, J., & Wetzel, R. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.06675 · Cited by 68
Price, W. (2017) · Michigan Law Review · doi:10.36644/mlr.116.3.regulating · Cited by 64
Trust Region Based Adversarial Attack on Neural Networks
Yao, Z., Gholami, A., Xu, P., Keutzer, K., & Mahoney, M. W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01161 · Cited by 64
Identify Susceptible Locations in Medical Records via Adversarial Attacks on Deep Predictive Models
Sun, M., Tang, F., Yi, J., Wang, F., & Zhou, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219909 · Cited by 61
Hackers easily fool artificial intelligences
Hutson, M. (2018) · Science · doi:10.1126/science.361.6399.215 · Cited by 54
Cyberattack on Britain’s National Health Service — A Wake-up Call for Modern Medicine
Clarke, R., & Youngstein, T. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1706754 · Cited by 53
Beyond Genes and Molecules — A Precision Delivery Initiative for Precision Medicine
Parikh, R. B., Schwartz, J. S., & Navathe, A. S. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1613224 · Cited by 42
Cyberattacks on Medical Devices and Hospital Networks: Legal Gaps and Regulatory Solutions
Wellington, K. B. (2014) · Santa Clara computer and high-technology law journal · Cited by 35
Major global cyber-attack hits NHS and delays treatment
O’Dowd, A. (2017) · BMJ · doi:10.1136/bmj.j2357 · Cited by 34
Kim, H., Jung, D. C., & Choi, B. W. (2019) · Journal of the Korean Society of Radiology · doi:10.3348/jksr.2019.80.2.259 · Cited by 34
Consent in the time of COVID-19
Turnham, H. L., Dunn, M., Hill, E., Thornburn, G. T., & Wilkinson, D. J. C. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2020-106402 · Cited by 31
Foster, C. (2010) · Mortality · doi:10.1080/13576275.2010.482782 · Cited by 25
Cyberinsights : COVID-19 and What It Means for Cybersecurity
Wirth, A. (2020) · Biomedical Instrumentation & Technology · doi:10.2345/0899-8205-54.3.216 · Cited by 24
Thrombophilia testing in the inpatient setting: impact of an educational intervention
Kwang, H., Mou, E., Richman, I. B., Kumar, A. D., Berube, C., Kaimal, R., Ahuja, N. K., Harman, S. M., Johnson, T. P., Shah, N., et al. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0889-6 · Cited by 10
Risk disclosure and informed consent: Practice caveats in the post Montgomery era
Lee, D. W. H., & Lai, P. B. S. (2020) · Surgical Practice · doi:10.1111/1744-1633.12423 · Cited by 5
The Obligation to Provide Information where Valid Consent is Not Needed
Walker, T. W. M. (2017) · Kennedy Institute of Ethics journal · doi:10.1353/ken.2017.0040 · Cited by 2
Research works printed but not linked: 2
- Parzeller M Wenk M Zedler B Rothschild M Aufklärung und Einwilligung bei ärztlichen Eingriffen Deutsches Ärzteblatt 2007 104 576 586
- Samek WM Grégoire V Andrea H Lars Kai M Klaus-Robert R Explainable AI: interpreting, explaining and visualizing deep learning 2019 New York Springer
Cited by
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591
“In [15], the authors identified three types of security risks related to black-box AI: network attacks, system bias, and mismatch attacks.”
BackgroundExplainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305
“Kiener [59] looked specifically at security in healthcare and identified three”
BackgroundKiseleva, A., Kotzinos, D., & De Hert, P. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.879603 · Cited by 278
Čartolovni, A., Tomičić, A., & Lazić Mosler, E. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104738 · Cited by 267
Coppola, F., Faggioni, L., Gabelloni, M., De Vietro, F., Mendola, V., Cattabriga, A., Cocozza, M. A., Vara, G., Piccinino, A., Lo Monaco, S., et al. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.710982 · Cited by 94
Metadata
- Authors
- Kiener, M.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Kiener, Maximilian
- ORCID
- Kiener, Maximilian
- Venue
- AI & SOCIETY
- Published
- 2020-10-22
- Volume
- 36
- Issue
- 3
- Pages
- 705-713
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- PMC
- PMC7580986
- OpenAlex
- W3094553643
- Cited by
- 97
- References
- 61
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 25 · 2023: 14 · 2022: 16 · 2021: 8
- FWCI
- 2.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI & Society (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- tub.dok (Hamburg University of Technology) (submittedVersion)
- Funders
- University of Oxford
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics and bioethics in healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- artificial intelligence in healthcare, black-box AI, risk disclosure, informed consent, physician-patient relationship, cyberattacks, algorithmic bias, duty to warn, COVID-19 pandemic, medical ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.