Cited by 446Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIExplainable Artificial Intelligence (XAI)

Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI

Open access
Durán, Juan Manuel, Jongsma, Karin Rolanda·2021·Journal of Medical Ethics

Abstract

The use of black box algorithms in medicine has raised scholarly concerns due to their opaqueness and lack of trustworthiness. Concerns about potential bias, accountability and responsibility, patient autonomy and compromised trust transpire with black box algorithms. These worries connect epistemic concerns with normative issues. In this paper, we outline that black box algorithms are less problematic for epistemic reasons than many scholars seem to believe. By outlining that more transparency in algorithms is not always necessary, and by explaining that computational processes are indeed methodologically opaque to humans, we argue that the reliability of algorithms provides reasons for trusting the outcomes of medical artificial intelligence (AI). To this end, we explain how computational reliabilism , which does not require transparency and supports the reliability of algorithms, justifies the belief that results of medical AI are to be trusted. We also argue that several ethical concerns remain with black box algorithms, even when the results are trustworthy. Having justified knowledge from reliable indicators is, therefore, necessary but not sufficient for normatively justifying physicians to act. This means that deliberation about the results of reliable algorithms is required to find out what is a desirable action. Thus understood, we argue that such challenges should not dismiss the use of black box algorithms altogether but should inform the way in which these algorithms are designed and implemented. When physicians are trained to acquire the necessary skills and expertise, and collaborate with medical informatics and data scientists, black box algorithms can contribute to improving medical care.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/medethics-2020-106820

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Previous work on automation bias suggests a tendency toward overreliance on LLM output, which may lead to poor compliance with manual review.

    “Furthermore, previous work on automation bias suggests a tendency toward overreliance on LLM output, which may lead to poor compliance with manual review.28, 29, 30”

    From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [29]

    In “Professional responsibility and expertise” (page unavailable), the paper warns that medical AI recommendations should not be followed blindly without further assessment. This supports the concern about uncritical reliance, but the paper does not establish a tendency to overrely on LLM output or link it to poor compliance with manual review. (LLM, conf=0.76)

    Experimental — accuracy not yet measured
  • A medical machine-learning tool can recognize cancerous growth from images as verified by physicians, but it cannot identify the indicators that suggested abnormalities, leaving physicians unable to refine their techniques, refine the model, or understand its limits.

    “one medical ML tool successfully recognizes the signs of cancerous growth from images of a patient's body, as verified by further examination by physicians; however, the same tool is unable to identify the indicators that suggested the presence of abnormalities within its samples.”

    From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [22]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured
  • Despite nearly fifty years of development, the same black box problem remains in medical AI applications.

    “Despite nearly a half-century of development, the same problem remains in the medical field's AI applications.”

    From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [22]

    The Background (page unavailable) describes black-box opacity as an ongoing challenge in medical AI, including physicians’ difficulty explaining how an algorithm reaches a diagnosis. It does not establish that this is the same problem persisting after nearly half a century of development; searches for that timeframe and historical comparison found no supporting passage. (LLM, conf=0.85)

    Experimental — accuracy not yet measured
  • The term 'epistemic opacity' does not actually have a univocal and well-defined meaning but refers to a multi-dimensional, plural and (epistemic) context-dependent problem.

    “The reader be warned: the term 'epistemic opacity' 21 does not actually have a univocal and well-defined meaning but refers to a multi-dimensional, plural and (epistemic) context-dependent problem”
  • The complexity and opacity of AI algorithms can make it challenging to comprehend how algorithms arrive at specific decisions, potentially reducing the ability to make informed judgments about the appropriateness and correctness of AI-generated recommendations or diagnoses.

    “Moreover, the complexity and opacity of AI algorithms can make it challenging to comprehend how algorithms arrive at specific decisions, potentially reducing the ability to make informed judgments about the appropriateness and correctness of AI-generated recommendations or diagnoses [30].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. [5]5. Pasquale F. The black box Society. Harvard University Press, 2015.
  2. [37]37. Vulsteke C, del P, Ortega Arevalo M. Artificial intelligence for the oncologist: hype, hubris, or reality? Belgian J Med Oncol 2018;12(7):330-3.
  3. [53]53. de Fine Licht K, de Fine Licht J. Artificial intelligence, transparency, and public decision-making. AI Soc 2020;35(4):917-26.
  4. [58]58. et alAngwin J, Larson J, Mattu SL. Machine bias. pro Publica, 2016. Available: https:// www. propublica. org/ article/ machine-bias-risk-assessments-in-criminal-sentencing [Accessed 16 Feb 2021]. on March 30, 2021 by guest. Protected by copyright. http://jme.bmj.com/ J Med Ethics: first published as 10.1136/medethics-2020-106820 on 18 March 2021. Downloaded from

Cited by

Sorted by citations

Authors
Durán, J. M., & Jongsma, K. R.
Affiliations
  • Delft University of Technology (NL)
  • Utrecht University (NL)
Corresponding author
Durán, Juan Manuel
Venue
Journal of Medical Ethics
Publisher
BMJ
Published
2021-03-18
Pages
medethics-2020-106820
ISSN
  • 0306-6800 (ISSN-L)
  • 1473-4257
OpenAlex
W3136000444
Cited by
446
References
56
Citations by year
2026: 62 · 2025: 98 · 2024: 123 · 2023: 95 · 2022: 46 · 2021: 22
FWCI
20.84
Type
journal article
Language
en
Open access
Locations
  • Journal of Medical Ethics (publishedVersion)
  • PubMed (publishedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • KNAW Research Portal (The Royal Netherlands Academy of Arts and Sciences) (submittedVersion)
  • Research Repository (Delft University of Technology) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
black-box algorithms, artificial intelligence in healthcare, trust in AI, algorithmic transparency, epistemology of AI, algorithmic bias, accountability and responsibility, patient autonomy, epistemic justification, normative justification, physician-AI collaboration, algorithmic opacity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review