The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
“I’m afraid I can’t let you do that, Doctor”: meaningful disagreements with AI in medical contexts
Open accessAbstract
This paper explores the role and resolution of disagreements between physicians and their diagnostic AI-based decision support systems (DSS). With an ever-growing number of applications for these independently operating diagnostic tools, it becomes less and less clear what a physician ought to do in case their diagnosis is in faultless conflict with the results of the DSS. The consequences of such uncertainty can ultimately lead to effects detrimental to the intended purpose of such machines, e.g. by shifting the burden of proof towards a physician. Thus, we require normative clarity for integrating these machines without affecting established, trusted, and relied upon workflows. In reconstructing different causes of conflicts between physicians and their AI-based tools—inspired by the approach of “meaningful human control” over autonomous systems and the challenges to resolve them—we will delineate normative conditions for “meaningful disagreements”. These incorporate the potential of DSS to take on more tasks and outline how the moral responsibility of a physician can be preserved in an increasingly automated clinical work environment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, AI models introduce a new type of uncertainty, which can undermine clinicians'trust, especially when models function as opaque “black boxes” [45–47].”
From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [46]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 1×
“Different analyses have been put forward categorizing the types of DSS in medical contexts (Topol 2019, Braun et al 2020, for a review, see Sutton et al 2020).”
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · cited 2×
“Specifying this distinction for diagnostic contexts, examples for C-AI may be a chatbot guiding the anamnesis of patients, or an algorithm helping to take and sort the relevant medical images of a patient, which we call knowledge-based evidence (Sutton et al 2020).”
BackgroundArtificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · cited 2×
“To systematize different types of DSS in the diagnostic process, we follow Yu et al. (2019) and Braun et al (2020) in distinguishing three types of DSS according to the level of “meaningful human control”.”
BackgroundMethodEpistemology of Disagreement: The Good News
Christensen, D. (2007) · The Philosophical Review · doi:10.1215/00318108-2006-035 · Cited by 970 · cited 1×
“Following the preconditions of human–human disagreement (as studied in philosophy, cf. e.g. Christensen 2007 and see 3.2), meaningful disagreement requires the acknowledgment of the reasonableness on the side of both disagreeing parties and one’s own epistemic limitations.”
BackgroundInfluentialMittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“The relevance of the cognitive work, the ability of some DSS to access non-knowledge-based evidence, and the altogether insufficient explainability (Mittelstadt et al. 2019) gives credence to the perceptions of those conflicts as disagreements.”
BackgroundOn the ethics of algorithmic decision-making in healthcare
Grote, T., & Berens, P. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-105586 · Cited by 551 · cited 2×
“A first problematic consequence of integrating DSS in diagnostic processes can be an increased reluctance of doctors to make their own, independent diagnoses (Grote and Berens 2020), as the correctness of these diagnoses can be challenged by AI.”
BackgroundSecond opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · cited 1×
“Some have even suggested (Kompa et al. 2021; Mozannar and Sontag 2020) that recommendations of an AI may be tailor-made to fit the success score of individual physicians.”
BackgroundHuman nature and conduct: An introduction to social psychology.
Dewey, J. (1922) · doi:10.1037/14663-000 · Cited by 464 · cited 1×
“The idea to positively reframe conflicts is indebted to the tradition of American pragmatism, notably the work of John Dewey, whom we quote here at length (Dewey 1922, 301): What is to be done with […] facts of disharmony and conflict?”
BackgroundComputer knows best? The need for value-flexibility in medical AI
McDougall, R. J. (2019) · Journal of Medical Ethics · doi:10.1136/medethics-2018-105118 · Cited by 262
Human–machine partnership with artificial intelligence for chest radiograph diagnosis
Patel, B. N., Rosenberg, L., Willcox, G., Baltaxe, D., Lyons, M., Irvin, J., Rajpurkar, P., Amrhein, T., Gupta, R., Halabi, S., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0189-7 · Cited by 254 · cited 1×
“Others have taken the autonomous processes of these machines to amount to something more action-like, rendering the description of these interactions more akin to cooperation, collaborations or partnerships (e.g., Patel et al. (2019) for diagnostic imaging, see Nyholm (2018) for a general analysis).”
Background
Research works printed but not linked: 3
- Flach P (2019) Performance evaluation in machine learning: the good, the bad, the ugly, and the way forward. Proc AAAI Conf Artif Intell 33(1):9808–9814
- Heilinger J-C (2020) Cosmopolitan responsibility. Global injustice, relational equality, and individual agency. de Gruyter, Berlin
- Mozannar H, Sontag D (2020) Consistent estimators for learning to defer to an expert. In: International conference on machine learning. PMLR, pp 7076–7087
Cited by
Multimodal generative AI for medical image interpretation
Rao, V. M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E. J., & Rajpurkar, P. (2025) · Nature · doi:10.1038/s41586-025-08675-y · Cited by 179
“Although some studies have investigated physician decision-making when using AI 123,124 , there is still a great need for further work studying reliance in the context of GenMI.”
The Ethics of AI Ethics. A Constructive Critique
Heilinger, J. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00557-9 · Cited by 120
“…a number of central ethical issues and problems have been identified and established as ‘core questions’ arising in the context of AI 2 : • How can we understand, explain and control—if ever—the internal workings within a complex AI system (cf. e.g. Kempt et al., 2022; Mittelstadt et al., 2019)?”
BackgroundSelf-citationSchillaci, C. E., de Cosmo, L. M., Piper, L., Nicotra, M., & Guido, G. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.123025 · Cited by 97
Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22.
Cecconi, M., Greco, M., Shickel, B., Angus, D. C., Bailey, H., Bignami, E., Calandra, T., Celi, L. A., Einav, S., Elbers, P., et al. (2025) · Critical Care · doi:10.1186/s13054-025-05532-2 · Cited by 80
Heinrichs, H., Kies, A., Nagel, S. K., & Kiessling, F. (2025) · Journal of Medical Internet Research · doi:10.2196/74187 · Cited by 66
“While ongoing research explores AI agents as potential teammates in health care [52] or as providers of second opinions [53], their success depends on how they integrate without undermining the central expertise of physicians.”
Self-citation
Metadata
- Authors
- Kempt, H., Heilinger, J.-C., & Nagel, S. K.
- Affiliations
- RWTH Aachen University (DE)
- Corresponding author
- Kempt, Hendrik
- Venue
- AI & SOCIETY
- Published
- 2022-03-16
- Volume
- 38
- Issue
- 4
- Pages
- 1407-1414
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- OpenAlex
- W4221081267
- Cited by
- 31
- References
- 23
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 7 · 2023: 3 · 2022: 1
- FWCI
- 1.74
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI & Society (publishedVersion, cc-by)
- RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
- Funders
- RWTH Aachen University
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- diagnostic decision support systems, meaningful human control, physician moral responsibility, clinical workflow, burden of proof, human-AI collaboration, medical ethics, artificial intelligence in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “I’m afraid I can’t let you do that, Doctor”: meaningful disagreements with AI in medical contexts. https://app.cochat.ai/papers/10.1007/s00146-022-01418-x
- The paper
- Kempt, H., Heilinger, J.-C., & Nagel, S. K. (2023). “I’m afraid I can’t let you do that, Doctor”: meaningful disagreements with AI in medical contexts. AI & SOCIETY. https://doi.org/10.1007/s00146-022-01418-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.