What Are Humans Doing in the Loop? Co-Reasoning and Practical Judgment When Using Machine Learning-Driven Decision Aids
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Within the ethical debate on Machine Learning-driven decision support systems (ML_CDSS), notions such as "human in the loop" or "meaningful human control" are often cited as being necessary for ethical legitimacy. In addition, ethical principles usually serve as the major point of reference in ethical guidance documents, stating that conflicts between principles need to be weighed and balanced against each other. Starting from a neo-Kantian viewpoint inspired by Onora O'Neill, this article makes a concrete suggestion of how to interpret the role of the "human in the loop" and to overcome the perspective of rivaling ethical principles in the evaluation of AI in health care. We argue that patients should be perceived as "fellow workers" and epistemic partners in the interpretation of ML_CDSS outputs. We further highlight that a meaningful process of integrating (rather than weighing and balancing) ethical principles is most appropriate in the evaluation of medical AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Theory of Justice: Revised Edition
Rawls, J. (1999) · doi:10.4159/9780674042582 · Cited by 9,455
Epistemic Injustice: Power and the Ethics of Knowing
Fricker, M. (2007) · doi:10.1093/acprof:oso/9780198237907.001.0001 · Cited by 9,198 · cited 1×
“A key claim in these accounts is that patients are knowers in a medically relevant sense and that doctors commit a kind of “epistemic injustice” (Fricker 2007) when they fail to respect patients’ status as knowers.”
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774 · cited 2×
“The mentioned AI4People’s Ethical Framework highlights the task of weighing efficacy against control over decision-making (Floridi et al. 2018).”
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019) · Nature Medicine · doi:10.1038/s41591-018-0268-3 · Cited by 3,148
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · cited 1×
“The potential pitfalls identified relate to, among other things, the context of patient safety, the fragmentation of clinical workflows, negative impact on user skills (“de-skilling” of physicians) or lack of interoperability (Sutton et al. 2020).”
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402
Principles alone cannot guarantee ethical AI
Mittelstadt, B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0114-4 · Cited by 1,886 · cited 4×
“Progress in data science and in the availability of clinical data has recently fueled a rapid development and the introduction of Machine Learning-driven CDSS (ML_CDSS) in various clinical workflows (London 2018).”
InfluentialAutomation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“” It has been documented that clinicians often substitute their own judgment with CDSS recommendations—even in cases where their original judgment is superior (Goddard et al. 2012).”
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · cited 1×
“From an ethical perspective, epistemic issues (e.g. misguided or inconclusive evidence) need to be considered together with normative aspects (e.g. unfair outcomes) and problems of traceability (e.g. explainability) of the CDSS functioning and concrete recommendations (Morley et al. 2020).”
Cited by
From “Human in the Loop” to a Participatory System of Governance for AI in Healthcare
Griffen, Z., & Owens, K. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2377114 · Cited by 22
Battle of the Bots: Solving Clinical Cases in Osteoarticular Infections With Large Language Models
Borgonovo, F., Matsuo, T., Petri, F., Amin Alavi, S. M., Mazudie Ndjonko, L. C., Gori, A., & Berbari, E. F. (2025) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2025.100230 · Cited by 10
Scapegoat-in-the-loop? Human control over medical AI and the (mis) attribution of responsibility
Ranisch, R. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2377137 · Cited by 6
Metadata
- Authors
- Salloch, S., & Eriksen, A.
- Venue
- The American Journal of Bioethics
- Publisher
- Informa UK Limited
- Published
- 2024-05-20
- Volume
- 24
- Issue
- 9
- Pages
- 67-78
- ISSN
- 1526-5161 (ISSN-L)
- 1536-0075
- OpenAlex
- W4398143245
- Cited by
- 51
- References
- 24
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What Are Humans Doing in the Loop? Co-Reasoning and Practical Judgment When Using Machine Learning-Driven Decision Aids. https://app.cochat.ai/papers/10.1080/15265161.2024.2353800
- The paper
- Salloch, S., & Eriksen, A. (2024). What Are Humans Doing in the Loop? Co-Reasoning and Practical Judgment When Using Machine Learning-Driven Decision Aids. The American Journal of Bioethics. https://doi.org/10.1080/15265161.2024.2353800
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.