Epistemo-ethical constraints on AI-human decision making for diagnostic purposes
Open accessAbstract
This paper approaches the interaction of a health professional with an AI system for diagnostic purposes as a hybrid decision making process and conceptualizes epistemo-ethical constraints on this process. We argue for the importance of the understanding of the underlying machine epistemology in order to raise awareness of and facilitate realistic expectations from AI as a decision support system, both among healthcare professionals and the potential benefiters (patients). Understanding the epistemic abilities and limitations of such systems is essential if we are to integrate AI into the decision making processes in a way that takes into account its applicability boundaries. This will help to mitigate potential harm due to misjudgments and, as a result, to raise the trust—understood here as a belief in reliability of—in the AI system. We aim at a minimal requirement for AI meta-explanation which should distinguish machine epistemic processes from similar processes in human epistemology in order to avoid confusion and error in judgment and application. An informed approach to the integration of AI systems into the decision making for diagnostic purposes is crucial given its high impact on health and well-being of patients.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The inherent statistical and predictive architecture combined with the intransparency of LLMs presents significant hurdles in validating the clinical accuracy and reliability of their outputs.
“The inherent statistical and predictive architecture combined with the intransparency of LLMs presents significant hurdles in validating the clinical accuracy and reliability of their outputs91–93.”
From The ethics of ChatGPT in medicine and healthcare: a systematic review on large language models (LLMs) (2024) · cites this paper as [93]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Zadeh, L. A. (1965) · Information and Control · doi:10.1016/s0019-9958(65)90241-x · Cited by 67,855 · see citing works
The Concept of a Linguistic Variable and Its Application to Approximate Reasoning—I
Zadeh, L. A. (1975) · Information Sciences · doi:10.1016/0020-0255(75)90036-5 · Cited by 12,187 · cited 1× · see citing works
“In the domain of soft computing, the distinction between hard and fuzzy concepts was best explained by Zadeh (1965, 1975) and is insightful for the explanation of uncertainty.”
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1× · see citing works
“…areas of AI in healthcare are diverse and include ophthalmology (e.g. screening for glaucoma, diabetic retinopathy, hypertensive retinopathy; cf. Gulshan et al., 2016; Poplin et al., 2018), dermatology (e.g. skin cancer, cosmetic care; cf. e.g. Elder et al., 2021; Rundle et al., 2021),…”
BackgroundFuzzy sets as a basis for a theory of possibility
Zadeh, L. A. (1999) · Fuzzy Sets and Systems · doi:10.1016/s0165-0114(99)80004-9 · Cited by 7,301 · see citing works
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954 · cited 1× · see citing works
“As diagnosis is itself a fuzzy set, the output 18 In this context, this term refers to an artificial system capable of thought, consciousness and emotions in a genuinely human way (Searle 1980).”
BackgroundPrediction of cardiovascular risk factors from retinal fundus photographs via deep learning
Poplin, R., Varadarajan, A. V., Blumer, K., Liu, Y., McConnell, M. V., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0195-0 · Cited by 1,865 · cited 1× · see citing works
“…are diverse and include ophthalmology (e.g. screening for glaucoma, diabetic retinopathy, hypertensive retinopathy; cf. Gulshan et al., 2016; Poplin et al., 2018), dermatology (e.g. skin cancer, cosmetic care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et…”
BackgroundThe rise of artificial intelligence in healthcare applications
Bohr, A., & Memarzadeh, K. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00002-2 · Cited by 1,636 · see citing works
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215 · see citing works
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · see citing works
(2019) · Cited by 986 · see citing works
Chollet, F. (2017) · Cited by 742 · see citing works
Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening
Wu, N., Phang, J., Park, J., Shen, Y., Huang, Z., Zorin, M., Jastrzȩbski, S., Févry, T., Katsnelson, J., Kim, E. S., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2945514 · Cited by 695 · cited 2× · see citing works
“…2018), dermatology (e.g. skin cancer, cosmetic care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and pulmonology (a range of acute and chronic conditions; see e.g. Kaplan…”
BackgroundOn the ethics of algorithmic decision-making in healthcare
Grote, T., & Berens, P. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-105586 · Cited by 551 · cited 1× · see citing works
“…specific epistemic responsibilities of each component (on the need to clearly delineate the epistemic tasks of AI and the human expert in the hybrid intelligence cf. van Baalen et al., 2021; Boon, 2020), and to minimize “trade-offs at the epistemic and the normative level” (Grote & Berens, 2020).”
BackgroundJussupow, E., Spohrer, K., Heinzl, A., & Gawlitza, J. (2021) · Information Systems Research · doi:10.1287/isre.2020.0980 · Cited by 422 · cited 1× · see citing works
“7 Despite being framed in different terms, augmented rather than hybrid decision making, Jussupow et al. (2021) offer interesting insights about meta-cognitive skills required for the incorporation of AI output in the diagnostic process of a medical professional.”
BackgroundCheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · cited 2× · see citing works
“For simplic-ity’s sake, we will use a hypothetical case, similar to the one generated by the system in Irvin et al. (2019).”
MethodSwabey, W. C., Ogden, C. K., Richards, I. A., Postgate, J. P., Malinowski, B., & Crookshank, F. G. (1924) · The Philosophical Review · doi:10.2307/2179408 · Cited by 385 · see citing works
AI-based computer-aided diagnosis (AI-CAD): the latest review to read first
Fujita, H. (2020) · Radiological Physics and Technology · doi:10.1007/s12194-019-00552-4 · Cited by 308 · see citing works
Understanding from Machine Learning Models
Sullivan, E. (2019) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axz035 · Cited by 229 · cited 2× · see citing works
“17 In this respect, note Sullivan (2019) who claims that models such as DNN contribute to “how-possibly explanations”, i.e. explanations about possible correlations and dependencies, and not “how-actually” or “why-questions” about the actual phenomenon.”
BackgroundArtificial intelligence in the interpretation of breast cancer on MRI
Sheth, D., & Giger, M. L. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26878 · Cited by 220 · cited 1× · see citing works
“…care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and pulmonology (a range of acute and chronic conditions; see e.g. Kaplan et al. 2021; Almaslukh, 2021; Almalki et al. 2021).”
BackgroundKaplan, A. G., Cao, H., FitzGerald, J. M., Iannotti, N., Yang, E., Kocks, J. W. H., Κostikas, Κ., Price, D. B., Reddel, H. K., Tsiligianni, I., et al. (2021) · The Journal of Allergy and Clinical Immunology In Practice · doi:10.1016/j.jaip.2021.02.014 · Cited by 219 · cited 1× · see citing works
“…care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and pulmonology (a range of acute and chronic conditions; see e.g. Kaplan et al. 2021; Almaslukh, 2021; Almalki et al. 2021).”
BackgroundArtificial intelligence for anterior segment diseases: Emerging applications in ophthalmology
Ting, D. S. J., Foo, V. H. X., Yang, L. W. Y., Sia, J. T., Ang, M., Lin, H., Chodosh, J., Mehta, J. S., & Ting, D. S. W. (2020) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2019-315651 · Cited by 205 · see citing works
The current state of artificial intelligence in ophthalmology
Kapoor, R., Walters, S. P., & Al‐Aswad, L. A. (2018) · Survey of Ophthalmology · doi:10.1016/j.survophthal.2018.09.002 · Cited by 198 · see citing works
Black-box artificial intelligence: an epistemological and critical analysis
Carabantes, M. (2019) · AI & Society · doi:10.1007/s00146-019-00888-w · Cited by 180 · see citing works
Ogden, C. K., & Richards, I. A. (1969) · Cited by 122 · see citing works
Your evidence? Machine learning algorithms for medical diagnosis and prediction
Heinrichs, B., & Eickhoff, S. B. (2019) · Human Brain Mapping · doi:10.1002/hbm.24886 · Cited by 99 · see citing works
A Novel Method for COVID-19 Diagnosis Using Artificial Intelligence in Chest X-ray Images
Almalki, Y. E., Qayyum, A., Irfan, M., Haider, N., Głowacz, A., Alshehri, F. M., Alduraibi, S. K., Alshamrani, K. A., Basha, M. A. A., Alduraibi, A. K., et al. (2021) · Healthcare · doi:10.3390/healthcare9050522 · Cited by 88 · cited 1× · see citing works
“…care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and pulmonology (a range of acute and chronic conditions; see e.g. Kaplan et al. 2021; Almaslukh, 2021; Almalki et al. 2021).”
BackgroundMayo, R. C., Kent, D., Sen, L. Q. C., Kapoor, M. M., Leung, J. W. T., & Watanabe, A. T. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-018-0168-6 · Cited by 87 · cited 1× · see citing works
“…2016; Poplin et al., 2018), dermatology (e.g. skin cancer, cosmetic care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and pulmonology (a range of acute and chronic…”
BackgroundThe role of artificial intelligence in cosmetic dermatology—Current, upcoming, and future trends
Elder, A., Ring, C. M., Heitmiller, K., Gabriel, Z., & Saedi, N. (2020) · Journal of Cosmetic Dermatology · doi:10.1111/jocd.13797 · Cited by 87 · cited 1× · see citing works
“…retinopathy, hypertensive retinopathy; cf. Gulshan et al., 2016; Poplin et al., 2018), dermatology (e.g. skin cancer, cosmetic care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger,…”
BackgroundFrom clinical decision support to clinical reasoning support systems
van Baalen, S., Boon, M., & Verhoef, P. (2021) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13541 · Cited by 84 · cited 2× · see citing works
“Here we are dealing with a new phenomenon of hybrid epistemology (or “hybrid intelligence” as in van Baalen et al., 2021).”
BackgroundCausality and Probability in the Sciences
Russo, F., & Williamson, J. (2007) · Cited by 80 · see citing works
Deep neural network improves the estimation of polygenic risk scores for breast cancer
Badré, A., Zhang, L., Muchero, W., Reynolds, J. C., & Pan, C. (2020) · Journal of Human Genetics · doi:10.1038/s10038-020-00832-7 · Cited by 66 · cited 1× · see citing works
“…(e.g. skin cancer, cosmetic care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and pulmonology (a range of acute and chronic conditions; see e.g. Kaplan et al. 2021;…”
BackgroundKudina, O., & de Boer, B. (2021) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13535 · Cited by 34 · see citing works
A Lightweight Deep Learning‐Based Pneumonia Detection Approach for Energy‐Efficient Medical Systems
Almaslukh, B. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/5556635 · Cited by 32 · cited 1× · see citing works
“…care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and pulmonology (a range of acute and chronic conditions; see e.g. Kaplan et al. 2021; Almaslukh, 2021; Almalki et al. 2021).”
BackgroundArtificial intelligence in dermatology
Rundle, C. W., Hollingsworth, P., & Dellavalle, R. P. (2021) · Clinics in Dermatology · doi:10.1016/j.clindermatol.2021.03.011 · Cited by 16 · cited 1× · see citing works
“…hypertensive retinopathy; cf. Gulshan et al., 2016; Poplin et al., 2018), dermatology (e.g. skin cancer, cosmetic care; cf. e.g. Elder et al., 2021; Rundle et al., 2021), mammography (see e.g. Mayo et al., 2019; Wu et al. 2019; Badré et al. 2021; Liu et al. 2021; Sheth & Giger, 2020), and…”
BackgroundHow Scientists Are Brought Back into Science—The Error of Empiricism
Boon, M. (2020) · Human perspectives in health sciences and technology · doi:10.1007/978-3-030-25001-0_4 · Cited by 15 · cited 2× · see citing works
“This should help to draw the line between the specific epistemic responsibilities of each component (on the need to clearly delineate the epistemic tasks of AI and the human expert in the hybrid intelligence cf. van Baalen et al., 2021; Boon, 2020), and to minimize “trade-offs at the epistemic and the normative level”…”
Background
Flagged first, then by citations
1 of the 36 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Liu, Y., Zhong, X., Cheng, J., Pi, Y., Cai, H., Jiang, L., Yang, P., Xiang, Y., Jianan, W., Li, L., Yi, Z., & Zhao, Z. (2021). Automatic rapid identification of malignant carcinoma bone metastatic lesions by deep neural network based artificial intelligence. Journal of Nuclear Medicine, 62(1), 1180.
- Peirce, C. S. (1998). The essential peirce. Peirce edition project (Vol. 2). Indiana University Press.
- Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., Lungren, M. P., & Ng, A. Y. (2017). CheXnet: Radiologist-level pneumonia detec- tion on chest X-rays with deep learning. arXiv preprint. arXiv: 1711. 05225.
Cited by
Sorted by citations
Haltaufderheide, J., & Ranisch, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01157-x · Cited by 396 · see citing works
“95 The inclination of LLMs to output erroneous information underscores the need for human oversight and continual validation of machine-generated output, as our dataset demonstrates.”
BackgroundAI in Context: Harnessing Domain Knowledge for Smarter Machine Learning
Miller, T. I., Durlik, I., Łobodzińska, A., Dorobczyński, L., & Jasionowski, R. (2024) · Applied Sciences · doi:10.3390/app142411612 · Cited by 40 · see citing works
Disrupted self, therapy, and the limits of conversational AI
Babushkina, D., & de Boer, B. (2024) · Philosophical Psychology · doi:10.1080/09515089.2024.2397004 · Cited by 13 · see citing works
“When based on text-generating software, CAs scrap the data-carriers (Babushkina & Votsis, 2022) and generate platitudinal lines in a conversation.”
Self-citationBarman, K. G., Caron, S., Sullivan, E., de Regt, H. D., de Austri, R. D., Boon, M., Färber, M., Fröse, S., Golling, T., Lopez, L. G., et al. (2025) · The European Physical Journal C · doi:10.1140/epjc/s10052-025-14707-8 · Cited by 35 · see citing works
“It is claimed that LLMs are transforming scientific research by creating a ’hybrid intelligence’ that augments human cognition and reshapes research processes [29, 30].”
Steerling, E., Svedberg, P., Nilsén, P., Siira, E., & Nygren, J. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1565080 · Cited by 13 · see citing works
Pozzi, G., & Durán, J. M. (2024) · AI & Society · doi:10.1007/s00146-024-01875-6 · Cited by 10 · see citing works
“Relatedly, Babushkina and Votsis (2022) consider primarily how epistemological constraints of ML systems in the context of medical diagnoses lead to ethical considerations in terms of epistemic responsibility.”
Harmful epistemic dependence on medical machine learning and its moral implications
Pozzi, G., Buijsman, S., & van den Hoven, J. (2025) · Journal of Medical Ethics · doi:10.1136/jme-2024-110552 · Cited by 7 · see citing works
“For instance, Babushkina and Votsis 42 argue that human–AI collaboration in healthcare gives rise to what they call ‘hybrid epistemology,’ which entails an entanglement of human cognitive processes and ML procedures that uniquely transform information.”
Enabler or inhibitor? The effect of AI crafting on presenteeism among medical staff
Liu, R., Deng, W., Wang, T., & Yang, T. (2025) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2025.2492632 · Cited by 7 · see citing works
“These challenges often involve errors or mis-judgements in decision-making processes (Babushkina & Votsis, 2022) which, in certain cases, may have life-threatening consequences.”
Curiello, S., Iannuzzi, E., Meissner, D., & Nigro, C. (2025) · European Journal of Innovation Management · doi:10.1108/ejim-01-2024-0078 · Cited by 7 · see citing works
The Ethics and Epistemology of Clinician-AI Disagreement in Medicine: Beyond Opposition
Pozzi, G., Sand, M., & Jongsma, K. R. (2026) · The American Journal of Bioethics · doi:10.1080/15265161.2026.2632008 · Cited by 3 · see citing works
Michiel, S., Moissact, I., & Sean, C. (2024) · Safety and Health For Medical Workers · doi:10.69725/shmw.v1i2.151 · Cited by 3 · see citing works
Cognition Technology: AI and the Disruption of Epistemic Agency
Babushkina, D. (2026) · Springer eBooks · doi:10.1007/978-3-032-15112-4_47 · Cited by 2 · see citing works
Self-citationAI literacy in healthcare organisations: implementing article 4 of the EU AI Act with M-SHALF
Palmieri, S., Declerck, J., Priou, S., Lea, N., & Kalra, D. (2026) · npj Health Systems · doi:10.1038/s44401-026-00104-0 · Cited by 4 · see citing works
Ethical AI Development and Deployment
Gaurav, A., Gupta, B. B., & Castiglione, A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-3860-5.ch004 · Cited by 2 · see citing works
Human-AI Interaction, Moral Responsibility and Epistemic Credit
Largo, J. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01104-6 · Cited by 1 · see citing works
Conceptual Disconnect: The Prospect of Concept-Based AI Explanations
Buijsman, S. (2025) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-031-99654-2_1
Duration and Externalized Intelligence: A Bergsonian Critique of Large Language Models
Hamamra, B. (2026) · Human Arenas · doi:10.1007/s42087-026-00692-1
The Struggle for Authenticity and the Problem of Simulated Narratives
Babushkina, D. (2026) · The Journal of Humanistic Counseling · doi:10.1002/johc.70063
Self-citationMehan, A. (2026) · Urban Planning · doi:10.17645/up.12553
“…twins, and platform urbanism, and its associated tropes create false representations of digital solutions for complex urban problems without acknowledging the material and ecological costs of the infrastructures underpinning them (Babushkina & Votsis, 2022; Chiappini, 2020; Cristofari, 2023).”
da Costa, A. R., Pedrotti, C. H. S., & Accorsi, T. A. D. (2026) · Einstein (São Paulo) · doi:10.31744/einstein_journal/2026ae2371
- Authors
- Babushkina, D., & Votsis, A.
- Affiliations
- University of Twente (NL)
- Corresponding author
- Babushkina, Dina
- Venue
- Ethics and Information Technology
- Published
- 2022-04-19
- Volume
- 24
- Issue
- 2
- ISSN
- 1388-1957 (ISSN-L)
- 1572-8439
- OpenAlex
- W4224209498
- Cited by
- 20
- References
- 36
- Citations by year
- 2026: 8 · 2025: 6 · 2024: 6
- FWCI
- 0.91
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Ethics and Information Technology (publishedVersion, cc-by)
- University of Twente Research Information (publishedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- artificial intelligence in healthcare, diagnostic decision-making, AI decision support systems, trust in AI, human-AI interaction, hybrid decision-making, patient trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
