Patient safety and quality improvement: Ethical principles for a regulatory approach to bias in healthcare machine learning
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Accumulating evidence demonstrates the impact of bias that reflects social inequality on the performance of machine learning (ML) models in health care. Given their intended placement within healthcare decision making more broadly, ML tools require attention to adequately quantify the impact of bias and reduce its potential to exacerbate inequalities. We suggest that taking a patient safety and quality improvement approach to bias can support the quantification of bias-related effects on ML. Drawing from the ethical principles underpinning these approaches, we argue that patient safety and quality improvement lenses support the quantification of relevant performance metrics, in order to minimize harm while promoting accountability, justice, and transparency. We identify specific methods for operationalizing these principles with the goal of attending to bias to support better decision making in light of controllable and uncontrollable factors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is advised that trial outcomes are also evaluated per clinically relevant user group12 or per affected non-user group (also in terms of algorithmic bias)12,24,91.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [91]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Performance analysis by population subgroups or specific problematic use cases is recommended to identify algorithmic bias6,7,23,24,26,42,61,72,79,91.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [91]
In the citing paper
“To further facilitate the implementation and consecutive monitoring, the literature recommends automatic AIPM deployment (moving software from testing to production environments with automated processes) and the facilitation of shadow deployment66,91, which enables prospective local validation (see phase 3) of new versions and updates19.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [91]
In the citing paper
“For example, end users may be made aware of the possibility of automation bias and trained to maintain vigilance22,26,57,87,91.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [91]
In the citing paper
“Second, the monitoring of dataset shift is also repeatedly mentioned in the literature5,10,22,72,79,91.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [91]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 2×
“7,8,15 When model errors are unevenly distributed then, so too are the risks.”
BackgroundUnequal treatment: confronting racial and ethnic disparities in health care
(2003) · Choice Reviews Online · doi:10.5860/choice.40-5843 · Cited by 7,043
Social determinants of health inequalities
Marmot, M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)71146-6 · Cited by 5,086
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Self-citationWord embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · cited 1×
“7 16 It is less clear how to deal with this challenge.”
BackgroundMachine Learning and Health Care Disparities in Dermatology
Adamson, A. S., & Smith, A. (2018) · JAMA Dermatology · doi:10.1001/jamadermatol.2018.2348 · Cited by 639 · cited 1×
“14 Attending to variations in decisional accuracy across subgroups can promote justice by encouraging due diligence with respect to appropriate model use across these groups.”
BackgroundHidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411
Hernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., & Shah, N. H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa088 · Cited by 410
Can AI Help Reduce Disparities in General Medical and Mental Health Care?
(2019) · AMA Journal of Ethics · doi:10.1001/amajethics.2019.167 · Cited by 401
Assessing risk, automating racism
Benjamin, R. (2019) · Science · doi:10.1126/science.aaz3873 · Cited by 378 · cited 2×
“The problem of bias for healthcare ML Recent evidence indicates that model performance discrepancies, as a function of protected identities (meaning characteristics protected by human and civil rights legislation [eg, race, gender]), can have significant health implications for vulnerable groups, potentially worsening…”
Background
Research works printed but not linked: 9
- 2. Zhang H, Lu AX, Abdalla M, McDermott M, Ghassemi M. Hurtful words: quantifying biases in clinical contextual word embeddings. In: proceedings of the ACM Conference on Health, Inference, and Learning; 2020: 110–20.
- Smedley, 2002, Unequal Treatment: Confronting Racial and Ethnic Disparities in Healthcare
- 9. Buolamwini J, Gebru T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In: conference on Fairness, Accountability and Transparency; 2018; 77–91.
- 10. Dwork C, Hardt M, Pitassi T, Reingold O, Zemel R. Fairness through awareness. In proceedings of the 3rd Innovations in Theoretical Computer Science Conference; 2012; 214–26.
- 12. De-Arteaga M, Romanov A, Wallach H, et al. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In: proceedings of the Conference on Fairness, Accountability, and Transparency; 2019: 120–28.
- 13. Bolukbasi T, Chang KW, Zou JY, Saligrama V, Kalai AT. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In: NIPS ’16: Proceedings of the 30th International Conference on Neural Information Processing Systems; 2016: 4356–64.
- 15. Zhang H, Lu AX, Abdalla M, McDermott M, Ghassemi M. Hurtful words: quantifying biases in clinical contextual word embeddings. In proceedings of the ACM Conference on Health, Inference, and Learning 2020; 110–20.
- 19. Cummings M. Automation bias in intelligent time critical decision support systems. In: AIAA 1st Intelligent Systems Technical Conference; 2004.
- 26. Drysdale E, Dolatabadi E, Chivers C, et al. White Paper: implementing AI in healthcare. Vector-SickKids Health AI Deployment Symposium. 2020. https://vectorinstitute.ai/wp-content/uploads/2020/03/implementing-ai-in-healthcare.pdf Accessed May 20, 2020.
Cited by
Items 101–130 of 130
Leonard, F., Lyttle, M. D., O’Sullivan, D. M., Gilligan, J., Roland, D., & Barrett, M. J. (2025) · medRxiv · doi:10.1101/2025.06.29.25330501 · Cited by 1
Justo Ñaupa, A. (2021) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 1
Influence of Safety Climate on Trucking Accidents: A Machine Learning ApproachPreprint
Lan, T., Goh, Y. M., Safiena, S., Lytle, B., He, Y., & Huang, Y.-H. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4088440
Effect of Computation and Cognitive Bias in Healthcare Intelligence and Pharmacogenomics
Panda, G. K., Sahu, I. K., & Sahu, D. P. (2022) · Studies in computational intelligence · doi:10.1007/978-981-19-2416-3_4
HealSmart Homes: Integrating Technology for Enhanced Convalescent Care
Kanimozhi, T. (2024) · Journal of Electrical Systems · doi:10.52783/jes.3537
Ethical dilemmas in wound care: challenges and opportunity for healthcare professionals
Subrata, S. A. (2025) · British Journal of Healthcare Management · doi:10.12968/bjhc.2025.0021
Antes, A. L., Burrous, S., Sisk, B. A., Schuelke, M. J., Keune, J. D., & DuBois, J. M. (2021) · Figshare · doi:10.6084/m9.figshare.15026439
Antes, A. L., Burrous, S., Sisk, B. A., Schuelke, M. J., Keune, J. D., & DuBois, J. M. (2021) · Figshare · doi:10.6084/m9.figshare.15026430
Research Group Critical Information Infrastructures (Cii), Sunyaev, A., Renner, M., Toussaint, P. A., Thiebes, S., & Lins, S. (2022) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000150078
Antes, A. L., Burrous, S., Sisk, B. A., Schuelke, M. J., Keune, J. D., & DuBois, J. M. (2021) · Figshare · doi:10.6084/m9.figshare.15026436
Antes, A. L., Burrous, S., Sisk, B. A., Schuelke, M. J., Keune, J. D., & DuBois, J. M. (2021) · Figshare · doi:10.6084/m9.figshare.15026433
Diego Chavez, C. (2024) · HARVEST (University of Saskatchewan)
Artificial Intelligence as a Catalyst for Precision Medicine
Battapothu, V. (2025) · IJARCCE · doi:10.17148/ijarcce.2023.121228
Artificial intelligence-enabled personalized medicine: strategies and challenges
Chavda, V. P., Vyas, A., Rathore, N., Sheth, T. J., Shah, N., & Shah, S. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-34059-8.00017-8
Varagani, S. H. N. S. (2025) · International Journal on Research and Development - A Management Review · doi:10.65521/ijrdmr.v14i1.1527
Ponduru, P. S. (2026) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem62213
McKeegan, G. F. (2026) · Evidence-based practices in behavioral health · doi:10.1007/978-3-032-18372-9_5
Xie, X., Liu, C., Liu, Y., & Zhou, Y. (2026) · Chinese general practice journal. · doi:10.1016/j.cgpj.2026.100114
Embracing Digital as a Paradigm Shift in Medical Affairs
Lalwani, A., Zdon, J., Addicks, M., Marzouki, Y., Seo, D., & Chisholm, O. (2026) · Pharmaceutical Medicine · doi:10.1007/s40290-026-00623-z
Al-Hinai, S., Al-Balushi, A., & Al-Maskari, M. (2026) · International Journal of Pharmaceutical and Phytopharmacological Research · doi:10.51847/ni2b8sajli
Continuous assurance for AI-driven clinical decision support systems
Al‐Horani, R. A., & Tadesse, A. F. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1870819 · Cited by 1
Papanicolaou, D., Perdikeas, S., & Jones, G. B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1880641
Sabri, O., & Alabdali, S. A. (2026) · Healthcare · doi:10.3390/healthcare14162506
Regulations and ethical considerations in digital pathology and artificial intelligence
Gullapalli, R. R., & Parwani, A. V. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-40383-5.00018-2
yman, C., Benamri, I., Ferjouchia, H., Ferjouchia, Z., & Rabhi, A. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12460-x
Mahfouz, K., Abdelaziz, R., Adel, S., Mostafa, D., Nabil, T., & Saad, R. (2025) · Journal of Health Informatics and Digital Systems · doi:10.68159/y292926779
Assess Knowledge of Healthcare Technologies and informatics enabled by Artificial Intelligence
Altamimi, A. S. (2024) · Journal of Propulsion Technology
Transparency Analysis of Deep Learning Models in Medical Data Using SHAP and LIME
Evander, A., & Quinn, L. A. (2026) · Jurnal Teknik Informatika C I T Medicom
Precision Social Medicine: Bridging the Gap Between Social Care and Health Systems Integration
Gabbay, J. M., Garg, A., Cabana, M. D., & Fiori, K. P. (2026) · Hospital Pediatrics · doi:10.1542/hpeds.2026-009715
Fehr, J., Citro, B., Malpani, R., Lippert, C., & Madai, V. I. (2024) · doi:10.3389/fdgth.2024.1267290.s001
Metadata
- Authors
- McCradden, M. D., Joshi, S., Anderson, J. A., Mazwi, M., Goldenberg, A., & Zlotnik Shaul, R.
- Affiliations
- Hospital for Sick Children (CA)
- Vector Institute (CA)
- University of Toronto (CA)
- Canadian Institute for Advanced Research (CA)
- Bioethics Department, The Hospital for Sick Children, Toronto, Ontario, Canada
- Corresponding author
- McCradden, Melissa D
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-06-25
- Volume
- 27
- Issue
- 12
- Pages
- 2024-2027
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7727331
- OpenAlex
- W3037637030
- Cited by
- 111
- References
- 14
- Citations by year
- 2026: 17 · 2025: 15 · 2024: 21 · 2023: 20 · 2022: 18 · 2021: 15 · 2020: 5
- FWCI
- 19.73
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic bias, machine learning in healthcare, patient safety, quality improvement, ethical principles, social inequality, health equity, accountability, justice, transparency, bias quantification, healthcare decision-making, regulatory approaches
- MeSH terms
- Artificial Intelligence, Data Collection, Humans, Prejudice, Government Regulation, Healthcare Disparities, Quality Improvement, Patient Safety, Social Determinants of Health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.