Cited by 645Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

Ethical Machine Learning in Healthcare

Open access
Chen, Irene Y., Pierson, Emma, Rose, Sherri, Joshi, Shalmali, Ferryman, Kadija, Ghassemi, Marzyeh·2021·Annual Review of Biomedical Data Science

Abstract

The use of machine learning (ML) in healthcare raises numerous ethical concerns, especially as models can amplify existing health inequities. Here, we outline ethical considerations for equitable ML in the advancement of healthcare. Specifically, we frame ethics of ML in healthcare through the lens of social justice. We describe ongoing efforts and outline challenges in a proposed pipeline of ethical ML in health, ranging from problem selection to postdeployment considerations. We close by summarizing recommendations to address these challenges.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1146/annurev-biodatasci-092820-114757

Cites

92 of 92 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Memory raises significant concerns regarding privacy, data management, and bias.

    “At the same time, memory also raises significant concerns regarding privacy, data management, and bias [17].”

    From Medical AI Agents for Clinical Decision Support: Viewpoint Using the Planning, Action, Reflection, and Memory (PARM) Analytical Lens (2026) · cites this paper as [17]

    The Data Collection section discusses bias in health data, and its Population-Specific Data Losses subsection describes how patient-privacy protections can introduce statistical bias (page unavailable). These passages support concerns about privacy and bias in healthcare data, but they do not say that memory raises those concerns or address memory-related data management. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • A wide variety of governance best practices in the form of frameworks, governance structures, checklists, and guidelines have been proposed.

    “As AI solutions have emerged, a wide variety of governance best practices in the form of frameworks, governance structures, checklists, and guidelines have been proposed7–23.”

    From Advancing healthcare AI governance through a comprehensive maturity model based on systematic review (2026) · cites this paper as [9]

    The Regulation and Model and Data Documentation sections describe proposed frameworks and checklists for healthcare ML. They do not establish the claimed range of proposed governance structures and guidelines; the Regulation section instead says comprehensive guidance for healthcare model research and deployment does not yet exist. Page numbers are unavailable in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • A recurring theme across multiple articles was the necessity for an overarching governance body to assist in selecting AI tools and guide validation, implementation, and monitoring processes.

    “A recurring theme across multiple articles was the necessity for an overarching governance body7–10,17. This entity would not only assist in selecting AI tools and software for implementation but also guide the validation, implementation, and monitoring processes.”

    From Advancing healthcare AI governance through a comprehensive maturity model based on systematic review (2026) · cites this paper as [9]

    The Postdeployment Considerations > Regulation section says the FDA regulates healthcare ML models and calls for a comprehensive framework to audit and regulate them, including postmarket evaluation. It does not call for an overarching governance body that selects AI tools or guides their validation and implementation. The indexed passages provide no page numbers. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • Most articles briefly addressed the importance of clear objectives for AI implementation in healthcare, with only a few offering actionable guidance.

    “Most articles briefly addressed the importance of clear objectives for AI implementation in healthcare, with only a few offering actionable guidance9,11,14,17,21.”

    From Advancing healthcare AI governance through a comprehensive maturity model based on systematic review (2026) · cites this paper as [9]

    The Recommendations section says researchers should reflect on a model’s goals and articulate them in a preanalysis plan, supporting the importance of clear objectives. It does not assess whether most articles address objectives only briefly or whether few offer actionable guidance. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • AI governance frameworks specifically designed for healthcare largely emphasize high-level ethical principles or focus on technical considerations related to algorithm development, implementation, and maintenance.

    “However, these frameworks largely emphasize high-level ethical principles30–32 or focus on technical considerations related to algorithm development, implementation, and maintenance33,34.”

    From Establishing organizational AI governance in healthcare: a case study in Canada (2025) · cites this paper as [31]

    The Algorithm Development and Postdeployment Considerations sections discuss technical choices, deployment, and ongoing model audits, supporting this paper as an example of work addressing those stages. But the Introduction and Problem Selection section also address social justice, research funding, and whose health needs receive attention; describing the paper as focused on technical considerations alone omits that scope. The paper does not establish what healthcare AI governance frameworks largely emphasize as a group. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. [4]Ghassemi M, Naumann T, Schulam P, Beam AL, Chen IY, Ranganath R. 2020. A review of challenges and opportunities in machine learning for health. AMIA Summits Transl. Sci. Proc 2020: 191– 200 32477638 PMC7233077
  2. [7]Zhang H, Lu AX, Abdalla M, McDermott M, Ghassemi M. 2020. Hurtful words: quantifying biases in clinical contextual word embeddings. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 110– 20. New York: Assoc. Comput. Mach.
  3. [14]Chen I, Johansson FD, Sontag D. 2018. Why is my classifier discriminatory? In Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NIPS 2018), ed. Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, pp. 3539– 50. https://proceedings.neurips.cc/paper/2018/file/1f1baa5b8edac74eb4eaa329f14a0361-Paper.pdf
  4. [16]Ustun B, Liu Y, Parkes D. 2019. Fairness without harm: decoupled classifiers with preference guarantees. Proc. Mach. Learn. Res 97: 6373– 82
  5. [18]Veatch RM, Guidry-Grimes LK. 2019. The Basics of Bioethics. New York: Routledge. 4th ed.
  6. [21]Powers M, Faden R. 2006. Social Justice: The Moral Foundations of Public Health and Health Policy. New York: Oxford Univ. Press 18259708
  7. [32]Pierson L, Millum J. 2019. Grant reviews and health research priority setting: Do research funders uphold widely endorsed ethical principles? Paper presented at Global Health Bioeth. Conf., Oxford, 1–2 July
  8. [39]Rouse C 2009. Uncertain Suffering: Racial Health Care Disparities and Sickle Cell Disease. Berkeley: Univ. Calif. Press
  9. [67]Haneuse SJA, Shortreed SM. 2017. On the use of electronic health records. In Methods in Comparative Effectiveness Research, ed. Gatsonis C, Morton SC, pp. 469– 502. New York: Chapman & Hall/CRC
  10. [72]Dredze M 2012. How social media will change public health. IEEE Intell. Syst 27: 81– 84
  11. [73]Abebe R, Hill S, Vaughan JW, Small PM, Schwartz HA. 2019. Using search queries to understand health information needs in Africa. In Proceedings of the Thirteenth International AAAI Conference on Web and Social Media, pp. 3– 14. Palo Alto, CA: AAAI
  12. [74]Giorgi S, Preoţiuc-Pietro D, Buffone A, Rieman D, Ungar L, Schwartz HA. 2018. The remarkable benefit of user-level aggregation for lexical-based population-level predictions. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1167– 72. Stroudsburg, PA: Assoc. Comput. Linguist.
  13. [76]Jamison DT, Feacham RG, Makgoba MW, Bos ER, Baingana FK, 2006. Disease and Mortality in Sub-Saharan Africa. Washington, DC: World Bank. 2nd ed. 21290641
  14. [86]Geruso M, Layton T. 2015. Upcoding: evidence from medicare on squishy risk adjustment. J. Pol. Econ 128(3). 10.1086/704756 PMC7384673 32719571 · DOI
  15. [87]Natarajan N, Dhillon IS, Ravikumar PK, Tewari A. 2013. Learning with noisy labels. In Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NIPS 2013), ed. Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, pp. 1196– 204. https://papers.nips.cc/paper/2013/file/3871bd64012152bfb53fdf04b401193f-Paper.pdf
  16. [95]Lohaus M, Perrot M, von Luxburg U. 2020. Too relaxed to be fair. Proc. Mach. Learn. Res 119: 6360– 69
  17. [96]Sagawa S, Koh PW, Hashimoto TB, Liang P. 2020. Distributionally robust neural network. Paper presented at the Eighth International Conference on Learning Representations (ICLR 2020), Apr. 26–May 1. https://openreview.net/pdf?id=ryxGuJrFvS
  18. [101]Van der Laan MJ, Rose S. 2011. Targeted Learning: Causal Inference for Observational and Experimental Data. New York: Springer-Verlag
  19. [102]Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, 2018. Double/debiased machine learning for treatment and structural parameters. Econom. J 21(1): C1– 68
  20. [104]Franks A, D’Amour A, Feller A. 2019. Flexible sensitivity analysis for observational studies without observable implications. J. Am. Stat. Assoc 115(532): 1730– 76
  21. [109]Harrell FE Jr. 2015. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. New York: Springer. 2nd ed.
  22. [110]James G, Witten D, Hastie T, Tibshirani R. 2013. An Introduction to Statistical Learning: With Applications in R. New York: Springer-Verlag
  23. [111]Koh PW, Nguyen T, Tang YS, Mussmann S, Pierson E, 2020. Concept bottleneck models. Proc. Mach. Learn. Res 119: 5338– 48
  24. [112]Sagawa S, Raghunathan A, Koh PW, Liang P. 2020. An investigation of why overparameterization exacerbates spurious correlations. Proc. Mach. Learn. Res 119: 8346– 56
  25. [113]Flach PA. 2003. The geometry of ROC space: understanding machine learning metrics through ROC isometrics. In Proceedings of the 20th International Conference on Machine Learning, ed. Fawcett T, Mishra N, pp. 194– 201. Palo Alto, CA: AAAI
  26. [114]Vyas DA, Eisenstein LG, Jones DS. 2020. Hidden in plain sight—reconsidering the use of race correction in clinical algorithms. New Engl. J. Med 383: 874– 82 32853499
  27. [115]Dwork C, Hardt M, Pitassi T, Reingold O, Zemel R. 2012. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214– 26. New York: Assoc. Comput. Mach.
  28. [116]Dwork C, Ilvento C. 2018. Fairness under composition. arXiv:1806.06122 [cs.LG] · arXiv
  29. [117]Chouldechova A, Roth A. 2020. A snapshot of the frontiers of fairness in machine learning. Commun. ACM 63: 82– 89
  30. [118]Calders T, Karim A, Kamiran F, Ali W, Zhang X. 2013. Controlling attribute effect in linear regression. In 2013 IEEE 13th International Conference on Data Mining, pp. 71– 80. Los Alamitos, CA: IEEE Comput. Soc.
  31. [119]Zafar MB, Valera I, Rogriguez MG, Gummadi KP. 2017. Fairness constraints: mechanisms for fair classification. Proc. Mach. Learn. Res 54: 962– 70
  32. [126]Noseworthy PA, Attia ZI, Brewer LC, Hayes SN, Yao X, 2020. Assessing and mitigating bias in medical artificial intelligence: the effects of race and ethnicity on a deep learning model for ECG analysis. Circ. Arrhythm. Electrophysiol 13: e007988 32064914 PMC7158877
  33. [127]Inst. Med. 2002. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. Washington, DC: Natl. Acad. Press
  34. [132]Nestor B, McDermott M, Boag W, Berner G, Naumann T, 2019. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. Proc. Mach. Learn. Res 106: 381– 405
  35. [133]Bissoto A, Fornaciali M, Valle E, Avila S. 2019. (De)constructing bias on skin lesion datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. https://openaccess.thecvf.com/content_CVPRW_2019/html/ISIC/Bissoto_DeConstructing_Bias_on_Skin_Lesion_Datasets_CVPRW_2019_paper.html
  36. [138]Saleh S, Boag W, Erdman L, Naumann T. 2020. Clinical collabsheets: 53 questions to guide a clinical collaboration. Proc. Mach. Learn. Res 126: 783– 812
  37. [139]Madaio MA, Stark L, Wortman Vaughan J, Wallach H. 2020. Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Pap. 318. New York: Assoc. Comput. Mach.
  38. [141]Gebru T, Morgenstern J, Vecchione B, Vaughan JW, Wallach H, 2018. Datasheets for datasets. arXiv:1803.09010 [cs.DB] · arXiv
  39. [149]Lyndon A, McNulty J, VanderWal B, Gabel K, Huwe V, Main E. 2015. Cumulative quantitative assessment of blood loss. In CMQCC Obstet. Hemorrhage Toolkit Vers. 2, pp. 80– 85. Stanford, CA: Calif. Matern. Qual. Care Collab. https://www.cmqcc.org/content/cumulative-quantitative-assessment-blood-loss

Cited by

Sorted by citations

Authors
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Microsoft (United States) (US)
  • Stanford University (US)
  • Vector Institute (CA)
  • New York University (US)
Venue
Annual Review of Biomedical Data Science
Published
2021-07-20
Volume
4
Issue
1
Pages
123-144
ISSN
2574-3414 (ISSN-L)
OpenAlex
W3087893815
Cited by
645
References
92
Citations by year
2026: 169 · 2025: 149 · 2024: 128 · 2023: 91 · 2022: 74 · 2021: 28 · 2020: 6
FWCI
14.21
Type
journal article
Language
en
Open access
Locations
  • Annual Review of Biomedical Data Science (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
National Institute on Minority Health and Health Disparities (DP2MD012722)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
health care, health disparities, social justice, algorithmic fairness, problem selection, algorithmic bias
MeSH terms
Machine Learning, Delivery of Health Care, Health Facilities, Morals, Social Justice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review