CoChat
Sign Up
Cited by 750Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)

Algorithmic fairness in artificial intelligence for medicine and healthcare

Open access
Chen, Richard J., Wang, Judy J., Williamson, Drew F. K., Chen, Tiffany Y., Lipkova, Jana, Lu, Ming Y., Sahai, Sharifa, Mahmood, Faisal·2023·Nature Biomedical Engineering
DOI 10.1038/s41551-023-01056-8PMC PMC10632090
Open this paper in CoChat — free accountOpen PDF

Abstract

In healthcare, the development and deployment of insufficiently fair systems of artificial intelligence (AI) can undermine the delivery of equitable care. Assessments of AI models stratified across subpopulations have revealed inequalities in how patients are diagnosed, treated and billed. In this Perspective, we outline fairness in machine learning through the lens of healthcare, and discuss how algorithmic biases (in data acquisition, genetic variation and intra-observer labelling variability, in particular) arise in clinical workflows and the resulting healthcare disparities. We also review emerging technology for mitigating biases via disentanglement, federated learning and model explainability, and their role in the development of AI-based software as a medical device.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41551-023-01056-8

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • As models incorporate more modalities, their complexity and opacity increase, raising concerns about interpretability, bias, and fairness.

    “as models incorporate more modalities, their complexity and opacity increase, raising concerns about interpretability, bias, and fairness (R. J. Chen et al., 2023).”

    From Multimodal neuroimaging and AI integration in cognitive disorders: advances, challenges, and future directions for precision medicine (2026) · cites this paper as R. J. Chen et al., 2023

    In “Fitting algorithm design to comply with regulatory requirements for explainability” (page unavailable), the paper says interpretability varies across data modalities and model architectures and links explainability to fairness assessment. It does not say that incorporating more modalities increases model complexity or opacity, or that this increase causes the stated concerns. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • ML models have been revealed to diagnose, treat, and bill patients inconsistently across subpopulations.

    “Alongside population‐level evaluations, there has been burgeoning awareness about the ethical implications of ML models, which have been revealed to diagnose, treat, and bill patients inconsistently across subpopulations [31].”

    From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [31]

    The Introduction and ‘Lack of representation in biomedical datasets’ section report that AI models underdiagnose some underserved subgroups at disproportionate rates. The paper discusses possible effects on treatment decisions, but the passages found do not establish that models have been shown to treat patients inconsistently across subpopulations; they also do not document inconsistent billing by ML models. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Research has revealed numerous stereotypical associations and implicit biases in large image-text multimodal models, particularly regarding race, skin tone, and gender.

    “Research has revealed numerous stereotypical associations and implicit biases in large image-text multimodal models, particularly with regard to ‘measurable’ and therefore enactable categories of race, skin tone, and gender”

    From ‘We can see a savage’: a case study of the colonial gaze in generative AI algorithms (2026) · cites this paper as Chen et al. 2023

    The ‘Lack of representation in biomedical datasets’ section reports under-representation of darker-skinned patients and underdiagnosis of female, Black and Hispanic patients by medical-image algorithms (page unavailable). That supports the broader point that AI systems can exhibit demographic bias, but the passages do not establish stereotypical associations or implicit biases specifically in large image–text multimodal models. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Average subgroup performance is not sufficient: clinically relevant degradation may still emerge in specific age-race-sex strata.

    “Even so, average subgroup performance is not sufficient: clinically relevant degradation may still emerge in specific age-race-sex strata, and fairness in medical AI is shaped by data acquisition and workflow as well as by model architecture.”

    From AI-ECG for Echocardiography Triage in Structural Heart Disease: Evidence, Implementation, and Future Directions (2026) · cites this paper as [67]

    The ‘Healthcare disparities arising from dataset shift’ section says demographic and image-acquisition differences can cause disparate performance at the subgroup level; the ‘Image acquisition and measurement variation’ section also links acquisition protocols to model performance. These passages support the concern about subgroup degradation and factors beyond model architecture, but they do not specifically establish that average subgroup results conceal clinically relevant degradation in combined age–race–sex strata (pages unavailable in our copy). (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • R. J. Chen et al. (2023) describe bias reduction efforts during data preprocessing, during model training, and during postprocessing.

    “For example, R. J. Chen et al. (2023) describe bias reduction efforts during data preprocessing, during model training, and during postprocessing.”

    From Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future (2025) · cites this paper as R. J. Chen et al. (2023)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 305 · Cited by 750

Research works printed but not linked: 86

  1. [1]Buolamwini J & Gebru T Gender shades: intersectional accuracy disparities in commercial gender classification. In Conf. on Fairness, Accountability and Transparency 77– 91 (PMLR, 2018).
  2. [56]Hardt M, Price E & Srebro N Equality of opportunity in supervised learning. In Adv. Neural Information Processing Systems (2016).
  3. [64]Kleinberg J, Mullainathan S & Raghavan M Inherent trade-offs in the fair determination of risk scores In 8th Innovations in Theoretical Computer Science Conf. (ITCS 2017)
  4. [66]Joseph M, Kearns M, Morgenstern JH & Roth A Fairness in learning: classic and contextual bandits. In Adv. Neural Information Processing Systems (2016).
  5. [69]Calmon FP, Wei D, Vinzamuri B, Ramamurthy KN & Varshney KR Optimized pre-processing for discrimination prevention In Proc. 31st Int. Conf. Neural Information Processing Systems 3995– 4004 (2017).
  6. [71]Jiang H & Nachum O Identifying and correcting label bias in machine learning In Int. Conf. Artificial Intelligence and Statistics 702– 712 (PMLR, 2020).
  7. [74]Zafar MB, Valera I, Rogriguez MG & Gummadi KP Fairness constraints: mechanisms for fair classification. In Artificial Intelligence and Statistics 962– 970 (PMLR, 2017).
  8. [76]Goh G, Cotter A, Gupta M & Friedlander MP Satisfying real-world goals with dataset constraints. In Adv. Neural Information Processing Systems (2016).
  9. [77]Agarwal A A reductions approach to fair classification In Int. Conf. Machine Learning 60– 69 (PMLR, 2018).
  10. [79]Pleiss G, Raghavan M, Wu F, Kleinberg J & Weinberger KQ On fairness and calibration. In Adv. Neural Information Processing Systems (2017).
  11. [80]Chouldechova A, Benavides-Prado D, Fialko O & Vaithianathan R A case study of algorithm assisted decision making in child maltreatment hotline screening decisions In Conf. Fairness, Accountability and Transparency 134– 148 (PMLR, 2018).
  12. [81]Abernethy J, Awasthi P, Kleindessner M, Morgenstern J & Zhang J Active sampling for min-max fairness In Int. Conf. Machine Learning 53– 65, (PMLR, 2022).
  13. [82]Iosifidis V & Ntoutsi E Dealing with bias via data augmentation in supervised learning scenarios. In Proc. Int. Workshop on Bias in Information, Algorithms, and Systems (eds. Bates J) (2018).
  14. [84]Barocas S & Selbst AD Big data’s disparate impact. Calif. Law Rev 104, 671 (2016).
  15. [87]Alabi D, Immorlica N & Kalai A Unleashing linear optimizers for group-fair learning and optimization In Conf. Learning Theory 2043–2066 (PMLR, 2018).
  16. [88]Kearns M, Neel S, Roth A & Wu ZS Preventing fairness gerrymandering: auditing and learning for subgroup fairness In Int. Conf. Machine Learning 2564–2572 (PMLR, 2018).
  17. [93]Zemel R, Wu Y, Swersky K, Pitassi T & Dwork C Learning fair representations In Int. Conf. Machine Learning 325–333 (PMLR, 2013).
  18. [98]Chen I, Johansson FD & Sontag D Why is my classifier discriminatory? In Adv. Neural Information Processing Systems (2018).
  19. [100]Rolf E, Worledge T, Recht B & Jordan MI Representation matters: assessing the importance of subgroup allocations in training data In Int. Conf. Machine Learning 9040–9051 (2021).
  20. [101]Zhao H & Gordon G Inherent tradeoffs in learning fair representations. In Adv. Neural Information Processing Systems 32, 15675– 15685 (2019).
  21. [103]Pfohl SR Recommendations for Algorithmic Fairness Assessments of Predictive Models in Healthcare: Evidence from Large-scale Empirical Analyses PhD thesis, Stanford Univ. (2021).
  22. [106]Giguere S Fairness guarantees under demographic shift In Int. Conf. Learning Representations (2021).
  23. [109]Hashimoto T, Srivastava M, Namkoong H & Liang P Fairness without demographics in repeated loss minimization In Int. Conf. Machine Learning 1929– 1938 (PMLR, 2018).
  24. [117]Subbaswamy A, Schulam P & Saria S Preventing failures due to dataset shift: learning predictive models that transport In 22nd Int. Conf. Artificial Intelligence and Statistics 3118– 3127 (PMLR, 2019).
  25. [120]Singh H, Singh R, Mhasawade V & Chunara R Fair predictors under distribution shift In NeurIPS Workshop on Fair ML for Health (2019).
  26. [121]Bernhardt M, Jones C & Glocker B Investigating underdiagnosis of ai algorithms in the presence of multiple sources of dataset bias. Nat. Med 28, 1157– 1158 (2022). 35710993
  27. [122]Ghosh A & Shanbhag A FairCanary: rapid continuous explainable fairness In Proc. AAAI/ACM Conf. AI, Ethics, and Society (2022).
  28. [123]Sagawa S, Koh PW, Hashimoto TB & Liang P Distributionally robust neural networks In Int. Conf. Learning Representations (2020).
  29. [124]Yang Y, Zhang H, Katabi D & Ghassemi M Change is hard: a closer look at subpopulation shift In Int. Conf. Machine Learning (2023).
  30. [125]Zong Y, Yang Y & Hospedales T MEDFAIR: benchmarking fairness for medical imaging In Int. Conf. Learning Representations (2023).
  31. [168]Dwork C, Immorlica N, Kalai AT & Leiserson M Decoupled classifiers for fair and efficient machine learning In Conf. Fairness, Accountability and Transparency (PMLR, 2018).
  32. [169]Lipton Z, McAuley J & Chouldechova A Does mitigating ml’s impact disparity require treatment disparity? In Adv. Neural Information Processing Systems (2018).
  33. [171]Lohaus M, Kleindessner M, Kenthapadi K, Locatello F & Russell C Are two heads the same as one? Identifying disparate treatment in fair neural networks. In Adv. Neural Information Processing Systems (2022).
  34. [179]Ganin Y & Lempitsky V Unsupervised domain adaptation by backpropagation In Int. Conf. Machine Learning 1180– 1189 (PMLR, 2015).
  35. [183]Lu J Learning under concept drift: a review. IEEE Trans. Knowl. Data Eng 31, 2346– 2363 (2018).
  36. [186]Zhou H, Chen Y & Lipton ZC Evaluating model performance in medical datasets over time In Proc. Conf. Health, Inference, and Learning (2023).
  37. [187]Scholkopf B On causal and anticausal learning In Int. Conf. Machine Learning (2012).
  38. [188]Lipton Z, Wang Y-X & Smola A Detecting and correcting for label shift with black box predictors In Int. Conf. Machine Learning 3122– 3130 (PMLR, 2018).
  39. [232]Liang J, Hu D & Feng J Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation In Int. Conf. Machine Learning 6028– 6039 (PMLR, 2020).
  40. [235]Peterson D, Kanani P & Marathe VJ Private federated learning with domain adaptation In Federated Learning for Data Privacy and Confidentiality Workshop in NeurIPS (2019).
  41. [236]Peng X, Huang Z, Zhu Y & Saenko K Federated adversarial domain adaptation In Int. Conf. Learning Representations (2020).
  42. [238]Li T, Sanjabi M, Beirami A & Smith V Fair resource allocation in federated learning In Int. Conf. Learning Representations (2020).
  43. [239]Mohri M, Sivek G & Suresh AT Agnostic federated learning In Int. Conf. Machine Learning 4615– 4625 (PMLR, 2019).
  44. [242]Chen D, Gao D, Kuang W, Li Y & Ding B pFL-Bench: a comprehensive benchmark for personalized federated learning In 36th Conf. Neural Information Processing Systems Datasets and Benchmarks Track (2022).
  45. [243]Chai J & Wang X Self-supervised fair representation learning without demographics. In Adv. Neural Information Processing Systems (2022).
  46. [249]Lin Y, Han S, Mao H, Wang Y & Dally WJ Deep gradient compression: reducing the communication bandwidth for distributed training In Int. Conf. Learning Representations (2018).
  47. [250]McMahan B, Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics 1273– 1282 (PMLR, 2017).
  48. [251]Li T Federated optimization in heterogeneous networks. In Proc. Mach. Learn. Syst 2, 429– 450 (2020).
  49. [254]Luo Z, Wang Y, Wang Z, Sun Z & Tan T Disentangled federated learning for tackling attributes skew via invariant aggregation and diversity transferring In Int. Conf. Machine Learning 14527– 14541 (PMLR, 2022).
  50. [260]Shi C, Blei D & Veitch V Adapting neural networks for the estimation of treatment effects. In Adv. Neural Information Processing Systems (2019).
  51. [261]Yoon J, Jordon J & Van Der Schaar M GANITE: estimation of individualized treatment effects using generative adversarial nets In Int. Conf. Learning Representations (2018).
  52. [267]Wadsworth C, Vera F & Piech C Achieving fairness through adversarial learning: an application to recidivism prediction In FAT/ML Workshop (2018).
  53. [269]Madras D, Creager E, Pitassi T & Zemel R Learning adversarially fair and transferable representations In Int. Conf. Machine Learning 3384– 3393 (PMLR, 2018).
  54. [270]Madras D, Creager E, Pitassi T & Zemel R Learning adversarially fair and transferable representations In Proc. 35th Int. Conf. Machine Learning (eds. Dy J & Krause A) 3384– 3393 (PMLR, 2018).
  55. [271]Chen X, Fain B, Lyu L & Munagala K Proportionally fair clustering In Proc. 36th Int. Conf. Machine Learning (eds. Chaudhuri K & Salakhutdinov R) 1032– 1041 (PMLR, 2019).
  56. [275]Chen Y, Raab R, Wang J & Liu Y Fairness transferability subject to bounded distribution shift. In Adv. Neural Information Processing Systems (2022).
  57. [276]An B, Che Z, Ding M & Huang F Transferring fairness under distribution shifts via fair consistency regularization. In Adv. Neural Information Processing Systems (2022).
  58. [285]Chen RT, Li X, Grosse R & Duvenaud D Isolating sources of disentanglement in variational autoencoders. In Adv. Neural Information Processing Systems (2018).
  59. [286]Kim H & Mnih A Disentangling by factorising In Int. Conf. Machine Learning 2649– 2658 (PMLR, 2018).
  60. [287]Higgins I beta-VAE: learning basic visual concepts with a constrained variational framework In Int Conf. Learning Representations (2017).
  61. [291]Xu Y, He H, Shen T & Jaakkola TS Controlling directions orthogonal to a classifier In Int. Conf. Learning Representations (2022).
  62. [292]Cisse M & Koyejo S Fairness and representation learning. In NeurIPS Invited Talk 2019; https://cs.stanford.edu/~sanmi/documents/Representation_Learning_Fairness_NeurIPS19_Tutorial.pdf (2019).
  63. [294]Locatello F On the fairness of disentangled representations. In Adv. Neural Information Processing Systems (2019).
  64. [295]Lee J, Kim E, Lee J, Lee J & Choo J Learning debiased representation via disentangled feature augmentation. In Adv. Neural Information Processing Systems 34, 25123– 25133 (2021).
  65. [303]Chen T, Kornblith S, Norouzi M & Hinton G A simple framework for contrastive learning of visual representations In Int. Conf. Machine Learning 1597– 1607 (PMLR, 2020).
  66. [308]Simonyan K, Vedaldi A & Zisserman A Deep inside convolutional networks: visualising image classification models and saliency maps In Workshop at Int. Conf. Learning Representations (2014).
  67. [312]Patro BN, Lunayach M, Patel S & Namboodiri VP U-CAM: visual explanation using uncertainty based class activation maps In Proc. IEEE/CVF Int. Conf. Computer Vision 7444– 7453 (2019).
  68. [318]Kindermans P-J in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning 267– 280 (Springer, 2019).
  69. [320]Adebayo J Sanity checks for saliency maps. In Adv. Neural Information Processing Systems (2018).
  70. [324]Adebayo J, Muelly M, Liccardi I & Kim B Debugging tests for model explanations. In Adv. Neural Information Processing Syst 33, 700– 712 (2020).
  71. [326]Sundararajan M, Taly A & Yan Q Axiomatic attribution for deep networks In Int. Conf. Machine Learning 3319– 3328 (PMLR, 2017).
  72. [334]Lundberg SM Explaining quantitative measures of fairness In Fair & Responsible AI Workshop @ CHI2020; https://scottlundberg.com/files/fairness_explanations.pdf (2020).
  73. [341]Zhang H, Singh H, Ghassemi M & Joshi S “Why did the model fail?”: attributing model performance changes to distribution shifts In Int. Conf. Machine Learning (2023).
  74. [342]Ghorbani A & Zou J Data Shapley: equitable valuation of data for machine learning In Int. Conf. Machine Learn. 97, 2242– 2251 (2019).
  75. [348]Yufei C Bayes-MIL: a new probabilistic perspective on attention-based multiple instance learning for whole slide images In Int. Conf. Learning Representations (2023).
  76. [352]Javed SA, Juyal D, Padigela H, Taylor-Weiner A & Yu L Additive MIL: intrinsically interpretable multiple instance learning for pathology. In Adv. Neural Information Processing Systems (2022).
  77. [368]Boag W, Suresh H, Celi LA, Szolovits P & Ghassemi M Racial disparities and mistrust in end-of-life care In Machine Learning for Healthcare Conf. 587– 602 (PMLR, 2018).
  78. [377]Madras D, Pitassi T & Zemel R Predict responsibly: improving fairness and accuracy by learning to defer. In Adv. Neural Information Processing Systems (2018).
  79. [378]Louizos C, Swersky K, Li Y, Welling M & Zemel R The variational fair autoencoder In Int. Conf. Learning Representations (2016).
  80. [380]Smith JW, Everhart J, Dickson W, Knowler W & Johannes R Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proc. Symp. Computer Applications in Medical Care 261– 265 (1988).
  81. [385]Madras D, Creager E, Pitassi T & Zemel R Fairness through causal awareness: learning causal latent-variable models for biased data In Proc. Conf. Fairness, Accountability, and Transparency 30, 349– 358 (Association for Computing Machinery, 2019).
  82. [387]Kleindessner M, Samadi S, Awasthi P & Morgenstern J Guarantees for spectral clustering with fairness constraints In Int. Conf. Machine Learning 3458– 3467 (2019).
  83. [389]Garg S, Balakrishnan S & Lipton ZC Domain adaptation under open set label shift. In Adv. Neural Information Processing Systems (2022).
  84. [390]Pham TH, Zhang X & Zhang P Fairness and accuracy under domain generalization In Int. Conf. Learning Representations (2023). PMC13532314 42683346
  85. [391]Barocas S, Hardt M & Narayanan A Fairness in machine learning. NIPS Tutor 1, 2017 (2017).
  86. [392]Liu LT, Simchowitz M & Hardt M The implicit fairness criterion of unconstrained learning In Int. Conf. Machine Learning 4051– 4060 (PMLR, 2017).

Authors
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F.
Affiliations
  • Broad Institute (US)
  • Boston University (US)
Corresponding author
Mahmood, Faisal
ORCID
Venue
Nature Biomedical Engineering
Publisher
Springer Science and Business Media LLC
Published
2023-06-28
Volume
7
Issue
6
Pages
719-742
ISSN
2157-846X (ISSN-L)
DOI
10.1038/s41551-023-01056-8
PMC
PMC10632090
OpenAlex
W4382394524
Cited by
750
References
305
Citations by year
2026: 319 · 2025: 278 · 2024: 138 · 2023: 15
FWCI
29.33
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Biomedical Engineering (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion, other-oa)
Funders
U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences (R35GM138216)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
algorithmic fairness, algorithmic bias, health care disparities, health care equity, machine learning in medicine, genetic variation, disentanglement, federated learning, model explainability, software as a medical device, clinical workflow
MeSH terms
Machine Learning, Artificial Intelligence, Delivery of Health Care, Humans, Medicine, Software

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review