CoChat

Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare

Open access
Pfohl, Stephen, Xu, Yizhe, Foryciarz, Agata, Ignatiadis, Nikolaos, Genkins, Julian, Shah, Nigam · 2022 · 2022 ACM Conference on Fairness Accountability and Transparency

The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.

Abstract

A growing body of work uses the paradigm of algorithmic fairness to frame the development of techniques to anticipate and proactively mitigate the introduction or exacerbation of health inequities that may follow from the use of model-guided decision-making. We evaluate the interplay between measures of model performance, fairness, and the expected utility of decision-making to offer practical recommendations for the operationalization of algorithmic fairness principles for the development and evaluation of predictive models in healthcare. We conduct an empirical case-study via development of models to estimate the ten-year risk of atherosclerotic cardiovascular disease to inform statin initiation in accordance with clinical practice guidelines. We demonstrate that approaches that incorporate fairness considerations into the model training objective typically do not improve model performance or confer greater net benefit for any of the studied patient populations compared to the use of standard learning paradigms followed by threshold selection concordant with patient preferences, evidence of intervention effectiveness, and model calibration. These results hold when the measured outcomes are not subject to differential measurement error across patient populations and threshold selection is unconstrained, regardless of whether differences in model performance metrics, such as in true and false positive error rates, are present. In closing, we argue for focusing model development efforts on developing calibrated models that predict outcomes well for all patient populations while emphasizing that such efforts are complementary to transparent reporting, participatory design, and reasoning about the impact of model-informed interventions in context.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3531146.3533166

Cites

82 of 82 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 82 references

Research works printed but not linked: 39

  1. [24] Ruha Benjamin. Assessing risk, automating racism. Science , 366(6464):421-422, 2019. -
  2. [113] Diana Cagliero, Natalie Deuitch, Nigam Shah, and Danton Char. Evaluating ethical concerns with machine learning to guide advance care planning. In 2021 Western Medical Research Conference , volume 69, pages 103-296. BMJ Publishing Group Limited, January 2021. -
  3. [114] Elad Eban, Mariano Schain, Alan Mackey, Ariel Gordon, Ryan Rifkin, and Gal Elidan. Scalable learning of non-decomposable objectives. In Artificial intelligence and statistics , pages 832-840. PMLR, 2017.
  4. [23] Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi. Ethical machine learning in healthcare. Annual Review of Biomedical Data Science , 4, 2020. -
  5. [5] Irene Y Chen, Peter Szolovits, and Marzyeh Ghassemi. Can ai help reduce disparities in general medical and mental health care? AMA journal of ethics , 21(2):167-179, 2019. -
  6. 10.5555/3295222.3295224 · doi:10.5555/3295222.3295224
  7. [6] R Yates Coley, Eric Johnson, Gregory E Simon, Maricela Cruz, and Susan M Shortreed. Racial/ethnic disparities in the performance of prediction models for death by suicide after mental health visits. JAMA psychiatry , 2021. -
  8. [27] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. arXiv preprint arXiv:1803.09010 , 2018. - · arXiv
  9. [37] David C Goff, Donald M Lloyd-Jones, Glen Bennett, Sean Coady, Ralph B D'agostino, Raymond Gibbons, Philip Greenland, Daniel T Lackland, Daniel Levy, Christopher J O'donnell, et al. 2013 acc/aha guideline on the assessment of cardiovascular risk: a report of the american college of cardiology/american heart association task force on practice guidelines. Journal of the American College of Cardiology , 63 (25 Part B):2935-2959, 2014. -
  10. [112] Úrsula Hébert-Johnson, Michael P. Kim, Omer Reingold, and Guy N. Rothblum. Calibration for the (Computationally-Identifiable) Masses. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1939-1948, Stockholmsmässan, Stockholm Sweden, 2017. PMLR. -
  11. [64] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research , 13(Mar):723-773, 2012. -
  12. [66] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning , pages 2029-2037. PMLR, 2018. -
  13. [58] Christina Ilvento. Metric learning for individual fairness. In 1st Symposium on Foundations of Responsible Computing (FORC 2020) . Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2020. -
  14. [88] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015. -
  15. [89] James M Robins and Dianne M Finkelstein. Correcting for noncompliance and dependent censoring in an aids clinical trial with inverse probability of censoring weighted (ipcw) log-rank tests. Biometrics , 56(3):779-788, 2000. -
  16. [90] Håvard Kvamme and Ørnulf Borgan. Continuous and discrete-time survival prediction with neural networks. arXiv preprint arXiv:1910.06724 , 2019. - · arXiv
  17. [19] Lydia T Liu, Max Simchowitz, and Moritz Hardt. The Implicit Fairness Criterion of Unconstrained Learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4051-4060, Long Beach, California, USA, 2019. PMLR. URL http: //proceedings.mlr.press/v97/liu19f.html . -
  18. [56] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning Adversarially Fair and Transferable Representations. Proceedings of the 35th International Conference on Machine Learning , 80:3384-3393, feb 2018. ISSN 1938-7228. URL http://proceedings.mlr.press/v80/madras18a. htmlhttp://arxiv.org/abs/1802.06309 . -
  19. [62] Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi objective perspective. In International Conference on Machine Learning , pages 6755-6764. PMLR, 2020. -
  20. [63] Robert Chen, Brendan Lucier, Yaron Singer, and Vasilis Syrgkanis. Robust optimization for non-convex objectives. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 4708-4717, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964. -
  21. [1] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science , 366(6464):447-453, 2019. -
  22. [78] Gulsen Ozen, Murat Sunbul, Pamir Atagunduz, Haner Direskeneli, Kursat Tigen, and Nevsun Inanc. The 2013 acc/aha 10-year atherosclerotic cardiovascular disease risk index is better than score and qrisk ii in rheumatoid arthritis: is it enough? Rheumatology , 55(3):513-522, 2016. -
  23. [8] Yoonyoung Park, Jianying Hu, Moninder Singh, Issa Sylla, Irene Dankwa-Mullan, Eileen Koski, and Amar K Das. Comparison of methods to reduce bias from clinical prediction models of postpartum depression. JAMA network open , 4(4):e213909-e213909, 2021. -
  24. [10] Stephen Pfohl, Ben Marafino, Adrien Coulet, Fatima Rodriguez, Latha Palaniappan, and Nigam H. Shah. Creating Fair Models of Atherosclerotic Cardiovascular Disease Risk. In AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society , 2019. -
  25. [49] Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q. Weinberger. On fairness and calibration. In Advances in Neural Information Processing Systems , pages 5680-5689, sep 2017. -
  26. 10.1093/jamia · doi:10.1093/jamia
  27. [60] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally Robust Neural Networks. In International Conference on Learning Representations , 2020. -
  28. [26] Mark P Sendak, Michael Gao, Nathan Brajer, and Suresh Balu. Presenting machine learning model information to clinical end users with model facts labels. NPJ digital medicine , 3(1):1-4, 2020. -
  29. [13] Laleh Seyyed-Kalantari, Haoran Zhang, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine , pages 1-7, 2021. -
  30. [57] Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2164-2173. PMLR, 2019. -
  31. [91] Handrean Soran, Jonathan D Schofield, and Paul N Durrington. Cholesterol, not just cardiovascular risk, is important in deciding who should receive statin treatment. European heart journal , 36(43): 2975-2983, 2015. -
  32. [92] Cholesterol Treatment Trialists et al. Efficacy and safety of cholesterol-lowering treatment: prospective meta-analysis of data from 90 056 participants in 14 randomised trials of statins. The Lancet , 366 (9493):1267-1278, 2005. -
  33. Harini Suresh and John V. Guttag. 2019. A Framework for Understanding Unintended Consequences of Machine Learning. www.aaai.orghttp://arxiv.org/abs/1901.10002
  34. [102] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. -
  35. [104] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alch'eBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf . -
  36. [48] Robert C. Williamson and Aditya Krishna Menon. Fairness risk measures. In 36th International Conference on Machine Learning, ICML 2019 , volume 2019-June, pages 11763-11774. International Machine Learning Society (IMLS), jan 2019. ISBN 9781510886988. -
  37. [59] Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. Learning NonDiscriminatory Predictors. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory , volume 65 of Proceedings of Machine Learning Research , pages 1920-1953, Amsterdam, Netherlands, 2017. PMLR. -
  38. [106] Steve Yadlowsky, Sanjay Basu, and Lu Tian. A calibration metric for risk scores with survival data. In Machine Learning for Healthcare Conference , pages 424-450, 2019. -
  39. [52] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, Krishna P Gummadi, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness Constraints: Mechanisms for Fair Classification. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 962-970, Fort Lauderdale, FL, USA, 2017. PMLR. -

Cited by

Metadata

Authors
Pfohl, S., Xu, Y., Foryciarz, A., Ignatiadis, N., Genkins, J., & Shah, N.
Affiliations
Stanford University (US)
Venue
2022 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2022-06-20
Pages
1039-1052
OpenAlex
W4283162970
Cited by
30
References
69
Citations by year
2026: 7 · 2025: 8 · 2024: 6 · 2023: 8 · 2022: 1
FWCI
5.71
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Heart, Lung, and Blood Institute (R01 HL144555)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
algorithmic fairness, health care, health disparities, net benefit, model calibration, threshold selection, training objectives, statin initiation, prediction model, model evaluation metrics, measurement error

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.