CoChat

Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare

Open access
Pfohl, Stephen, Xu, Yizhe, Foryciarz, Agata, Ignatiadis, Nikolaos, Genkins, Julian, Shah, Nigam · 2022 · 2022 ACM Conference on Fairness Accountability and Transparency

The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.

Abstract

A growing body of work uses the paradigm of algorithmic fairness to frame the development of techniques to anticipate and proactively mitigate the introduction or exacerbation of health inequities that may follow from the use of model-guided decision-making. We evaluate the interplay between measures of model performance, fairness, and the expected utility of decision-making to offer practical recommendations for the operationalization of algorithmic fairness principles for the development and evaluation of predictive models in healthcare. We conduct an empirical case-study via development of models to estimate the ten-year risk of atherosclerotic cardiovascular disease to inform statin initiation in accordance with clinical practice guidelines. We demonstrate that approaches that incorporate fairness considerations into the model training objective typically do not improve model performance or confer greater net benefit for any of the studied patient populations compared to the use of standard learning paradigms followed by threshold selection concordant with patient preferences, evidence of intervention effectiveness, and model calibration. These results hold when the measured outcomes are not subject to differential measurement error across patient populations and threshold selection is unconstrained, regardless of whether differences in model performance metrics, such as in true and false positive error rates, are present. In closing, we argue for focusing model development efforts on developing calibrated models that predict outcomes well for all patient populations while emphasizing that such efforts are complementary to transparent reporting, participatory design, and reasoning about the impact of model-informed interventions in context.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3531146.3533166

Cites

82 of 82 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 31

  1. Ruha Benjamin. 2019. Assessing risk, automating racism. Science 366, 6464 (2019), 421–422.
  2. Diana Cagliero, Natalie Deuitch, Nigam Shah, and Danton Char. 2021. Evaluating ethical concerns with machine learning to guide advance care planning. In 2021 Western Medical Research Conference, Vol. 69. BMJ Publishing Group Limited, 103–296.
  3. Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi. 2020. Ethical Machine Learning in Healthcare. Annual Review of Biomedical Data Science 4 (2020).
  4. Irene Y Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019. Can AI help reduce disparities in general medical and mental health care?AMA journal of ethics 21, 2 (2019), 167–179.
  5. 10.5555/3295222.3295224 · doi:10.5555/3295222.3295224
  6. R Yates Coley, Eric Johnson, Gregory E Simon, Maricela Cruz, and Susan M Shortreed. 2021. Racial/ethnic disparities in the performance of prediction models for death by suicide after mental health visits. JAMA psychiatry (2021).
  7. Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna Wallach Hal Daumé III and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010(2018). · arXiv
  8. David C Goff, Donald M Lloyd-Jones, Glen Bennett, Sean Coady, Ralph B D’agostino, Raymond Gibbons, Philip Greenland, Daniel T Lackland, Daniel Levy, Christopher J O’donnell, 2014. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Journal of the American College of Cardiology 63, 25 Part B(2014), 2935–2959.
  9. Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel two-sample test. Journal of Machine Learning Research 13, Mar (2012), 723–773.
  10. Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. 2018. Does distributionally robust supervised learning give robust classifiers?. In International Conference on Machine Learning. PMLR, 2029–2037.
  11. Christina Ilvento. 2020. Metric Learning for Individual Fairness. In 1st Symposium on Foundations of Responsible Computing (FORC 2020). Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
  12. Guido W Imbens and Donald B Rubin. 2015. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.
  13. Håvard Kvamme and Ørnulf Borgan. 2019. Continuous and discrete-time survival prediction with neural networks. arXiv preprint arXiv:1910.06724(2019). · arXiv
  14. Lydia T Liu, Max Simchowitz, and Moritz Hardt. 2019. The Implicit Fairness Criterion of Unconstrained Learning. In Proceedings of the 36th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 4051–4060. http://proceedings.mlr.press/v97/liu19f.html
  15. David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2018. Learning Adversarially Fair and Transferable Representations. Proceedings of the 35th International Conference on Machine Learning 80 (feb 2018), 3384–3393. arxiv:1802.06309http://proceedings.mlr.press/v80/madras18a.htmlhttp://arxiv.org/abs/1802.06309
  16. Natalia Martinez, Martin Bertran, and Guillermo Sapiro. 2020. Minimax Pareto fairness: A multi objective perspective. In International Conference on Machine Learning. PMLR, 6755–6764.
  17. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447–453.
  18. Gulsen Ozen, Murat Sunbul, Pamir Atagunduz, Haner Direskeneli, Kursat Tigen, and Nevsun Inanc. 2016. The 2013 ACC/AHA 10-year atherosclerotic cardiovascular disease risk index is better than SCORE and QRisk II in rheumatoid arthritis: is it enough?Rheumatology 55, 3 (2016), 513–522.
  19. Yoonyoung Park, Jianying Hu, Moninder Singh, Issa Sylla, Irene Dankwa-Mullan, Eileen Koski, and Amar K Das. 2021. Comparison of Methods to Reduce Bias From Clinical Prediction Models of Postpartum Depression. JAMA network open 4, 4 (2021), e213909–e213909.
  20. Stephen Pfohl, Ben Marafino, Adrien Coulet, Fatima Rodriguez, Latha Palaniappan, and Nigam H. Shah. 2019. Creating Fair Models of Atherosclerotic Cardiovascular Disease Risk. In AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society. arxiv:1809.04663 · arXiv
  21. Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q. Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680–5689. arxiv:1709.02012 · arXiv
  22. 10.1093/jamia · doi:10.1093/jamia
  23. Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2020. Distributionally Robust Neural Networks. In International Conference on Learning Representations.
  24. Mark P Sendak, Michael Gao, Nathan Brajer, and Suresh Balu. 2020. Presenting machine learning model information to clinical end users with model facts labels. NPJ digital medicine 3, 1 (2020), 1–4.
  25. Laleh Seyyed-Kalantari, Haoran Zhang, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. 2021. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine (2021), 1–7.
  26. Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. 2019. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2164–2173.
  27. Handrean Soran, Jonathan D Schofield, and Paul N Durrington. 2015. Cholesterol, not just cardiovascular risk, is important in deciding who should receive statin treatment. European heart journal 36, 43 (2015), 2975–2983.
  28. Harini Suresh and John V. Guttag. 2019. A Framework for Understanding Unintended Consequences of Machine Learning. www.aaai.orghttp://arxiv.org/abs/1901.10002
  29. Robert C. Williamson and Aditya Krishna Menon. 2019. Fairness risk measures. In 36th International Conference on Machine Learning, ICML 2019, Vol. 2019-June. International Machine Learning Society (IMLS), 11763–11774. arxiv:1901.08665https://arxiv.org/abs/1901.08665v1
  30. Steve Yadlowsky, Sanjay Basu, and Lu Tian. 2019. A calibration metric for risk scores with survival data. In Machine Learning for Healthcare Conference. 424–450.
  31. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, Krishna P Gummadi, Manuel Gomez Rogriguez, and Krishna P Gummadi. 2017. Fairness Constraints: Mechanisms for Fair Classification. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 54), Aarti Singhand Jerry Zhu (Eds.). PMLR, Fort Lauderdale, FL, USA, 962–970.

Cited by

See all 30

Metadata

Authors
Pfohl, S., Xu, Y., Foryciarz, A., Ignatiadis, N., Genkins, J., & Shah, N.
Affiliations
Stanford University (US)
Venue
2022 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2022-06-20
Pages
1039-1052
OpenAlex
W4283162970
Cited by
30
References
69
Citations by year
2026: 7 · 2025: 8 · 2024: 6 · 2023: 8 · 2022: 1
FWCI
5.71
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Heart, Lung, and Blood Institute (R01 HL144555)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
algorithmic fairness, health care, health disparities, net benefit, model calibration, threshold selection, training objectives, statin initiation, prediction model, model evaluation metrics, measurement error

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.