CoChat
Sign Up
Cited by 63Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models

Open access
Wang, H Echo, Landers, Matthew, Adams, Roy, Subbaswamy, Adarsh, Kharrazi, Hadi, Gaskin, Darrell J, Saria, Suchi·2022·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocac065PMC PMC9277650
Open this paper in CoChat — free account

Abstract

OBJECTIVE: Health care providers increasingly rely upon predictive algorithms when making important treatment decisions, however, evidence indicates that these tools can lead to inequitable outcomes across racial and socio-economic groups. In this study, we introduce a bias evaluation checklist that allows model developers and health care providers a means to systematically appraise a model's potential to introduce bias. MATERIALS AND METHODS: Our methods include developing a bias evaluation checklist, a scoping literature review to identify 30-day hospital readmission prediction models, and assessing the selected models using the checklist. RESULTS: We selected 4 models for evaluation: LACE, HOSPITAL, Johns Hopkins ACG, and HATRIX. Our assessment identified critical ways in which these algorithms can perpetuate health care inequalities. We found that LACE and HOSPITAL have the greatest potential for introducing bias, Johns Hopkins ACG has the most areas of uncertainty, and HATRIX has the fewest causes for concern. DISCUSSION: Our approach gives model developers and health care providers a practical and systematic method for evaluating bias in predictive models. Traditional bias identification methods do not elucidate sources of bias and are thus insufficient for mitigation efforts. With our checklist, bias can be addressed and eliminated before a model is fully developed or deployed. CONCLUSION: The potential for algorithms to perpetuate biased outcomes is not isolated to readmission prediction models; rather, we believe our results have implications for predictive models across health care. We offer a systematic method for evaluating potential bias with sufficient flexibility to be utilized across models and applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocac065

Cites

92 of 92 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Bias audits for models tend to be piecemeal, typically performed only during model development and validation.

    “Bias audits for models tend to be piecemeal—typically performed only during model development and validation; health equity considerations should commence during data collection and curation through post-deployment monitoring. 12”

    From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [12]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 17

  1. [12]Verma S, Rubin J. Fairness definitions explained. In: Proceedings of the International Workshop on Software Fairness. Gothenburg, Sweden, May 29, 2018: 1– 7.
  2. [13]Chouldechova A, Roth A. A Snapshot of the frontiers of fairness in machine learning. Communications of the ACM, 2020, 63 (5): 82– 89
  3. [15]Zafar M, Valera I, Gomez Rodriguez M, et al Fairness constraints: mechanisms for fair classification. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS). Ft. Lauderdale, FL, USA, 2017.
  4. [16]Komiyama J, Takeda A, Honda J, et al Nonconvex optimization for regression with fairness constraints. In: Proceedings of the 35 th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018.
  5. [18]Zhang B, Lemoine B, Mitchell M. Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. New Orleans LA USA, December 27, 2018: 335– 340.
  6. [19]Kamishima T, Akaho S, Sakuma J. Fairness-aware learning through regularization approach In: 2011 IEEE 11th International Conference on Data Mining Workshops. Vancouver, Canada, 2011: 643– 650.
  7. [21]Agarwal A, Beygelzimer A, Dudík M, et al A reductions approach to fair classification. In Proceedings of FAT ML, Halifax, Nova Scotia, Canada, 2017.
  8. [23]Hutchinson B, Mitchell M. 50 years of test (un)fairness. In: Proceedings of the Conference on Fairness, Accountability, and Transparency. Atlanta, GA, USA, January 29, 2019: 49– 58.
  9. [24]Glymour B, Herington J. Measuring the biases that matter. In: Proceedings of the Conference on Fairness, Accountability, and Transparency. Atlanta, GA, USA, January 29, 2019: 269– 278.
  10. [40]Smedley BD, Stith AY. Care, Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Unequal Treatment. Vol. 94. Washington: National Academies Press; 2002: 666– 668.
  11. [55]Lemke K. A predictive model to identify patients at risk of unplanned 30-day acute care hospital readmission. In: 2013 IEEE International Conference on Healthcare Informatics. Washington DC, USA, 2013: 551– 556.
  12. [74]Datta A, Tschantz MC, Datta A. Automated experiments on ad privacy settings. Proc Priv Enhancing Technol 2015; 2015 (1): 92– 112.
  13. [102]Hendrycks D, Mu N, Cubuk ED, et al AugMix: A simple data processing method to improve robustness and uncertainty. In: The International Conference on Learning Representations (ICLR), Virtual 2020.
  14. [107]Tonekaboni S, Joshi S, McCradden MD, et al What clinicians want: contextualizing explainable machine learning for clinical end use. In: Proceedings of Machine Learning Research, Vancouver, Canada, 2019: 1– 21.
  15. [108]Henry KE, Kornfield R, Sridharan A, et al Human-machine teaming in clinical care: clinicians’ experiences with a deployed machine learning system for sepsis. NPJ Digital Medicine. Accepted 2022. PMC9304371 35864312
  16. [110]Corbett-Davies S, Pierson E, Feller A, et al Algorithmic decision making and the cost of fairness. In: Proceedings of the 23rd ACM SIGKDD International Conference on knowledge discovery and data mining. Halifax NS Canada, August 4, 2017: 797– 806.
  17. [114]Ekstrand MD, Joshaghani R, Mehrpouyan H. Privacy for all: ensuring fair and equitable privacy protections. In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency. Vol. 81. PMLR; 2018: 35– 47.

Cited by

Sorted by citations

Related

Authors
Wang, H. E., Landers, M., Adams, R., Subbaswamy, A., Kharrazi, H., Gaskin, D. J., & Saria, S.
Affiliations
  • Johns Hopkins University (US)
  • University of Virginia (US)
Corresponding author
Saria, Suchi
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2022-05-17
Volume
29
Issue
8
Pages
1323-1333
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocac065
PMC
PMC9277650
OpenAlex
W4226438165
Cited by
63
References
92
Citations by year
2026: 11 · 2025: 7 · 2024: 18 · 2023: 20 · 2022: 7
FWCI
15.08
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
algorithmic bias, prediction model, health care disparities, lace, hospitals, algorithmic fairness
MeSH terms
Hospitals, Humans, Patient Readmission, Bias, Healthcare Disparities, Checklist

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review