Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 11 of the 42 sources its claims cite were checked for support.

Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities

Open access
Paulus, Jessica K., Kent, David M. · 2020 · npj Digital Medicine

Abstract

The machine learning community has become alert to the ways that predictive algorithms can inadvertently introduce unfairness in decision-making. Herein, we discuss how concepts of algorithmic fairness might apply in healthcare, where predictive algorithms are being increasingly used to support decision-making. Central to our discussion is the distinction between algorithmic fairness and algorithmic bias. Fairness concerns apply specifically when algorithms are used to support polar decisions (i.e., where one pole of prediction leads to decisions that are generally more desired than the other), such as when predictions are used to allocate scarce health care resources to a group of patients that could all benefit. We review different fairness criteria and demonstrate their mutual incompatibility. Even when models are used to balance benefits-harms to make optimal decisions for individuals (i.e., for non-polar decisions)–and fairness concerns are not germane–model, data or sampling issues can lead to biased predictions that support decisions that are differentially harmful/beneficial across groups. We review these potential sources of bias, and also discuss ways to diagnose and remedy algorithmic bias. We note that remedies for algorithmic fairness may be more problematic, since we lack agreed upon definitions of fairness. Finally, we propose a provisional framework for the evaluation of clinical prediction models offered for further elaboration and refinement. Given the proliferation of prediction models used to guide clinical decisions, developing consensus for how these concerns can be addressed should be prioritized.

Overview

CoChat Summary

TL;DR

This perspective distinguishes algorithmic fairness from algorithmic bias in clinical prediction, arguing fairness concerns are germane mainly for “polar” decisions where one prediction pole is generally desired or undesired, while bias can affect both polar and non-polar decisions. It reviews mutually incompatible fairness criteria, sources of biased data/modeling, and proposes a provisional framework for evaluating clinical prediction models.

Key findings

  • Fairness concerns apply specifically to “polar” predictions—where one pole leads to decisions generally more desired than the other, such as allocating scarce health care resources—whereas non-polar decisions aimed at balancing benefits and harms for an individual do not raise the same fairness concerns, though algorithmic bias can still occur.
  • Common fairness criteria are mutually incompatible: equalized error rates and calibration/test fairness cannot both be satisfied when outcome rates differ across groups, except under perfect prediction. In the paper’s numerical illustration with a low-prevalence group at 20% and high-prevalence group at 30%, a test with 80% sensitivity and specificity yields positive predictive values of 50% vs ~63% and false omission rates of ~6% vs ~10%; holding risk-strata outcome rates constant would nearly double the false positive rate from 20% to ~39%.
  • Algorithmic bias can arise from label bias, feature bias, differential missingness, subgroup validity problems, and sampling bias; examples include surveillance-sensitive cancer overdiagnosis in affluent communities, ICU mortality predictions shaped by practice patterns, the racially homogeneous Framingham study, and polygenic risk scores derived largely on European populations performing poorly on non-European populations.
  • The proposed framework separates ensuring unbiased models—diagnosing subgroup performance and using group main effects, interactions, or stratified models—from addressing fairness in polar contexts via input-focused approaches (class-blind allocation, avoiding race proxies, transparency) or output-focused approaches (fairness constraints, different decision thresholds), with no agreed mathematical solution to define fairness.

Methods

  • Perspective/review article that synthesizes methodological research from computer science and relevant epidemiological principles; no primary empirical data collection is reported.
  • Uses conceptual analysis, mathematical demonstration of fairness-criteria incompatibility, a numerical illustration with N = 100 confusion matrices, and examples such as COMPAS data from over 7000 arrests and the Framingham Heart Study.
  • Proposes a provisional framework for evaluating algorithmic bias and fairness in clinical prediction and decision-making, distinguishing polar from non-polar predictions.

Limitations

  • Stated: the framework is simplified and provisional, intended as a starting point; some algorithms are applied in different decisional contexts with different ethical concerns (e.g., race-aware estimated GFR equations used for transplant prioritization and medication dosing); polarity may be non-obvious; some objections to using race in prediction models have little to do with unfairness as described; and identifying/mitigating algorithmic disparities involves political and legal complexities requiring integration of statistical and legal thinking.
  • Evident from the text: the framework is not empirically validated in the paper; fairness definitions remain contested and lack agreed mathematical solutions; causal definitions of fairness are described as fundamentally unidentifiable in observational data except with unverifiable assumptions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-020-0304-9

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Mutual incompatibility of fairness criteria. For two groups with different outcome rates, a predictive test can have consistent error rates or consistent calibration but not both. We present outcomes using coarsened prediction scores, thresholded to divide the population (N = 100) into low and high risk strata. Confusion matrices for a low prevalence group with a 20% outcome rate (Matrix A, red) and a high prevalence group with a 30% outcome rate (Matrices B and C, green) are shown. For the low prevalence group, a predictive test with an 80% sensitivity and specificity identifies a high risk (test+) strata with an outcome rate of 50% (i.e., the positive predictive value) and a low risk (test−) strata with an outcome rate of ~6% (i.e., the false omission rate). However, as shown in Matrix B, the same sensitivity and specificity in the higher prevalence group gives rise to outcome rates of ~63% and ~10% in the high and low risk-strata, respectively. This violates the criterion of test fairness, since the meaning of a positive or negative test differs across the two groups. Holding risk-strata specific outcome rates constant would require a higher sensitivity and lower specificity (Matrix C). This violates the fairness criteria of equalized error rates. For example, the Type I error rate (i.e., the false positive rate) would almost double from 20% in the low prevalence population to ~39% in the higher prevalence population. The diagnostic odds ratio was fixed at ~16 across this example, whole numbers are used to ease interpretation.
Fig. 1 Mutual incompatibility of fairness criteria. For two groups with different outcome rates, a predictive test can have consistent error rates or consistent calibration but not both. We present outcomes using coarsened prediction scores, thresholded to divide the population (N = 100) into low and high risk strata. Confusion matrices for a low prevalence group with a 20% outcome rate (Matrix A, red) and a high prevalence group with a 30% outcome rate (Matrices B and C, green) are shown. For the low prevalence group, a predictive test with an 80% sensitivity and specificity identifies a high risk (test+) strata with an outcome rate of 50% (i.e., the positive predictive value) and a low risk (test−) strata with an outcome rate of ~6% (i.e., the false omission rate). However, as shown in Matrix B, the same sensitivity and specificity in the higher prevalence group gives rise to outcome rates of ~63% and ~10% in the high and low risk-strata, respectively. This violates the criterion of test fairness, since the meaning of a positive or negative test differs across the two groups. Holding risk-strata specific outcome rates constant would require a higher sensitivity and lower specificity (Matrix C). This violates the fairness criteria of equalized error rates. For example, the Type I error rate (i.e., the false positive rate) would almost double from 20% in the low prevalence population to ~39% in the higher prevalence population. The diagnostic odds ratio was fixed at ~16 across this example, whole numbers are used to ease interpretation.CC BY · PubMed Central
Fig. 2 Non-polar and polar prediction-supported health care decisions. Understanding the specific decisional context of a prediction-supported decision in healthcare is necessary to anticipate potential unfairness. In the medical context—particularly in the shared decision-making context—patients and providers often share a common goal of accurate prognostication in order to help balance benefits and harms for care individualization. Predictions supporting decisions in this context may be described as “non-polar” (a). On the other hand, when one “pole” of the prediction is associated with a clear benefit or a clear harm, predictions may be described as “polar” in nature. In cases of polar predictions, the decision maker’s interest in efficient decision making (i.e. based on accurate prognostication using all available information) is not aligned with the subject’s interest to have either a lower (e.g. screening for abuse risk) or higher (e.g. microallocation of organs) prediction. “Positively” polar predictions correspond to those where patients may have an interest to be ranked high to receive a service that may be available only to some of those who can potentially benefit (b). This is in distinction to “negatively” polar predictions, in which prediction is used for the targeting of an intervention perceived as punitive or coercive (e.g. such as involuntary commitment, screening for child abuse or mandatory quarantining those at high infectious risk) (c). Issues of fairness pertain specifically to predictions used in decisional contexts that induce predictive polarity—since these are contexts in which people advance claims that are potentially conflicting.
Fig. 2 Non-polar and polar prediction-supported health care decisions. Understanding the specific decisional context of a prediction-supported decision in healthcare is necessary to anticipate potential unfairness. In the medical context—particularly in the shared decision-making context—patients and providers often share a common goal of accurate prognostication in order to help balance benefits and harms for care individualization. Predictions supporting decisions in this context may be described as “non-polar” (a). On the other hand, when one “pole” of the prediction is associated with a clear benefit or a clear harm, predictions may be described as “polar” in nature. In cases of polar predictions, the decision maker’s interest in efficient decision making (i.e. based on accurate prognostication using all available information) is not aligned with the subject’s interest to have either a lower (e.g. screening for abuse risk) or higher (e.g. microallocation of organs) prediction. “Positively” polar predictions correspond to those where patients may have an interest to be ranked high to receive a service that may be available only to some of those who can potentially benefit (b). This is in distinction to “negatively” polar predictions, in which prediction is used for the targeting of an intervention perceived as punitive or coercive (e.g. such as involuntary commitment, screening for child abuse or mandatory quarantining those at high infectious risk) (c). Issues of fairness pertain specifically to predictions used in decisional contexts that induce predictive polarity—since these are contexts in which people advance claims that are potentially conflicting.CC BY · PubMed Central
Fig. 3 Mitigating algorithmic bias and unfairness in clinical decision-making. Bias arises through differential model performance across protected classes, such as across racial groups. a It is a concern in both polar and non-polar decision contexts and can be addressed by “debiasing” predictions, typically through the explicit encoding of the protected attribute to ameliorate subgroup validity issues, or by the more thoughtful selection of labels (in the case of labeling bias). Fairness concerns are exclusively a concern in polar decision contexts, and may persist even when prediction is not biased. b There are two broad and fundamentally very different unfairness mitigation approaches: (1) an input-focused approach, and (2) an output-focused approach (Fig. 3b). The goal of the input-focused approach is to promote class-blind allocation by meticulously avoiding the inclusion of race or race proxies. The output-focused approach evaluates fairness using criteria such as those described in Table 1 and Fig. 1. Fairness violations can be (partially) addressed through the use of “fairness constraints” (which systematically reclassify participants/patients to equalize allocation between groups) or by applying different decision thresholds across groups.
Fig. 3 Mitigating algorithmic bias and unfairness in clinical decision-making. Bias arises through differential model performance across protected classes, such as across racial groups. a It is a concern in both polar and non-polar decision contexts and can be addressed by “debiasing” predictions, typically through the explicit encoding of the protected attribute to ameliorate subgroup validity issues, or by the more thoughtful selection of labels (in the case of labeling bias). Fairness concerns are exclusively a concern in polar decision contexts, and may persist even when prediction is not biased. b There are two broad and fundamentally very different unfairness mitigation approaches: (1) an input-focused approach, and (2) an output-focused approach (Fig. 3b). The goal of the input-focused approach is to promote class-blind allocation by meticulously avoiding the inclusion of race or race proxies. The output-focused approach evaluates fairness using criteria such as those described in Table 1 and Fig. 1. Fairness violations can be (partially) addressed through the use of “fairness constraints” (which systematically reclassify participants/patients to equalize allocation between groups) or by applying different decision thresholds across groups.CC BY · PubMed Central

Verification

Claims the paper makes

7 Valid · 0 Not valid · 20 Not checked · 42 Own results and interpretations

26 more claims · 1 with their cited sources checked

Create a free account to see all 69 claims and to verify the 31 references still open.

See all claims — free account
  • It may state two separate claims as one. 18
  • Probably not a substantial claim. 25

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 47 references

Research works printed but not linked: 1

  1. Verma, S. & Rubin, J. In ACM/IEEE International Workshop on Software Fairness (ACM, Gothenburg, Sweden, 2018).

Cited by

See all 274

Metadata

Authors
Paulus, J. K., & Kent, D. M.
Affiliations
Tufts University (US)
Corresponding author
Kent, David M.
Venue
npj Digital Medicine
Published
2020-07-30
Volume
3
Issue
1
Pages
99-99
ISSN
2398-6352 (ISSN-L)
OpenAlex
W3046113390
Cited by
274
References
47
Citations by year
2026: 49 · 2025: 72 · 2024: 51 · 2023: 50 · 2022: 33 · 2021: 17 · 2020: 2
FWCI
4.95
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Patient-Centered Outcomes Research Institute (ME-1606-35555)
  • National Institutes of Health (UL1TR002544, TL1TR002546)
  • National Center for Advancing Translational Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
algorithmic fairness, algorithmic bias, health disparities, clinical prediction model, healthcare resource allocation, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities. https://app.cochat.ai/papers/10.1038/s41746-020-0304-9
The paper
Paulus, J. K., & Kent, D. M. (2020). Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities. npj Digital Medicine. https://doi.org/10.1038/s41746-020-0304-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.