The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 11 of the 42 sources its claims cite were checked for support.
Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities
Open accessAbstract
The machine learning community has become alert to the ways that predictive algorithms can inadvertently introduce unfairness in decision-making. Herein, we discuss how concepts of algorithmic fairness might apply in healthcare, where predictive algorithms are being increasingly used to support decision-making. Central to our discussion is the distinction between algorithmic fairness and algorithmic bias. Fairness concerns apply specifically when algorithms are used to support polar decisions (i.e., where one pole of prediction leads to decisions that are generally more desired than the other), such as when predictions are used to allocate scarce health care resources to a group of patients that could all benefit. We review different fairness criteria and demonstrate their mutual incompatibility. Even when models are used to balance benefits-harms to make optimal decisions for individuals (i.e., for non-polar decisions)–and fairness concerns are not germane–model, data or sampling issues can lead to biased predictions that support decisions that are differentially harmful/beneficial across groups. We review these potential sources of bias, and also discuss ways to diagnose and remedy algorithmic bias. We note that remedies for algorithmic fairness may be more problematic, since we lack agreed upon definitions of fairness. Finally, we propose a provisional framework for the evaluation of clinical prediction models offered for further elaboration and refinement. Given the proliferation of prediction models used to guide clinical decisions, developing consensus for how these concerns can be addressed should be prioritized.
Overview
CoChat Summary
TL;DR
This perspective distinguishes algorithmic fairness from algorithmic bias in clinical prediction, arguing fairness concerns are germane mainly for “polar” decisions where one prediction pole is generally desired or undesired, while bias can affect both polar and non-polar decisions. It reviews mutually incompatible fairness criteria, sources of biased data/modeling, and proposes a provisional framework for evaluating clinical prediction models.
Key findings
- Fairness concerns apply specifically to “polar” predictions—where one pole leads to decisions generally more desired than the other, such as allocating scarce health care resources—whereas non-polar decisions aimed at balancing benefits and harms for an individual do not raise the same fairness concerns, though algorithmic bias can still occur.
- Common fairness criteria are mutually incompatible: equalized error rates and calibration/test fairness cannot both be satisfied when outcome rates differ across groups, except under perfect prediction. In the paper’s numerical illustration with a low-prevalence group at 20% and high-prevalence group at 30%, a test with 80% sensitivity and specificity yields positive predictive values of 50% vs ~63% and false omission rates of ~6% vs ~10%; holding risk-strata outcome rates constant would nearly double the false positive rate from 20% to ~39%.
- Algorithmic bias can arise from label bias, feature bias, differential missingness, subgroup validity problems, and sampling bias; examples include surveillance-sensitive cancer overdiagnosis in affluent communities, ICU mortality predictions shaped by practice patterns, the racially homogeneous Framingham study, and polygenic risk scores derived largely on European populations performing poorly on non-European populations.
- The proposed framework separates ensuring unbiased models—diagnosing subgroup performance and using group main effects, interactions, or stratified models—from addressing fairness in polar contexts via input-focused approaches (class-blind allocation, avoiding race proxies, transparency) or output-focused approaches (fairness constraints, different decision thresholds), with no agreed mathematical solution to define fairness.
Methods
- Perspective/review article that synthesizes methodological research from computer science and relevant epidemiological principles; no primary empirical data collection is reported.
- Uses conceptual analysis, mathematical demonstration of fairness-criteria incompatibility, a numerical illustration with N = 100 confusion matrices, and examples such as COMPAS data from over 7000 arrests and the Framingham Heart Study.
- Proposes a provisional framework for evaluating algorithmic bias and fairness in clinical prediction and decision-making, distinguishing polar from non-polar predictions.
Limitations
- Stated: the framework is simplified and provisional, intended as a starting point; some algorithms are applied in different decisional contexts with different ethical concerns (e.g., race-aware estimated GFR equations used for transplant prioritization and medication dosing); polarity may be non-obvious; some objections to using race in prediction models have little to do with unfairness as described; and identifying/mitigating algorithmic disparities involves political and legal complexities requiring integration of statistical and legal thinking.
- Evident from the text: the framework is not empirically validated in the paper; fairness definitions remain contested and lack agreed mathematical solutions; causal definitions of fairness are described as fundamentally unidentifiable in observational data except with unverifiable assumptions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
7 Valid · 0 Not valid · 20 Not checked · 42 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“For similar reasons, there are concerns about the representativeness of the Precision Medicine Initiative (the “All of Us” Study28).”
§ Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities > Learning from biased data
What the sources say
- Supportedcites [28] Sabatello, M. & Appelbaum, P. S. The precision medicine nation. Hastings Cent. Rep. 47, 19–29 (2017). source
Our copy identifies the PMI's cohort program as the "All of Us Research Program" (Constructing Genomic Citizenship) and repeatedly raises concerns about its representativeness: the cohort must be "both large and reflective of the U.S.'s diversity—features that, to date, have been difficult to achieve" (Constructing Genomic Citizenship); enrollment decisions "will unavoidably shape the cohort's composition in ways that may limit the access of particular subgroups," with a tension between a representative sample of the US population and over-representing underrepresented/high-disease-burden groups (Demarcating Boundaries); and limitations of enrollment strategies plus difficulty enrolling minority racial/ethnic groups (Benefits). The representativeness of public respondents in PMI engagement is also flagged as unclear (Governance and Participation). (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Indeed, race is used explicitly in popular prediction models that inform the need for osteoporotic33, breast34–37 and prostate cancer screening38,39; statin use for coronary heart disease prevention40,41 and other common decisions.”
§ Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities > Should the use of protected characteristics in clinical prediction models (CPM) differ for polar versus non-polar predictions?
What the sources say
- Not checked yetcites [33] WHO. Centre for Metabolic Bone Diseases (University of Sheffield, 2008).
- Partly supportedcites [34] Banegas, M. P. et al. Projecting individualized absolute invasive breast cancer risk in US hispanic women. J. Natl. Cancer Inst. 109, djw215 (2017). sourceExperimental — accuracy not yet measured
The paper’s introductory text describes the Breast Cancer Risk Assessment Tool as estimating risk for specified racial and ethnic groups, and its Discussion describes a risk model developed specifically for Hispanic women. This supports the breast-cancer prediction-model portion attributed to this citation. The passages do not establish that these models are used to decide whether breast cancer screening is needed; the other conditions and decisions have separate citations in the citing sentence. (LLM, conf=0.83)
- Not checked yetcites [35] Gail, M. H. et al. Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J. Natl. Cancer Inst. 81, 1879–1886 (1989). source
- Not checked yetcites [36] Gail, M. H. et al. Projecting individualized absolute invasive breast cancer risk in African American women. J. Natl. Cancer Inst. 99, 1782–1792 (2007). source
- Partly supportedcites [37] Matsuno, R. K. et al. Projecting individualized absolute invasive breast cancer risk in Asian and Pacific Islander American women. J. Natl. Cancer Inst. 103, 951–961 (2011). sourceExperimental — accuracy not yet measured
The Comparisons with BCRAT section says breast cancer risk models use race- and ethnicity-specific data. The Implications section recommends the AABCS model for counseling and determining eligibility for prevention trials, but our copy does not say it informs decisions about breast cancer screening. The other clinical decisions in the citing sentence are attached to other citations. (LLM, conf=0.85)
- Supportedcites [38] Ankerst, D. P. et al. Prostate cancer prevention trial risk calculator 2.0 for the prediction of low-versus high-grade prostate cancer. Urology 83, 1362–1368 (2014). source
The Introduction says the PCPT Risk Calculator uses African American race among its clinical risk factors to help decide whether to biopsy. The Results section confirms that African American race is a predictor in the updated model. This supports the prostate-cancer portion of the citing sentence to which this source is attached. (LLM, conf=0.94)
- Not checked yetcites [39] National Cancer Institute. Prostate Cancer Prevention Trial Investigators (UT Health San Antonio, 2012).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“how principles of algorithmic fairness might apply in clinical decision-making has received little attention in the medical literature7,8.”
§ Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities > Background
What the sources say
- Partly supportedcites [7] Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G. & Chin, M. H. Ensuring fairness in machine learning to advance health equity. Ann. Intern Med. 169, 866–872 (2018). sourceExperimental — accuracy not yet measured
The paper’s Abstract (page unavailable) discusses how fairness principles can be applied to clinical machine-learning decisions. It does not say that this question has received little attention in the medical literature; searches for that characterization found no supporting passage. (LLM, conf=0.78)
- Partly supportedcites [8] Char, D. S., Shah, N. H. & Magnus, D. Implementing machine learning in health care–addressing ethical challenges. New Engl. J. Med. 378, 981–983 (2018). sourceExperimental — accuracy not yet measured
The paper’s main text discusses bias in health care algorithms and calls for machine-learning systems to reflect ethical standards in clinical care. It does not say that applying algorithmic fairness principles to clinical decisions has received little attention in the medical literature; that assessment of the literature is unsupported. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We have previously argued that much of the reluctance to use race in prediction models stems from overgeneralization of its potentially objectionable use in polar predictions in non-medical settings to its use for non-polar predictions in medical settings29.”
§ Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities > Should the use of protected characteristics in clinical prediction models (CPM) differ for polar versus non-polar predictions?
What the sources say
- Partly supportedcites [29] Paulus, J. K. & Kent, D. M. Race and ethnicity: a part of the equation for personalized clinical decision making? Circ. Cardiovasc. Qual. Outcomes. https://doi.org/10.1161/CIRCOUTCOMES.117.003823 (2017). sourceExperimental — accuracy not yet measured
The Summary and Conclusion (page unavailable) says that some reluctance to use race in clinical prediction models comes from overgeneralizing concerns from other contexts. The Important Distinctions section explains why predictions in law enforcement or insurance can disadvantage profiled individuals while medical predictions can help align care with patients’ goals. The source supports the distinction, but says “some,” not “much,” of the reluctance stems from overgeneralization. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Currently, there is no consensus or guidance on how protected characteristics–race in particular– should be incorporated in clinical prediction29.”
§ Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities > Should the use of protected characteristics in clinical prediction models (CPM) differ for polar versus non-polar predictions?
What the sources say
- Supportedcites [29] Paulus, J. K. & Kent, D. M. Race and ethnicity: a part of the equation for personalized clinical decision making? Circ. Cardiovasc. Qual. Outcomes. https://doi.org/10.1161/CIRCOUTCOMES.117.003823 (2017). source
The cited paper's section heading and text ('Race Effects are Sometimes Included in Clinical Predictive Models, but Usually Not: A Lack of Consensus Regarding Best Practice') and its Summary and Conclusion state directly that 'There is clearly a lack of consensus in how to cope with race as a predictor in CPMs' and that more stakeholder discussion is needed 'to develop consensus about best practice' — i.e., no settled guidance on incorporating race in clinical prediction models. The surrounding passages (Rationale for Excluding Race Effects; Caveats) reinforce the absence of agreed practice, noting only that actuarial/insurance use of race is proscribed and that the 2014 NQF Expert Panel recommendation concerns performance measures rather than prediction models, so nothing contradicts the 'no guidance' element. The claim's broader framing ('protected characteristics… race in particular') is a mild generalization of the paper's race/ethnicity focus, but the attributed point is the paper's own. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
26 more claims · 1 with their cited sources checked
Create a free account to see all 69 claims and to verify the 31 references still open.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 25
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since screening is not an accurate reflection of the incidence of cancer, this in-turn would result in a biased algorithm that targets screening to over-served communities, leading to further health disparities [31].”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [31]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Communities in which cancer screenings are frequently performed will have inflated incidences compared to underserved populations that would consequently be “labeled” as having a lower incidence of cancer due to lesser screening in those areas.”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [31]
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“The bias was detected because black patients were sicker than similarly scored white patients, and the algorithm was remedied through the use of a better label that more directly captures health need 45 .”
BackgroundAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945 · cited 1×
“Indeed, the widely-used Pooled Cohort Equation for coronary heart disease prediction addressed the subgroup validity issues identi fi ed in the Framingham score (i.e., poor model performance in ethnic minorities) by developing separate models for whites and African-Americans 42 .”
MethodThe Book of Why: The New Science of Cause and Effect
Pearl, J., & Mackenzie, D. N. (2018) · Cited by 3,779
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Gail, M. H., Brinton, L. A., Byar, D. P., Corle, D. K., Green, S. B., Schairer, C., & Mulvihill, J. J. (1989) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/81.24.1879 · Cited by 3,334
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 2×
“None-theless it ’ s important to bear in mind that fairness and the legal standard of disparate impact are not purely statistical concepts, and involve ethical, political and constitutional concerns 15 .”
BackgroundClinical use of current polygenic risk scores may exacerbate health disparities
Martin, A. R., Kanai, M., Kamatani, Y., Okada, Y., Neale, B. M., & Daly, M. J. (2019) · Nature Genetics · doi:10.1038/s41588-019-0379-x · Cited by 2,821
Current clinical use of polygenic scores will risk exacerbating health disparities
Martin, A. R., Kanai, M., Kamatani, Y., Okada, Y., Neale, B. M., & Daly, M. J. (2018) · doi:10.1101/441261 · Cited by 2,813
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Research works printed but not linked: 1
- Verma, S. & Rubin, J. In ACM/IEEE International Workshop on Software Fairness (ACM, Gothenburg, Sweden, 2018).
Cited by
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798
“50 62-64 Consideration of key subgroups is therefore important during the design (and data collection), analysis, reporting, and interpretation of findings.”
BackgroundBias in artificial intelligence algorithms and recommendations for mitigation
Nazer, L. H., Zatarah, R., Waldrip, S., Ke, J. X. C., Moukheiber, M., Khanna, A. K., Hicklen, R. S., Moukheiber, L., Moukheiber, D., Ma, H., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000278 · Cited by 621
“Another bias encountered in data collection is label bias, which is seen when the outcome variable is differentially ascertained or has a different meaning across groups [31].”
BackgroundReal-world data: A brief review of the methods, applications, challenges and opportunities
Liu, F., & Panagiotakos, D. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01768-6 · Cited by 496
“Algorithmic fairness is an increasingly popular research topic in literature [123– 127].”
BackgroundThe value of standards for health datasets in artificial intelligence-based applications
Arora, A., Alderman, J. E., Palmer, J., Ganapathi, S., Laws, E., McCradden, M. D., Oakden-Rayner, L., Pfohl, S. R., Ghassemi, M., McKay, F., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02608-w · Cited by 294
The limits of fair medical imaging AI in real-world generalization
Yang, Y., Zhang, H., Gichoya, J. W., Katabi, D., & Ghassemi, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-03113-4 · Cited by 285
Metadata
- Authors
- Paulus, J. K., & Kent, D. M.
- Affiliations
- Tufts University (US)
- Corresponding author
- Kent, David M.
- Venue
- npj Digital Medicine
- Published
- 2020-07-30
- Volume
- 3
- Issue
- 1
- Pages
- 99-99
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC7393367
- OpenAlex
- W3046113390
- Semantic Scholar
- 11686792dc3f8008a87b813bbefe59124145924d
- Cited by
- 274
- References
- 47
- Citations by year
- 2026: 49 · 2025: 72 · 2024: 51 · 2023: 50 · 2022: 33 · 2021: 17 · 2020: 2
- FWCI
- 4.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Patient-Centered Outcomes Research Institute (ME-1606-35555)
- National Institutes of Health (UL1TR002544, TL1TR002546)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- algorithmic fairness, algorithmic bias, health disparities, clinical prediction model, healthcare resource allocation, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities. https://app.cochat.ai/papers/10.1038/s41746-020-0304-9
- The paper
- Paulus, J. K., & Kent, D. M. (2020). Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities. npj Digital Medicine. https://doi.org/10.1038/s41746-020-0304-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.