Inherent Trade-Offs in the Fair Determination of Risk Scores
Open accessAbstract
Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When base rates differ between groups, any combination of demographic parity, equalised odds, and equal calibration cannot be satisfied simultaneously.
“In particular, when base rates (i.e. the proportions of positives) differ between groups, any combination of demographic parity, equalised odds, and equal calibration cannot be satisfied simultaneously [33,65].”
From Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree (2023) · cites this paper as [65]
The Characterization Theorems (p. 11) say that, with unequal base rates, statistical parity is inconsistent with calibration and with the two balance conditions taken together. But the claim's “any combination” is too broad: the paper allows simultaneous satisfaction of calibration and both balance conditions under perfect prediction (p. 11), and does not rule out every smaller combination. (LLM, conf=0.90)
Experimental — accuracy not yet measuredSatisfying all three fairness expectations (equal accuracy, equal false positive rates, equal false negative rates) at the same time would be impossible in most situations; meeting two will mean failing to comply with the third.
“recent work has established that satisfying all three at the same time would be impossible in most situations; meeting two will mean failing to comply with the third.4,7”
From Engaging the ethics of data science in practice (2017) · cites this paper as [7]
The characterization theorem (Section 1.2, p. 5) supports the central trade-off: all three conditions can hold together only with perfect prediction or equal base rates. But the paper defines its conditions as calibration within groups and balance in average scores for negative and positive cases (Section 1.1, p. 4), not equal accuracy and equal false-positive and false-negative rates. It calls the latter two conditions generalizations of equal error rates (Introduction, p. 2). (LLM, conf=0.88)
Experimental — accuracy not yet measuredTrying to satisfy a particular notion of fairness may cause harm in another dimension without the developers realizing it.
“By trying to satisfy a particular notion of fairness (e.g., according to a particular quantitative definition proposed in prior literature), one might actually cause harm in another dimension without realizing it”
From Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education? (2021) · cites this paper as Kleinberg, Mullainathan, Raghavan, 2016
Section 1.2 (pp. 5–6) establishes that, outside narrow exceptions, risk scores cannot satisfy all three fairness conditions and may be criticized as biased under another definition. This supports the claimed trade-off, but the paper does not establish that pursuing one definition causes harm or that developers fail to realize the trade-off. (LLM, conf=0.84)
Experimental — accuracy not yet measuredMuch of the recent debate surrounding machine bias in algorithmic scores has come down to the tension between competing notions of what it means for a probabilistic classification to be fair to different groups.
“Much of the recent debate surrounding “machine bias” in algorithmic scores has come down to the “tension between competing notions of what it means for a probabilistic classification to be fair to different groups” (Kleinberg, Mullainathan, and Raghavan 2016).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Kleinberg, Mullainathan, and Raghavan 2016
Key notions of fairness central to algorithmic debates are essentially incompatible; it is impossible for a risk score to satisfy both fairness criteria at the same time.
“Key notions of fairness that have been central to these algorithmic debates are essentially incompatible: “it's actually impossible for a risk score to satisfy both fairness criteria at the same time” (Corbett‐Davies et al. 2016; see also Kleinberg, Mullainathan, and Raghavan 2016).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Kleinberg, Mullainathan, and Raghavan 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 941
Research works printed but not linked: 21
- [3][3] Solon Barocas and Andrew D Selbst. Big data's disparate impact. California Law Review , 104, 2016. -
- [4][4] Toon Calders and Sicco Verwer. Three naive bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery , 21(2):277-292, 2010. -
- [5][5] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. arXiv preprint arXiv:1610.07524 , October 2016. - · arXiv
- [8][8] Cynthia S. Crowson, Elizabeth J. Atkinson, and Terry M. Therneau. Assessing calibration of prognostic risk scores. Statistical Methods in Medical Research , 25(4):1692-1706, 2016. -
- [9][9] Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. Proceedings on Privacy Enhancing Technologies , 2015(1):92-112, 2015. -
- [11][11] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science 2012, Cambridge, MA, USA, January 8-10, 2012 , pages 214-226, 2012. -
- [12][12] Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '15, pages 259-268, New York, NY, USA, 2015. ACM. -
- [14][14] Dean P. Foster and Rakesh V. Vohra. Asymptotic calibration. Biometrika , 85(2):379-390, 1998. -
- [15][15] Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. On the (im) possibility of fairness. arXiv preprint arXiv:1609.07236 , September 2016. - · arXiv
- [16][16] Howard N. Garb. Race bias, social class bias, and gender bias in clinical judgment. Clinical Psychology: Science and Practice , 4(2):99-120, 1997. -
- [18][18] Sara Hajian and Josep Domingo-Ferrer. A methodology for direct and indirect discrimination prevention in data mining. IEEE transactions on knowledge and data engineering , 25(7):1445-1459, 2013. -
- [19][19] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems , 2016. -
- [20][20] Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. In Advances in Neural Information Processing Systems , 2016. -
- [21][21] Faisal Kamiran and Toon Calders. Classifying without discriminating. 2009 2nd International Conference on Computer, Control and Communication, IC4 2009 , 2009. -
- [22][22] Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. Fairness-aware learning through regularization approach. Proceedings - IEEE International Conference on Data Mining, ICDM , pages 643-650, 2011. -
- [25][25] Andrea Romei and Salvatore Ruggieri. A multidisciplinary survey on discrimination analysis. The Knowledge Engineering Review , 29(05):582-638, 2014. -
- [26][26] Latanya Sweeney. Discrimination in online ad delivery. Communications of the ACM , 56(5):44-54, 2013. -
- [27][27] David R. Williams and Selina A. Mohammed. Discrimination and racial disparities in health: Evidence and needed research. J. Med. Behav. , 32(1), 2009. -
- [28][28] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna Gummadi. Learning fair classifiers. arXiv preprint arXiv:1507.05259 , 2015. - · arXiv
- [29][29] Richard S Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. Learning Fair Representations. Proceedings of the 30th International Conference on Machine Learning , 28:325-333, 2013. -
- [30][30] Indre Zliobaite. A survey on measuring indirect discrimination in machine learning. arXiv preprint arXiv:1511.00148 , 2015. · arXiv
- Authors
- Kleinberg, J., Mullainathan, S., & Raghavan, M.
- Affiliations
- Cornell University (US)
- Harvard University (US)
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1609.05807
- OpenAlex
- W2962922665
- Cited by
- 941
- References
- 0
- Citations by year
- 2026: 49 · 2025: 74 · 2024: 112 · 2023: 139 · 2022: 102 · 2021: 152 · 2020: 129 · 2019: 106 · 2018: 54 · 2017: 21 · 2016: 1 · 2013: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Leibniz international proceedings in informatics (cc-by)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- algorithmic fairness, risk score, probabilistic classification, fairness conditions, impossibility results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
