The DOI matches its DataCite record; 21 of the 35 references it cites were found in a registry or an index.
Inherent Trade-Offs in the Fair Determination of Risk Scores
Open accessAbstract
Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“each technical definition carry strong values assumptions which can often be in tension with each other (Friedler et al., 2016; Kleinberg et al., 2017).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Kleinberg et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing literature often treats fairness and performance as a trade-off problem”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Kleinberg et al., 2017
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Many fairness metrics have been proposed, and the most widely used criteria were shown to be incompatible ([20], [41]), and to present risks for the populations they were meant to protect ([24]).”
From Fairness and explainability in automatic decision-making systems. A challenge for computer science and law (2023) · cites this paper as [41]
The cited source says what the claim attributes to it.
In the citing paper
“commonly used statistical fairness criteria can be mutually incompatible except under restrictive conditions”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Kleinberg et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“Kleinberg, Mullainathan and Raghavan (2017) shows theoretically that these notions of fairness are usually incompatible.”
From An algorithm for removing sensitive information: Application to race-independent recidivism prediction (2019) · cites this paper as Kleinberg, Mullainathan and Raghavan (2017)
The cited source says what the claim attributes to it.
In the citing paper
“These two notions of fairness are indeed distinct.”
From An algorithm for removing sensitive information: Application to race-independent recidivism prediction (2019) · cites this paper as Kleinberg, Mullainathan and Raghavan (2017)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“propose a framework based on a task-specific externally defined similarity metric between individuals, seeking to achieve fairness through the goal that “similar people [be] treated similarly” [11].”
BackgroundDiscrimination and racial disparities in health: evidence and needed research
Williams, D. R., & Mohammed, S. A. (2009) · Journal of Behavioral Medicine · doi:10.1007/s10865-008-9185-0 · Cited by 3,371 · cited 1×
“Here too we can ask whether such decision-making is being applied uniformly across different groups of patients [16, 27], and in particular how medical tests may play a differential role for conditions that vary widely in frequency between these groups.”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Barocas and Selbst survey a range of ways in which data-analysis algorithms can lead to discriminatory outcomes [3], and review articles by Romei and Ruggieri [25] and Zliobaite [30] survey data-analytic and algorithmic methods for measuring discrimination.”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“consider the natural analogues of our conditions (B) and (C), balance for the negative and positive classes, in the case of classifiers that output binary “yes/no” predictions rather than real-valued scores as in our case [19].”
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 3×
“Also concurrent with our work and that of Hardt et al., Chouldechova [5] and Corbett-Davies et al. [6] (and see also [7]) consider binary prediction subject to these same analogues of the balance conditions for the negative and positive classes, together with a form of calibration adapted to binary prediction (requiri…”
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 1×
“This latter global requirement is a version of statistical parity [12, 4, 21, 22].”
BackgroundAlgorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. Z. (2017) · Knowledge Discovery and Data Mining · doi:10.1145/3097983.309809 · Cited by 1,464 · cited 1×
“[7] (and see also [6]) consider binary prediction subject to analogues of the balance conditions for the negative and positive classes, together with a form of calibration adapted to binary prediction.”
BackgroundDiscrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“Second, in a very different domain, researchers have begun to analyze the ways in which different genders and racial groups experience advertising and commercial content on the Internet differently [9, 26].”
BackgroundThree naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · cited 3×
“Work in these directions has developed learning algorithms that penalize violations of statistical parity [4, 22].”
BackgroundAutomated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination
Datta, A., Tschantz, M. C., & Datta, A. (2015) · Proceedings on Privacy Enhancing Technologies · doi:10.1515/popets-2015-0007 · Cited by 676 · cited 1×
“Second, in a very different domain, researchers have begun to analyze the ways in which different genders and racial groups experience advertising and commercial content on the Internet differently [9, 26].”
BackgroundFairness Constraints: Mechanisms for Fair Classification
Zafar, M. B., Valera, I., Rodriguez, M. G., & Gummadi, K. P. (2015) · arXiv · doi:10.48550/arxiv.1507.05259 · Cited by 645
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · cited 6×
“Kamiran and Calders [21] and Hajian and Domingo-Ferrer [18]seek to modify datasets to remove any information that might permit discrimination.”
BackgroundInfluentialFairness-aware Learning through Regularization Approach
Kamishima, T., Akaho, S., & Sakuma, J. (2011) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2011.83 · Cited by 383 · cited 3×
“Work in these directions has developed learning algorithms that penalize violations of statistical parity [4, 22].”
BackgroundA Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312 · cited 3×
“Kamiran and Calders [21] and Hajian and Domingo-Ferrer [18] seek to modify datasets to remove any information that might permit discrimination.”
BackgroundIEEE Transactions on Knowledge and Data Engineering
(2004) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2004.1269660 · Cited by 301 · cited 3×
“Kamiran and Calders [21] and Hajian and Domingo-Ferrer [18] seek to modify datasets to remove any information that might permit discrimination.”
BackgroundAssessing calibration of prognostic risk scores
Crowson, C. S., Atkinson, E. J., & Therneau, T. M. (2013) · Statistical Methods in Medical Research · doi:10.1177/0962280213497434 · Cited by 295 · cited 2×
“A first basic goal in this literature is that the probability estimates provided by the algorithm should be well-calibrated: if the algorithm identifies a set of people as having a probability z of constituting positive instances, then approximately a z fraction of this set should indeed be positive instances [8, 14].”
BackgroundA multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288 · cited 2×
“Barocas and Selbst survey a range of ways in which data-analysis algorithms can lead to discriminatory outcomes [3], and review articles by Romei and Ruggieri [25] and Zliobaite [30] survey data-analytic and algorithmic methods for measuring discrimination.”
BackgroundOn the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247 · cited 1×
“[15] defines two axiomatic pro pe ties of feature generation and shows that no mechanism can be fair under these two properties.”
BackgroundFairness in Learning: Classic and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., & Roth, A. (2016) · arXiv · doi:10.48550/arxiv.1605.07139 · Cited by 199 · cited 1×
“consider how fairness issues can arise during the process of learning, modeling this using a multi-armed bandit framework [20].”
MethodRace bias, social class bias, and gender bias in clinical judgment.
Garb, H. N. (1997) · Clinical Psychology: Science and Practice · doi:10.1111/j.1468-2850.1997.tb00104.x · Cited by 117 · cited 1×
“Here too we can ask whether such decision-making is being applied uniformly across different groups of patients [16, 27], and in particular how medical tests may play a differential role for conditions that vary widely in frequency between these groups.”
BackgroundA survey on measuring indirect discrimination in machine learning
Zliobaite, I. (2015) · arXiv · doi:10.48550/arxiv.1511.00148 · cited 1×
“Barocas and Selbst survey a range of ways in which data-analysis algorithms can lead to discriminatory outcomes [3], and review articles by Romei and Ruggieri [25] and Zliobaite [30] survey data-analytic and algorithmic methods for measuring discrimination.”
Background
Research works printed but not linked: 3
- [9][9] Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. Proceedings on Privacy Enhancing Technologies , 2015(1):92-112, 2015. -
- [14][14] Dean P. Foster and Rakesh V. Vohra. Asymptotic calibration. Biometrika , 85(2):379-390, 1998. -
- [29][29] Richard S Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. Learning Fair Representations. Proceedings of the 30th International Conference on Machine Learning , 28:325-333, 2013. -
Cited by
Sorted by citations
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
“It is important to note that according to [70], it is impossible to satisfy some of the fairness constraints at once except in highly constrained special cases.”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
“First, there is no way to arbitrate between irreconcilably conflicting definitions using purely mathematical means [14, 42].”
BackgroundAlgorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124
“Well-calibration [16].”
BackgroundResultInfluential
Metadata
- Authors
- Kleinberg, J., Mullainathan, S., & Raghavan, M.
- Affiliations
- Cornell University (US)
- Harvard University (US)
- University of California, Berkeley (US)
- Venue
- Schloss Dagstuhl – Leibniz-Zentrum für Informatik
- Publisher
- Schloss Dagstuhl – Leibniz-Zentrum für Informatik
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- OpenAlex
- W2962922665
- Cited by
- 943
- References
- 35
- Citations by year
- 2026: 49 · 2025: 74 · 2024: 112 · 2023: 139 · 2022: 102 · 2021: 153 · 2020: 129 · 2019: 107 · 2018: 54 · 2017: 21
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Leibniz international proceedings in informatics (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- algorithmic fairness, risk score, probabilistic classification, fairness conditions, impossibility results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
