Algorithmic Decision Making and the Cost of Fairness
Open accessThe DOI matches its Crossref record; 41 of the 43 references it cites were found in a registry or an index.
Abstract
Algorithms are now regularly used to decide whether defendants awaiting trial are too dangerous to be released back into the community. In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk. To mitigate such disparities, several techniques have recently been proposed to achieve algorithmic fairness. Here we reformulate algorithmic fairness as constrained optimization: the objective is to maximize public safety while satisfying formal fairness constraints designed to reduce racial disparities. We show that for several past definitions of fairness, the optimal algorithms that result require detaining defendants above race-specific risk thresholds. We further show that the optimal unconstrained algorithm requires applying a single, uniform threshold to all defendants. The unconstrained algorithm thus maximizes public safety while also satisfying one important understanding of equality: that all individuals are held to the same standard, irrespective of race. Because the optimal constrained and unconstrained algorithms generally differ, there is tension between improving public safety and satisfying prevailing notions of algorithmic fairness. By examining data from Broward County, Florida, we show that this trade-off can be large in practice. We focus on algorithms for pretrial release decisions, but the principles we discuss apply to other domains, and also to human decision makers carrying out structured decision rules.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There is always a trade-off between accuracy, fairness and privacy that needs to be taken into account as Corbett-Davies et al. [31] examined in their study.”
In the citing paper
“applications of AI to the tax domain -which need to deal with large volumes of taxpayer data and, in applications such as tax fraud detection, achieve high levels of accuracy -tend to rely on machine learning models that are not transparent by design.”
From Towards eXplainable Artificial Intelligence (XAI) in Tax Law: The Need for a Minimum Legal Standard (2022) · cites this paper as [111]
In the citing paper
“However, it is important to recognize that bias is not the only factor that can lead to unfairness, and achieving fairness may require additional efforts beyond bias mitigation [27].”
From Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies (2023) · cites this paper as [27]
In the citing paper
“Corbett-Davies and collaborators introduced risk-minimization approaches aimed at minimizing disparities [27,28].”
From Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies (2023) · cites this paper as [27]
In the citing paper
“several studies have observed that satisfying multiple fairness conditions at the same time is infeasible [7,9,22,23].”
From A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices (2018) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,384
The Economics of Discrimination.
Collard, D., & Becker, G. S. (1972) · The Economic Journal · doi:10.2307/2230052 · Cited by 3,573
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
The difference: how the power of diversity creates better groups, firms, schools, and societies
(2007) · Choice Reviews Online · doi:10.5860/choice.45-1534 · Cited by 2,280
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Platt, J. C. (2000) · Advances in Large-Margin Classifiers · doi:10.7551/mitpress/1113.003.0008 · Cited by 1,156
(2013) · Cited by 1,126
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694
The Economics of Discrimination.
Alchian, A. A., & Becker, G. (1958) · Journal of the American Statistical Association · doi:10.2307/2281986 · Cited by 674
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632
Lum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611
Evidence-Based Sentencing and the Scientific Rationalization of Discrimination
Starr, S. B. (2013) · bepress Legal Repository · Cited by 356
RISK, RACE, AND RECIDIVISM: PREDICTIVE BIAS AND DISPARATE IMPACT*
SKEEM, J. L., & LOWENKAMP, C. T. (2016) · Criminology · doi:10.1111/1745-9125.12123 · Cited by 317
A multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288
Precinct or prejudice? Understanding racial disparities in New York City’s stop-and-frisk policy
Goel, S., Rao, J. M., & Shroff, R. (2016) · The Annals of Applied Statistics · doi:10.1214/15-aoas897 · Cited by 285
Self-citationRisk Assessment in Criminal Sentencing
Monahan, J., & Skeem, J. L. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-092945 · Cited by 256
Fairness in Learning: Classic and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., & Roth, A. (2016) · arXiv · doi:10.48550/arxiv.1605.07139 · Cited by 199
Criminal Justice Forecasts of Risk: A Machine Learning Approach
Berk, R. A. (2012) · SpringerBriefs in computer science · doi:10.1007/978-1-4614-3085-8 · Cited by 192
UNIVERSITY OF TEXAS AT AUSTIN.: NETTIE LEE BENSON LATIN AMERICAN COLLECTION. MANUSCRIPT DEPARTMENT
(1991) · La revolución más allá del Bravo · doi:10.2307/j.ctv3dnqq7.15 · Cited by 179
Berk, R., Sherman, L., Barnes, G., Kurtz, E., & Ahlman, L. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00556.x · Cited by 178
Race, redlining, and residential mortgage loan performance
Berkovec, J. A., Canner, G. B., Gabriel, S. A., & Hannan, T. H. (1994) · The Journal of Real Estate Finance and Economics · doi:10.1007/bf01099279 · Cited by 163
Outcome Tests of Racial Disparities in Police Practices
Ayres, I. (2002) · Justice Research and Policy · doi:10.3818/jrp.4.1.2002.131 · Cited by 156
A Confidence-Based Approach for Balancing Fairness and Accuracy
Fish, B., Kun, J., & Lelkes, Á. D. (2016) · Proceedings of the 2016 SIAM International Conference on Data Mining · doi:10.1137/1.9781611974348.17 · Cited by 156
The problem of infra-marginality in outcome tests for discrimination
Simoiu, C., Corbett-Davies, S., & Goel, S. (2017) · The Annals of Applied Statistics · doi:10.1214/17-aoas1058 · Cited by 154
Self-citationKamiran, F., Žliobaitė, I., & Calders, T. (2013) · Knowledge and Information Systems · doi:10.1007/s10115-012-0584-8 · Cited by 99
An Alternative Test of Racial Prejudice in Motor Vehicle Searches: Theory and Evidence
Anwar, S., & Fang, H. (2005) · National Bureau of Economic Research · doi:10.3386/w11264 · Cited by 92
Simple Rules for Complex Decisions
Jung, J., Concannon, C., & Shroff, R. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2919024 · Cited by 87
Self-citationDiscrimination prevention in data mining for intrusion and crime detection
Hajian, S., Domingo-Ferrer, J., & Martinez-Balleste, A. (2011) · 2011 IEEE Symposium on Computational Intelligence in Cyber Security (CICS) · doi:10.1109/cicybs.2011.5949405 · Cited by 67
Fast Threshold Tests for Detecting DiscriminationPreprint
Pierson, E., Corbett‐Davies, S., & Goel, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08536 · Cited by 47
Self-citationThe Role of Race in Forecasts of Violent Crime
Berk, R. (2009) · Race and Social Problems · doi:10.1007/s12552-009-9017-z · Cited by 41
Personalized Risk Assessments in the Criminal Justice System
Goel, S., Rao, J. M., & Shroff, R. (2016) · American Economic Review · doi:10.1257/aer.p20161028 · Cited by 27
Self-citationHolder, E. (2015) · Federal Sentencing Reporter · doi:10.1525/fsr.2015.27.4.252 · Cited by 24
Data & Civil Rights: Criminal Justice PrimerPreprint
Rosenblat, A., Wikelius, K., Boyd, D., Gangadharan, S. P., & Yu, C. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2542262 · Cited by 7
Risk-Based Pretrial Release Recommendation and Supervision Guidelines
Danner, M. J. E., VanNostrand, M., & Spruce, L. M. (2015) · ODU Digital Commons (Old Dominion University) · Cited by 7
Research works printed but not linked: 7
- Lindsay C Ahlman and Ellen M Kurtz 2009. The APPD randomized controlled trial in low risk supervision: The effects on low risk supervision on rearrest. Philadelphia Adult Probation and Parole Department ( 2009 ). Lindsay C Ahlman and Ellen M Kurtz 2009. The APPD randomized controlled trial in low risk supervision: The effects on low risk supervision on rearrest. Philadelphia Adult Probation and Parole Department (2009).
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner 2016. Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. ProPublica (5 2016 ). Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner 2016. Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. ProPublica (5 2016).
- Fisher v. University of Texas at Austin . 2016 . Vol. 136 S. Ct . 2198, 2208 (2016). Fisher v. University of Texas at Austin. 2016. Vol. 136 S. Ct. 2198, 2208 (2016).
- Eric Holder . 2014. ( 2014 ). Remarks at the National Association of Criminal Defense Lawyers 57th Annual Meeting . Eric Holder. 2014. (2014). Remarks at the National Association of Criminal Defense Lawyers 57th Annual Meeting.
- Shahin Jabbari , Matthew Joseph , Michael Kearns , Jamie Morgenstern , and Aaron Roth . 2016. Fair Learning in Markovian Environments. arXiv preprint arXiv:1611.03071 ( 2016 ). Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. 2016. Fair Learning in Markovian Environments. arXiv preprint arXiv:1611.03071 (2016). · arXiv
- Jeff Larson , Surya Mattu , Lauren Kirchner , and Julia Angwin . 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (5 2016 ). Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (5 2016).
- Indré Žliobaité 2017. Measuring discrimination in algorithmic decision making Data Mining and Knowledge Discovery ( 2017 ). Indré Žliobaité 2017. Measuring discrimination in algorithmic decision making Data Mining and Knowledge Discovery (2017).
Cited by
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
Algorithms at Work: The New Contested Terrain of Control
Kellogg, K. C., Valentine, M. A., & Christin, A. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0174 · Cited by 2,732
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Fisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165
Metadata
- Authors
- Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A.
- Affiliations
- Stanford University, Stanford, CA, USA
- Stanford University (US)
- University of California, Berkeley (US)
- University of Chicago (US)
- Venue
- Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2017-08-04
- Pages
- 797-806
- OpenAlex
- W2584805976
- Cited by
- 1,149
- References
- 43
- Citations by year
- 2026: 62 · 2025: 113 · 2024: 141 · 2023: 173 · 2022: 146 · 2021: 162 · 2020: 150 · 2019: 119 · 2018: 67 · 2017: 13
- FWCI
- 238.91
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- John S. and James L. Knight Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
- Keywords
- algorithmic fairness, risk assessment algorithm, racial disparities, public safety, constrained optimization, risk thresholds, equal treatment, Broward County
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.