Against Predictive Optimization: On the Legitimacy of Decision-making Algorithms That Optimize Predictive Accuracy
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We formalize predictive optimization, a category of decision-making algorithms that use machine learning (ML)to predict future outcomes of interest about individuals. For example, pre-trial risk prediction algorithms such as COMPAS use ML to predict whether an individual will re-offend in the future. Our thesis is that predictive optimization raises a distinctive and serious set of normative concerns that cause it to fail on its own terms. To test this, we review 387 reports, articles, and web pages from academia, industry, non-profits, governments, and data science contests, and we find many real-world examples of predictive optimization. We select eight particularly consequential examples as case studies. Simultaneously, we develop a set of normative and technical critiques that challenge the claims made by the developers of these applications—in particular, claims of increased accuracy, efficiency, and fairness. Our key finding is that these critiques apply to each of the applications, are not easily evaded by redesigning the systems, and thus challenge whether these applications should be deployed. We argue that the burden of evidence for justifying why the deployment of predictive optimization is not harmful should rest with the developers of the tools. Based on our analysis, we provide a rubric of critical questions that can be used to deliberate or contest specific predictive optimization applications.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“To check if an application suffers from a limitation, we synthesize past literature and, in some cases, augment this with our own analysis.”
ResultDissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Additionally, nuisance calls by disgrun-tled neighbors, family members, or other potentially adversarial associates can negatively impact ASFT scores [72].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
The Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422
“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298 · cited 1×
“Human judgment is noisy and hence inaccurate [55], costly, biased, and arbitrary in the sense that it does not require even the fuzzy articulation of goals and values that the bureaucratic rule-making process requires.”
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 1×
“A canonical example is the cobra effect : when the colonial British government offered bounties for dead cobras to reduce the cobra population, the response instead was people breeding more cobras to kill.”
BackgroundAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 1×
“Human decision-making is messy, but one upside is that it combines decision-making with deliberation about bias and values.”
BackgroundRotundo, M., & Sackett, P. R. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.1.66 · Cited by 1,545 · cited 2×
“(1) Intervention versus prediction: Formulating pre-trial risk as a prediction problem does not do anything to increase our understanding of the underlying phenomenon, nor does it help us discover better interventions.”
BackgroundInfluentialAmazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · cited 2×
“Many decision-making systems were shut down once their actual deployed performance was revealed publicly by researchers or journalists [141, 186, 225].”
BackgroundMethodInfluentialAssessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 957
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 1×
“A canonical example is the cobra effect : when the colonial British government offered bounties for dead cobras to reduce the cobra population, the response instead was people breeding more cobras to kill.”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Confident Learning: Estimating Uncertainty in Dataset Labels
Northcutt, C., Jiang, L., & Chuang, I. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12125 · Cited by 795
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015) · American Economic Review · doi:10.1257/aer.p20151023 · Cited by 584 · cited 3×
“Prediction versus intervention: Kleinberg et al. [118] predict which patients are at risk of death 1–12 months after surgery.”
BackgroundMethodInfluentialBeyond prediction: using big data for policy problems
Athey, S. (2017) · Science · doi:10.1126/science.aal4321 · Cited by 575 · cited 1×
“While there are methods being developed to use machine learning to model this gap through techniques like causal modeling [12, 48, 51, 93, 123, 131, 204], they would require additional assumptions compared to off-the-shelf ML tools and/or additional data collection.”
MethodMethods in predictive techniques for mental health status on social media: a critical review
Chancellor, S., & De Choudhury, M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0233-7 · Cited by 557 · cited 2×
“Peer-reviewed research on suicide prediction is based on murky evidence and suffers from a lack of construct validity [35].”
BackgroundMethodRacial/Ethnic Differences in Physician Distrust in the United States
Armstrong, K., Ravenell, K. L., McMurphy, S., & Putt, M. (2007) · American Journal of Public Health · doi:10.2105/ajph.2005.080762 · Cited by 517 · cited 1×
“Additionally, nuisance calls by disgrun-tled neighbors, family members, or other potentially adversarial associates can negatively impact ASFT scores [72].”
BackgroundMitigating Gender Bias in Natural Language Processing: Literature Review
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1159 · Cited by 492
Tuskegee and the Health of Black Men*
Alsan, M., & Wanamaker, M. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx029 · Cited by 464 · cited 1×
“Additionally, nuisance calls by disgrun-tled neighbors, family members, or other potentially adversarial associates can negatively impact ASFT scores [72].”
BackgroundThe Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Zarsky, T. (2016) · Science, Technology, & Human Values · doi:10.1177/0162243915605575 · Cited by 422
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362 · cited 2×
“without Machine Learning Predictive systems can only meet their goals, such as minimizing crime or hiring good employees, to the extent that their predictions are accurate.”
BackgroundMeasuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322
Self-citationAdministrative Records Mask Racially Biased Policing
KNOX, D., LOWE, W., & MUMMOLO, J. (2020) · American Political Science Review · doi:10.1017/s0003055420000039 · Cited by 314
Administrative Records Mask Racially Biased Policing
Knox, D., Lowe, W., & Mummolo, J. (2019) · doi:10.2139/ssrn.3336338 · Cited by 314
Problem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283 · cited 1×
“For example, standardized tests such as the SAT and the GRE are administered by private companies but those companies are monopolies—there is usually no alternative to these tests for university applications.”
BackgroundSelf-citationAlgorithmic recourse: From counterfactual explanations to interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445899 · Cited by 274
Participation Is not a Design Fix for Machine Learning
Sloane, M., Moss, E., Awomolo, O., & Forlano, L. (2022) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3551624.3555285 · Cited by 267
The flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263 · cited 1×
“However, this itself is not without flaws, as human decision makers often adhere to or deviate from machine predictions in non-random ways [36, 90, 200].”
BackgroundThe Flaws of Policies Requiring Human Oversight of Government Algorithms
Green, B. (2021) · doi:10.2139/ssrn.3921216 · Cited by 259
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
The age of secrecy and unfairness in recidivism prediction
Rudin, C., Wang, C., & Coker, B. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.6ed64b30 · Cited by 177
Does Machine Learning Automate Moral Hazard and Error?
Mullainathan, S., & Obermeyer, Z. (2017) · American Economic Review · doi:10.1257/aer.p20171084 · Cited by 156
Layers of bias: A unified approach for understanding problems with risk assessment
Eckhouse, L., Lum, K., Conti-Cook, C., & Ciccolini, J. (2019) · Criminal Justice and Behavior · doi:10.1177/0093854818811379 · Cited by 156 · cited 1×
“We follow and build on many efforts to critique and resist the deployment of automated decision-making systems—including those from academia [29, 39, 68, 143, 198, 206, 218], non-profits [85, 149], and community organizations [1, 41, 164].”
BackgroundThe Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098066 · Cited by 138
Dropping out of school: A review of intervention programs
Prevatt, F., & Kelly, F. D. (2003) · Journal of School Psychology · doi:10.1016/s0022-4405(03)00087-6 · Cited by 136
Equitably Allocating Resources during Crises: Racial Differences in Mortality Prediction Models
Ashana, D. C., Anesi, G. L., Liu, V. X., Escobar, G. J., Chesley, C., Eneanya, N. D., Weissman, G. E., Miller, W. D., Harhay, M. O., & Halpern, S. D. (2021) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202012-4383oc · Cited by 98 · cited 1×
“Additionally, nuisance calls by disgrun-tled neighbors, family members, or other potentially adversarial associates can negatively impact ASFT scores [72].”
BackgroundInterventions over Predictions: Reframing the Ethical Debate for Actuarial Risk Assessment
Barabas, C., Dinakar, K., Ito, J., Virza, M., & Zittrain, J. (2017) · arXiv · doi:10.48550/arxiv.1712.08238 · Cited by 87
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82 · cited 1×
“We follow and build on many efforts to critique and resist the deployment of automated decision-making systems—including those from academia [29, 39, 68, 143, 198, 206, 218], non-profits [85, 149], and community organizations [1, 41, 164].”
BackgroundA Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2019) · arXiv · doi:10.48550/arxiv.1901.10002 · Cited by 82
Algorithmic Recourse: from Counterfactual Explanations to Interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2020) · arXiv · doi:10.48550/arxiv.2002.06278 · Cited by 76
Procurement As Policy: Administrative Process for Machine Learning
Mulligan, D. K., & Bamberger, K. A. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3464203 · Cited by 75 · cited 1×
“For example, standardized tests such as the SAT and the GRE are administered by private companies but those companies are monopolies—there is usually no alternative to these tests for university applications.”
BackgroundCategorizing Variants of Goodhart's Law
Manheim, D., & Garrabrant, S. (2018) · arXiv · doi:10.48550/arxiv.1803.04585 · Cited by 73
Creel, K., & Hellman, D. (2022) · Canadian Journal of Philosophy · doi:10.1017/can.2022.3 · Cited by 67
Algorithmic Risk Assessment in the Hands of Humans
Stevenson, M., & Doleac, J. L. (2019) · doi:10.2139/ssrn.3489440 · Cited by 60
Constructing Recidivism Risk for Sentencing
Eaglin, J. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2821136 · Cited by 45
'Improving ratings': audit in the British university system
Strathern, M. (1997) · European Review · doi:10.1002/(sici)1234-981x(199707)5:3<305::aid-euro184>3.0.co;2-4 · cited 1×
“There have been long debates about defining legitimacy, especially in political philosophy [5, 176, 221].”
Background
Cited by
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
Fairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey
Fabris, A., Baranowska, N., Dennis, M. J., Graus, D., Hacker, P., Saldivar, J., Zuiderveen Borgesius, F., & Biega, A. J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3696457 · Cited by 117
Ecosystem Graphs: The Social Footprint of Foundation Models
Bommasani, R., Soylu, D., Liao, T., Creel, K., & Liang, P. (2023) · doi:10.21203/rs.3.rs-2961271/v1 · Cited by 14
Gules‐Guctas, E. (2026) · Public Administration Review · doi:10.1111/puar.70043 · Cited by 7
“While correlational analyses can identify statistical relationships between variables, many systems apply these associations to causal decision-making procedures, despite limited scientific evidence supporting such an approach (Wang et al. 2024).”
Perceptions of the Fairness Impacts of Multiplicity in Machine Learning
Meyer, A. P., Kim, Y.-S., D'Antoni, L., & Albarghouthi, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713524 · Cited by 1
Metadata
- Authors
- Wang, A., Kapoor, S., Barocas, S., & Narayanan, A.
- Venue
- ACM Journal on Responsible Computing
- Published
- 2024-03-20
- Volume
- 1
- Issue
- 1
- Pages
- 1-45
- ISSN
- 2832-0565 (ISSN-L)
- OpenAlex
- W4389676498
- Cited by
- 57
- References
- 53
- FWCI
- 13.92
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.