The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Algorithmic Risk Assessment in the Hands of Humans
Abstract
We evaluate the impacts of adopting algorithmic risk assessments in sentencing. We find that judges changed sentencing practices in response to the risk assessment, but that discretion played a large role in mediating its impact. Judges deviated from the recommendations associated with the algorithm in systematic ways, suggestive of alternative objectives. As a result, risk assessment did not lead to detectable gains in terms of public safety or reduced incarceration rates. Using simulations, we show that strict adherence to the sentencing recommendations associated with the algorithm would have had benefits (less incarceration) but also some costs (increased sentences for youth). (JEL D81, D91, H76, K41, K42)
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In many municipalities, there is no guarantee that individuals with the same risk score will have the same outcome.
“In many municipalities, there is no guarantee that individuals with the same risk score will have the same outcome [35].”
From Efficiency vs. Equity: A Structured Interdisciplinary Review of AI in Criminal Justice Risk Assessments (2026) · cites this paper as [35]
Observations of sentencing rates in Virginia have shown that their own use of risk assessments as a sentencing tool in the 2000s resulted in longer sentences for black and young defendants.
“Observations of sentencing rates in Virginia have shown that their own use of risk assessments as a sentencing tool in the 2000s resulted in longer sentences for black and young defendants [35], a fact which was documented by the Virginia Criminal Sentencing Commission.”
From Efficiency vs. Equity: A Structured Interdisciplinary Review of AI in Criminal Justice Risk Assessments (2026) · cites this paper as [35]
COMPAS recommended 24% longer sentences for young defendants.
“These analyses uncovered that COMPAS recommended 24% longer sentences for young defendants [35] and that COMPAS was more than twice as likely to incorrectly identify a black defendant as a future criminal as compared to a white defendant [58] although Northpoint, the company that developed COMPAS, has disputed claims of bias in COMPAS [58].”
From Efficiency vs. Equity: A Structured Interdisciplinary Review of AI in Criminal Justice Risk Assessments (2026) · cites this paper as [35]
Citations
Cites
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014) · American Economic Review · doi:10.1257/aer.104.9.2633 · Cited by 1,717
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Doleac, J. L., & Hansen, B. (2020) · Journal of Labor Economics · doi:10.1086/705880 · Cited by 233
Self-citationAn impact assessment of machine learning risk forecasts on parole board decisions and recidivism
Berk, R. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9286-2 · Cited by 169
Johndrow, J. E., & Lum, K. (2019) · The Annals of Applied Statistics · doi:10.1214/18-aoas1201 · Cited by 141
Breaking bad: Mechanisms of social influence and the path to criminality in juvenile jails
Stevenson, M. (2017) · Review of Economics and Statistics · doi:10.1162/rest_a_00685 · Cited by 84
Self-citationImpact of risk assessment on judges’ fairness in sentencing relatively poor defendants
Skeem, J., Scurich, N., & Monahan, J. (2020) · Law and Human Behavior · doi:10.1037/lhb0000360 · Cited by 70
Personalized Risk Assessments in the Criminal Justice System
Goel, S., Rao, J. M., & Shroff, R. (2016) · American Economic Review · doi:10.1257/aer.p20161028 · Cited by 27
Flagged first, then by citations
Research works printed but not linked: 32
- Abaluck, Jason, Leila Agha, David C. Chan Jr., Daniel Singer, and Diana Zhu. 2020. "Fixing Misallocation with Guidelines: Awareness versus Adherence." NBER Working Paper 27467. · DOI
- 10.1093/qje/qjx028 · DOI
- 10.1093/qje/qjv003 · DOI · Paper page
- Albright, Alex. 2023. "The Hidden Effects of Algorithmic Recommendations." Unpublished. · DOI
- Angelino, Elaine, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin. 2018 "Learning Certi ably Optimal Rule Lists for Categorical Data." arXiv:1704.01701. · DOI
- 10.1257/pandp.20211080 · DOI
- 10.1162/qjec.2009.124.1.105 · DOI · Paper page
- 10.1111/j.1745-9133.2003.tb00121.x · DOI
- 10.1111/jels.12098 · DOI
- 10.1002/bdm.2155 · DOI · Paper page
- 10.3982/ECTA11757 · DOI · Paper page
- 10.1257/aer.p20161029 · DOI
- 10.1257/pandp.20201037 · DOI
- 10.1017/S0003055418000783 · DOI
- 10.1177/0093854819842589 · DOI
- Garrett Brandon L, 2020, California Law Review, 108, 439
- 10.1037/1076-8971.2.2.293 · DOI · Paper page
- Gruber, Jonathan, Benjamin R. Handel, Samuel H. Kina, and Jonathan T. Kolstad. 2020 "Managing Intelligence: Skilled Experts and AI in Markets for Complex Products." NBER Working Paper 27038. · DOI
- 10.1093/qje/qjx042 · DOI · Paper page
- Imai, Kosuke, Zhichao Jiang, James Greiner, Ryan Halen, and Sooahn Shin. 2020. "Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment." arXiv:2012.02845. · arXiv
- Jung, Jongbin, Connor Concannon, Ravi Shroff, Sharad Goel, and Daniel G. Goldstein. 2017. "Simple Rules for Complex Decisions." arXiv:1702:04690. · DOI
- Kasy, Maximilian, and Rediet Abebe. 2021. "Fairness, Equality, and Power in Algorithmic Decision-Making." In FAccT '21, Proceedings of the 2021, ACM Conference on Fairness, Accountability, and Transparency, 576-586. New York: Association for Computing Machinery. · DOI
- Kleinberg Jon, 2018, Quarterly Journal of Economics, 133, 237
- Lin, Zhiyuan, Jongbin Jung, Sharad Goel, and Jennifer Skeem. 2020. "The Limits of Human Predictions of Recidivism." Science Advances 6 (7): eaaz0652. · DOI
- Lipton, Zachary C., Alexandra Chouldechova, and Julian McAuley. 2017. "Does Mitigating ML's Impact Disparity Require Treatment Disparity?" arXiv:1711:07076.
- 10.1256/003590002320603584 · DOI
- 10.1002/jcop.22404 · DOI
- Sloan CarlyWill, 2023, Journal of Human Resources. https://doi. org/10.3368/jhr.0221-11470R3. · DOI
- Stevenson Megan, 2019, Minnesota Law Review, 103, 303
- Stevenson Megan, 2018, Washington University Law Review, 96, 681
- Stevenson, Megan T., and Doleac, Jennifer L. 2024. Data and Code for: "Algorithmic Risk Assessment in the Hands of Humans." Nashville, TN: American Economic Association; distributed by Inter-university Consortium for Political and Social Research, Ann Arbor, MI. https://doi.org/10.3886/ E193667V1.
- 10.36644/mlr.119.2.equal · DOI
Cited by
Sorted by citations
Argudo Palacios, A., & Mechraoui, A. (2026) · Cuadernos de RES PUBLICA en derecho y criminología · doi:10.46661/respublica.13244
Limits of Interventions in AI-Assisted Decision Making
Shrestha, S., Ahn, E., & Vorobeychik, Y. (2026) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v9i4.43309
Atkinson, G., & Casagrande, K. (2026) · Information · doi:10.3390/info17060574
Metadata
- Authors
- Stevenson, M. T., & Doleac, J. L.
- Affiliations
- University of Virginia (email: )
- Arnold Ventures (email: )
- Venue
- American Economic Journal: Economic Policy
- Publisher
- American Economic Association
- Published
- 2024-11-01
- Volume
- 16
- Issue
- 4
- Pages
- 382-414
- ISSN
- 1945-7731
- 1945-774X
- Cited by
- 21
- References
- 9
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
