Algorithmic Risk Assessment in the Hands of Humans
Open accessAbstract
We evaluate the impacts of adopting algorithmic risk assessments in sentencing. We find that judges changed sentencing practices in response to the risk assessment, but that discretion played a large role in mediating its impact. Judges deviated from the recommendations associated with the algorithm in systematic ways, suggestive of alternative objectives at sentencing. As a result, risk assessment did not lead to detectable gains in terms of public safety or reduced incarceration rates. Using simulations, we show that strict adherence to the sentencing recommendations associated with the algorithm would have led to some benefits (a sharp reduction in incarceration) but also some costs (a slight increase in recidivism and an increase in relative sentences for the young). Discretion mitigated the costs at the expense of reducing the benefits.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Assessment tools used to predict recidivism are now widespread in most jurisdictions.
“Assessment tools used to predict recidivism are now widespread in most jurisdictions: bail and sentencing judges (Stevenson and Doleac, 2019; Albright, 2019), parole board members (Berk, 2017), and probation officers (Kopf, 2014) use the assessment scores to determine the level of risk.”
From Prison Rehabilitation Programs: Efficiency and Targeting (2021) · cites this paper as Stevenson and Doleac, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 34 · Cited by 60
Research works printed but not linked: 24
- Agan, Amanda and Sonja Starr , 'Ban the Box, Criminal Records, and Racial Discrimination: A Field Experiment*,' The Quarterly Journal of Economics , 08 2017, 133 (1), 191-235.
- Arnold, David, Will Dobbie, and Crystal S Yang , 'Racial Bias in Bail Decisions*,' The Quarterly Journal of Economics , 05 2018, 133 (4), 1885-1932.
- Bayer, Patrick, Randi Hjalmarsson, and David Pozen , 'Building Criminal Capital behind Bars: Peer Effects in Juvenile Corrections*,' The Quarterly Journal of Economics , 02 2009, 124 (1), 105-147.
- Berdej, Carlos and Noam Yuchtman , 'Crime, Punishment, and Politics: An Analysis of Political Cycles in Criminal Sentencing,' The Review of Economics and Statistics , 2013, 95 (3), 741-756.
- Berk, Richard A., Susan B. Sorenson, and Geoffrey Barnes , 'Forecasting Domestic Violence: A Machine Learning Approach to Help Inform Arraignment Decisions,' Journal of Empirical Legal Studies , 2016, 13 (1), 94-115.
- Berk, Richard, Heather Ladd, Heidi Graziano, and Jong-Ho Baek , 'A Randomized Experiment Testing Inmate Classification Systems Inmate Classification,' Criminology & Public Policy , 2002, 2 , 215.
- Bushway, Shawn and Jeffrey Smith , 'Sentencing Using Statistical Treatment Rules: What We Don't Know Can Hurt Us,' Journal of Quantitative Criminology , Dec 2007, 23 (4), 377-387.
- Calonico, Sebastian, Matias D. Cattaneo, and Rocio Titiunik , 'Robust Nonparametric Confidence Intervals for Regression-Discontinuity Designs,' Econometrica , 2014, 82 (6), 2295-2326.
- Chalfin, Aaron, Oren Danieli, Andrew Hillis, Zubin Jelveh, Michael Luca, Jens Ludwig, and Sendhil Mullainathan , 'Productivity and Selection of Human Capital with Machine Learning,' American Economic Review , May 2016, 106 (5), 124-27.
- , 'The Impact of Algorithms on Judicial Discretion: Evidence from Regression Discontinuities,' Working Paper 2019. and Catherine Tucker , 'Algorithmic Bias: A Counterfactual Perspective,' Working Paper: NSF Trustworthy Algorithms December 2017. and , 'Economics, Fairness, and Algorithmic Bias,' In preparation for the Journal of Economics Perspectives March 2019. and Megan Stevenson , 'Algorithmic Social Engineering,' Working Paper 2019.
- Garrett, Brandon L., Alexander Jakubow, and John Monahan , 'Judicial Reliance on Risk Assessment in Sentencing Drug and Property Offenders: A Test of the Treatment Resource Hypothesis,' Criminal Justice and Behavior , 2019, 46 (6), 799-810. and John Monahan , 'Judging Risk,' California Law Review , 2018, forthcoming. and , 'Judging Risk,' Virginia Public Law and Legal Theory Research Paper Series 2018-27 May 2018.
- Gelbach, Jonah B. , 'When Do Covariates Matter? And Which Ones, and How Much?,' Journal of Labor Economics , 2016, 34 (2), 509-543.
- Glaeser, Edward L., Andrew Hillis, Scott Duke Kominers, and Michael Luca , 'Crowdsourcing City Government: Using Tournaments to Improve Inspection Accuracy,' American Economic Review , May 2016, 106 (5), 114-18.
- Goel, Sharad, Justin M. Rao, and Ravi Shroff , 'Personalized Risk Assessments in the Criminal Justice System,' American Economic Review , May 2016, 106 (5), 119-23.
- Jung, J., C. Concannon, R. Shroff, S. Goel, and D. G. Goldstein , 'Simple rules for complex decisions,' ArXiv e-prints , February 2017.
- Kleinberg, Jon, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan , 'Human Decisions and Machine Predictions*,' The Quarterly Journal of Economics , 2018, 133 (1), 237-293.
- , Jens Ludwig, Sendhil Mullainathan, and Cass R Sunstein , 'Discrimination in the Age of Algorithms,' Journal of Legal Analysis , 04 2019, 10 , 113-174.
- Kling, Jeffrey R. , 'Incarceration Length, Employment, and Earnings,' American Economic Review , June 2006, 96 (3), 863-876.
- Lim, Claire S. H., Jr. Snyder James M., and David Strmberg , 'The Judge, the Politician, and the Press: Newspaper Coverage and Criminal Sentencing across Electoral Systems,' American Economic Journal: Applied Economics , October 2015, 7 (4), 103-35.
- Loeffler, Charles E. , 'Does Imprisonment Alter the Life Course? Evidence on Crime and Employment From a Natural Experiment,' Criminology , 2013, 51 (1), 137-166.
- Mayson, Sandra , 'Bias In, Bias Out,' Yale Law Journal , June 2019, 128 (8).
- Starr, Sonja , 'Evidence-Based Sentencing and the Scientific Rationalization of Discrimination,' Stanford Law Review , 2014, 66.
- Stevenson, Megan , 'Breaking Bad: Mechanisms of Social Influence and the Path to Criminality in Juvenile Jails,' The Review of Economics and Statistics , 2017, 99 (5), 824-838.
- , 'Assessing Risk Assessment in Action,' Minnesota Law Review , 2019, 103. and Christopher Slobogin , 'Algoirithmic Risk Assessment and the Double Edged Sword of Youth,' Washington University Law Review , 2018, 96.
- Authors
- Stevenson, M., & Doleac, J. L.
- Affiliations
- University of Virginia (US)
- Arnold Ventures (US)
- Corresponding author
- Stevenson, Megan
- ORCID
- Stevenson, Megan
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2019
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W2990458283
- Cited by
- 60
- References
- 34
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 8 · 2023: 11 · 2022: 12 · 2021: 14 · 2020: 5 · 2019: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Econstor (Econstor) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Keywords
- algorithmic risk assessment, sentencing, judicial discretion, algorithmic decision-making, recidivism, incarceration rates, public safety, algorithmic bias, sentencing recommendations, juvenile offenders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
