Algorithmic Recommendations and Human Discretion
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Human decision-makers frequently override the recommendations generated by predictive algorithms, but it is unclear whether these discretionary overrides add valuable private information or reintroduce human biases and mistakes.We develop new quasi-experimental tools to measure the impact of human discretion over an algorithm on the accuracy of decisions, even when the outcome of interest is only selectively observed, in the context of bail decisions.We find that 90% of the judges in our setting underperform the algorithm when they make a discretionary override, with most making override decisions that are no better than random.Yet the remaining 10% of judges outperform the algorithm in terms of both accuracy and fairness when they make a discretionary override.We provide suggestive evidence on the behavior underlying these differences in judge performance, showing that the high-performing judges are more likely to use relevant private information and are less likely to overreact to highly salient events compared to the low-performing judges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Structural Equations, Treatment Effects, and Econometric Policy Evaluation1
Heckman, J. J., & Vytlacil, E. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00594.x · Cited by 1,220
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Dobbie, W., Goldin, J., & Yang, C. S. (2018) · American Economic Review · doi:10.1257/aer.20161503 · Cited by 564
Self-citationHuman Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Assessing Risk Assessment in Action
Stevenson, M. (2018) · Minnesota Law Review · doi:10.24926/265535.958 · Cited by 193
An impact assessment of machine learning risk forecasts on parole board decisions and recidivism
Berk, R. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9286-2 · Cited by 169
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.2139/ssrn.4505053 · Cited by 161
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.3386/w31422 · Cited by 161
Incidence and Costs of Personal and Property Crimes in the USA, 2017
Miller, T. R., Cohen, M. A., Swedler, D. I., Ali, B., & Hendrie, D. V. (2021) · Journal of Benefit-Cost Analysis · doi:10.1017/bca.2020.36 · Cited by 73
Governing by Algorithm? No Noise and (Potentially) Less Bias
Sunstein, C. R. (2021) · doi:10.2139/ssrn.3925240 · Cited by 52
Algorithmic Risk Assessment in the Hands of Humans
Stevenson, M., & Doleac, J. L. (2020) · doi:10.2139/ssrn.3513695 · Cited by 42
Does Cash Bail Deter Misconduct?
Ouss, A., & Stevenson, M. (2023) · American Economic Journal: Applied Economics · doi:10.1257/app.20210349 · Cited by 28
Cited by
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.2139/ssrn.4505053 · Cited by 161
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.3386/w31422 · Cited by 161
Voice AI in Firms: A Natural Field Experiment on Automated Job Interviews
Jabarian, B., & Henkel, L. (2025) · doi:10.2139/ssrn.5395709 · Cited by 9
Metadata
- Authors
- Angelova, V., Dobbie, W., & Yang, C.
- Venue
- National Bureau of Economic Research
- Publisher
- National Bureau of Economic Research
- Published
- 2023-09
- OpenAlex
- W4387246959
- Cited by
- 43
- References
- 13
- Type
- report
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.