Human Decisions and Machine Predictions
Open accessAbstract
Can machine learning improve human decision making? Bail decisions provide a good test case. Millions of times each year, judges make jail-or-release decisions that hinge on a prediction of what a defendant would do if released. The concreteness of the prediction task combined with the volume of data available makes this a promising machine-learning application. Yet comparing the algorithm to judges proves complicated. First, the available data are generated by prior judge decisions. We only observe crime outcomes for released defendants, not for those judges detained. This makes it hard to evaluate counterfactual decision rules based on algorithmic predictions. Second, judges may have a broader set of preferences than the variable the algorithm predicts; for instance, judges may care specifically about violent crimes or about racial inequities. We deal with these problems using different econometric strategies, such as quasi-random assignment of cases to judges. Even accounting for these concerns, our results suggest potentially large welfare gains: one policy simulation shows crime reductions up to 24.7% with no change in jailing rates, or jailing rate reductions up to 41.9% with no increase in crime rates. Moreover, all categories of crime, including violent crimes, show reductions; these gains can be achieved while simultaneously reducing racial disparities. These results suggest that while machine learning can be valuable, realizing this value requires integrating these tools into an economic framework: being clear about the link between predictions and decisions; specifying the scope of payoff functions; and constructing unbiased decision counterfactuals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evidence suggests that algorithms make policy-relevant predictions more accurately, fairly, and consistently than public servants.
“Evidence suggests that algorithms make policy-relevant predictions more accurately, fairly, and consistently than public servants ( Kleinberg et al., 2018 ;Kleinberg et al., 2015 ).”
From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as Kleinberg et al., 2018
The Introduction reports that an algorithmic release rule could improve on judges’ decisions while reducing racial disparities. Section VI.C attributes part of the gap in outcomes to inconsistency in judges’ decisions. The evidence concerns judges making pretrial-release decisions, not public servants or policy-relevant predictions generally, so the claim extends beyond the setting studied. (LLM, conf=0.87)
Experimental — accuracy not yet measuredTools that help developers pro-ethically design solutions that do not overly restrict the user's options in acting on predictions (i.e., tools that promote user autonomy) are in short supply.
“As such, tools that, for example, help developers pro-ethically design solutions that do not overly restrict the user’s options in acting on this prediction (i.e. tools that promote the user’s autonomy) are in short supply (Kleinberg et al. 2017).”
From From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices (2020) · cites this paper as Kleinberg et al. 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · see citing works
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650 · see citing works
(2012) · Cited by 9,062 · see citing works
Do Stock Prices Move Too Much to be Justified by Subsequent Changes in Dividends?
Shiller, R. J. (1980) · National Bureau of Economic Research · doi:10.3386/w0456 · Cited by 4,129 · see citing works
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · see citing works
“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298 · see citing works
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,187 · see citing works
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061 · see citing works
Machine Learning: An Applied Econometric Approach
Mullainathan, S., & Spiess, J. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.87 · Cited by 1,989 · see citing works
Self-citationThe robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,970 · see citing works
Black, F. (1986) · The Journal of Finance · doi:10.1111/j.1540-6261.1986.tb04513.x · Cited by 1,846 · see citing works
Clinical versus mechanical prediction: a meta-analysis
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000) · Psychological Assessment · doi:10.1037/1040-3590.12.1.19 · Cited by 1,640 · see citing works
Big Data: New Tricks for Econometrics
Varian, H. R. (2014) · The Journal of Economic Perspectives · doi:10.1257/jep.28.2.3 · Cited by 1,588 · see citing works
Recursive partitioning for heterogeneous causal effects
Athey, S., & Imbens, G. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510489113 · Cited by 1,586 · see citing works
Salience Theory of Choice Under Risk
Bordalo, P., Gennaioli, N., & Shleifer, A. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qjs018 · Cited by 1,310 · see citing works
Rosenbaum, P., & Rubin, D. (1983) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1983.tb01242.x · Cited by 1,264 · see citing works
Scaling to Very Very Large Corpora for Natural Language Disambiguation
Banko, M., & Brill, E. (2001) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/1073012.1073017 · Cited by 802 · see citing works
High-Dimensional Methods and Inference on Structural and Treatment Effects
Belloni, A., Chernozhukov, V., & Hansen, C. (2014) · Journal of Economic Perspectives · doi:10.1257/jep.28.2.29 · Cited by 722 · see citing works
Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015) · American Economic Review · doi:10.1257/aer.p20151023 · Cited by 584 · see citing works
Self-citationA case study of graduate admissions: Application of three principles of human decision making.
Dawes, R. M. (1971) · American Psychologist · doi:10.1037/h0030868 · Cited by 578 · see citing works
Incarceration length, employment, and earnings
Kling, J. R. (2006) · American Economic Review · doi:10.1257/aer.96.3.863 · Cited by 563 · see citing works
Juvenile incarceration, human capital, and future crime: Evidence from randomly assigned judges
Aizer, A., & Doyle, J. J. (2015) · The Quarterly Journal of Economics · doi:10.1093/qje/qjv003 · Cited by 541 · see citing works
Jacob, B. A., & Lefgren, L. (2008) · Journal of Labor Economics · doi:10.1086/522974 · Cited by 494 · see citing works
Interpretable classification models for recidivism prediction
Zeng, J., Ustun, B., & Rudin, C. (2017) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12227 · Cited by 220 · see citing works
Harcourt, B. E. (2015) · Federal Sentencing Reporter · doi:10.1525/fsr.2015.27.4.237 · Cited by 210 · see citing works
Criminal Recidivism after Prison and Electronic Monitoring
Tella, R. D., & Schargrodsky, E. (2013) · Journal of Political Economy · doi:10.1086/669786 · Cited by 200 · see citing works
Productivity and Selection of Human Capital with Machine Learning
Chalfin, A. J., Danieli, O., Hillis, A., Jelveh, Z., Luca, M., Ludwig, J., & Mullainathan, S. (2016) · American Economic Review · doi:10.1257/aer.p20161029 · Cited by 190 · see citing works
Self-citationTan, C., Lee, L., & Pang, B. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1017 · Cited by 188 · see citing works
An impact assessment of machine learning risk forecasts on parole board decisions and recidivism
Berk, R. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9286-2 · Cited by 169 · see citing works
NOISE: How to overcome the high, hidden cost of inconsistent decision making
Kahneman, D. J., Rosenfield, A. M., Gandhi, L., & Blaser, T. (2016) · Harvard business review · Cited by 156 · see citing works
Criminal Justice Forecasts of Risk
Berk, R. A. (2012) · SpringerBriefs in computer science · doi:10.1007/978-1-4614-3085-8 · Cited by 143 · see citing works
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098066 · Cited by 110 · see citing works
Self-citationMendel, B., & Shleifer, A. (2011) · Journal of Financial Economics · doi:10.1016/j.jfineco.2011.02.018 · Cited by 109 · see citing works
The Determinants of Productivity in Medical Testing: Intensity and Allocation of Care
Abaluck, J., Agha, L., Kabrhel, C., Raja, A. S., & Venkatesh, A. K. (2016) · American Economic Review · doi:10.1257/aer.20140260 · Cited by 94 · see citing works
Simple Rules for Complex DecisionsPreprint
Jung, J., Concannon, C., Shroff, R., Goel, S., & Goldstein, D. G. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2919024 · Cited by 91 · see citing works
Distortion of Justice: How the Inability to Pay Bail Affects Case OutcomesPreprint
Stevenson, M. T. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2777615 · Cited by 75 · see citing works
Learning Cost-Effective and Interpretable Treatment Regimes
Lakkaraju, H., & Rudin, C. D. (2017) · International Conference on Artificial Intelligence and Statistics · Cited by 61 · see citing works
Self-citationAlgorithm Appreciation: People Prefer Algorithmic To Human JudgmentPreprint
Logg, J. M., Minson, J., & Moore, D. A. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2941774 · Cited by 59 · see citing works
Incarceration, Recidivism and Employment
Bhuller, M., Dahl, G. B., Løken, K. V., & Mogstad, M. (2016) · National Bureau of Economic Research · doi:10.3386/w22648 · Cited by 59 · see citing works
What Does Performance in Graduate School Predict? Graduate Economics Education and Student Outcomes
Athey, S. C., Katz, L. F., Krueger, A. B., et al. (2007) · American Economic Review · doi:10.1257/aer.97.2.512 · Cited by 57 · see citing works
The Efficiency of Prediction in Criminology
Ohlin, L. E., & Duncan, O. D. (1949) · American Journal of Sociology · doi:10.1086/220398 · Cited by 45 · see citing works
OPTIMAL BAIL AND THE VALUE OF FREEDOM: EVIDENCE FROM THE PHILADELPHIA BAIL EXPERIMENT
Abrams, D., & Rohlfs, C. (2010) · Economic Inquiry · doi:10.1111/j.1465-7295.2010.00288.x · Cited by 44 · see citing works
Forecasts of Violence to Inform Sentencing Decisions
Berk, R. A., & Bleich, J. (2013) · Journal of Quantitative Criminology · doi:10.1007/s10940-013-9195-0 · Cited by 43 · see citing works
Policy Guidelines for Bail: An Experiment in Court Reform
Stojkovic, S., Goldkamp, J. S., & Gottfredson, M. R. (1987) · The Journal of Criminal Law and Criminology (1973-) · doi:10.2307/1143458 · Cited by 40 · see citing works
Sentencing Using Statistical Treatment Rules: What We Don’t Know Can Hurt Us
Bushway, S. D., & Smith, J. A. (2007) · Journal of Quantitative Criminology · doi:10.1007/s10940-007-9035-1 · Cited by 39 · see citing works
Dobbie, W., Goldin, J. S., & Yang, C. (2016) · National Bureau of Economic Research · doi:10.3386/w22511 · Cited by 28 · see citing works
A Bayesian Framework for Modeling Human Evaluations
Lakkaraju, H., Leskovec, J., Kleinberg, J. M., et al. (2015) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611974010.21 · Cited by 18 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Black Fischer Noise The Journal of Finance 41 1986 528 543
- Jacob Brian A Lefgren Lars Principals as agents: Subjective performance assessment in education Journal of Labor Economics 26 2008 101 136
- Kleinberg Jon Lakkaraju Himabindu Leskovec Jure Ludwig Jens Mullainathan Sendhil Replication Data for: ‘Human Decisions and Machine Predictions’ Harvard Dataverse 2017 10.7910/DVN/VWDGHT PMC5947971 29755141 · DOI
- Starr Sonja B Evidence-based sentencing and the scientific rationalization of discrimination Stanford Law Review 66 2014 803 872
- Toborg Mary A Pretrial release practices in the United States, 1976-1978 2nd ICPSR edition Ann Arbor, MI Inter-university Consortium on Political and Social Research ICPSR07972.v2 1997
Cited by
Sorted by citations
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
Self-citationResistance to Medical Artificial Intelligence
Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019) · Journal of Consumer Research · doi:10.1093/jcr/ucz013 · Cited by 2,015 · see citing works
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150 · see citing works
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497 · see citing works
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00165-5 · Cited by 882 · see citing works
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845 · see citing works
Self-citationFerrara, E. (2023) · Sci · doi:10.3390/sci6010003 · Cited by 875 · see citing works
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · see citing works
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607 · see citing works
Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015) · American Economic Review · doi:10.1257/aer.p20151023 · Cited by 584 · see citing works
Self-citationTowards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · see citing works
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · see citing works
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · see citing works
Measuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322 · see citing works
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · see citing works
Machine learning and human capital complementarities: Experimental evidence on bias mitigation
Choudhury, P., Starr, E., & Agarwal, R. (2020) · Strategic Management Journal · doi:10.1002/smj.3152 · Cited by 293 · see citing works
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · see citing works
Kleinberg, J., Ludwig, J., Mullainathan, S., & Rambachan, A. (2018) · AEA Papers and Proceedings · doi:10.1257/pandp.20181018 · Cited by 278 · see citing works
Self-citationProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307 · see citing works
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375833 · Cited by 241 · see citing works
Self-citationRace Discrimination: An Economic Perspective
Lang, K., & Spitzer, A. K.-L. (2020) · Journal of Economic Perspectives · doi:10.1257/jep.34.2.68 · Cited by 220 · see citing works
The flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263 · see citing works
Lassopack: model selection and prediction with regularized regression in Stata
Ahrens, A., Hansen, C. B., & Schaffer, M. E. (2020) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x20909697 · Cited by 212 · see citing works
A Taxonomy of Ethical Tensions in Inferring Mental Health States from Social Media
Chancellor, S., Birnbaum, M. L., Caine, E. D., Silenzio, V. M. B., & De Choudhury, M. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287587 · Cited by 214 · see citing works
Machine learning for environmental monitoring
Hino, M., Benami, E., & Brooks, N. (2018) · Nature Sustainability · doi:10.1038/s41893-018-0142-9 · Cited by 201 · see citing works
Janssen, M., Hartog, M., Matheus, R., Yi Ding, A., & Kuk, G. (2022) · Social Science Computer Review · doi:10.1177/0894439320980118 · Cited by 192 · see citing works
Giovanola, B., & Tiribelli, S. (2023) · AI & SOCIETY · doi:10.1007/s00146-022-01455-6 · Cited by 193 · see citing works
Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
Alkhatib, A., & Bernstein, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300760 · Cited by 188 · see citing works
Grimmelikhuijsen, S., & Meijer, A. (2022) · Perspectives on Public Management and Governance · doi:10.1093/ppmgov/gvac008 · Cited by 189 · see citing works
Keding, C., & Meissner, P. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120970 · Cited by 168 · see citing works
Evaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163 · see citing works
The emergence of economic rationality of GPT
Chen, Y., Liu, T. X., Shan, Y., & Zhong, S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2316205120 · Cited by 164 · see citing works
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.2139/ssrn.4505053 · Cited by 161 · see citing works
The Moral Psychology of Artificial Intelligence
Bonnefon, J.-F., Rahwan, I., & Shariff, A. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-030123-113559 · Cited by 151 · see citing works
Algorithmic Decision-Making and the Control Problem
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Minds and Machines · doi:10.1007/s11023-019-09513-7 · Cited by 145 · see citing works
Argyle, L. P., Bail, C. A., Busby, E. C., Gubler, J. R., Howe, T., Rytting, C., Sorensen, T., & Wingate, D. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311627120 · Cited by 144 · see citing works
Oswald, M. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0359 · Cited by 132 · see citing works
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123 · see citing works
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123 · see citing works
Chugunova, M., & Sele, D. (2022) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2022.101897 · Cited by 118 · see citing works
Human Decision Making with Machine Assistance: An Experiment on Bailing and Jailing
Grgić-Hlača, N., Engel, C., & Gummadi, K. P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359280 · Cited by 113 · see citing works
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098066 · Cited by 110 · see citing works
Self-citationSchlicker, N., Langer, M., Ötting, S. K., Baum, K., König, C. J., & Wallach, D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106837 · Cited by 106 · see citing works
What can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92 · see citing works
A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Nitsche, M., Kühl, N., & Vössing, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534128 · Cited by 91 · see citing works
Lee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78 · see citing works
Tolan, S., Miron, M., Gómez, E., & Castillo, C. (2019) · Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law · doi:10.1145/3322640.3326705 · Cited by 75 · see citing works
Explainability pitfalls: beyond dark patterns in explainable AI
Ehsan, U., & Riedl, M. O. (2024) · Patterns · doi:10.1016/j.patter.2024.100971 · Cited by 72 · see citing works
Fairness, accountability and transparency: Notes on algorithmic decision-making in criminal justice
Chiao, V. (2019) · International Journal of Law in Context · doi:10.1017/s1744552319000077 · Cited by 72 · see citing works
Levy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445522 · Cited by 73 · see citing works
Green, B., & Chen, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479562 · Cited by 69 · see citing works
AI, Explainability and Public Reason: The Argument from the Limitations of the Human Mind
Maclure, J. (2021) · Minds and Machines · doi:10.1007/s11023-021-09570-x · Cited by 67 · see citing works
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369 · see citing works
Li, D., Raymond, L., & Bergman, P. (2020) · doi:10.3386/w27736 · Cited by 64 · see citing works
Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58 · see citing works
Kobiella, C., Flores López, Y. S., Waltenberger, F., Draxler, F., & Schmidt, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641964 · Cited by 57 · see citing works
Casares, A. P. (2018) · Futures · doi:10.1016/j.futures.2018.05.002 · Cited by 54 · see citing works
OK computer: Worker perceptions of algorithmic recruitment
Fumagalli, E., Rezaei, S., & Salomons, A. (2022) · Research Policy · doi:10.1016/j.respol.2021.104420 · Cited by 46 · see citing works
Algorithmic Risk Assessment in the Hands of Humans
Stevenson, M., & Doleac, J. L. (2020) · doi:10.2139/ssrn.3513695 · Cited by 42 · see citing works
Rana, P., & Miller, D. C. (2019) · Environmental Research Letters · doi:10.1088/1748-9326/aafa8f · Cited by 41 · see citing works
The Crowdless Future? Generative AI and Creative Problem Solving
Boussioux, L., N. Lane, J., Zhang, M., Jacimovic, V., & Lakhani, K. R. (2023) · doi:10.2139/ssrn.4533642 · Cited by 343 · see citing works
Synergy Between AI and Robotics: A Comprehensive Integration
Ness, S., Shepherd, N. J., & Xuan, T. R. (2023) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2023/v16i4372 · Cited by 38 · see citing works
To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
Bo, J. Y., Wan, S., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714097 · Cited by 36 · see citing works
Giest, S., & Grimmelikhuijsen, S. (2020) · Information Polity · doi:10.3233/ip-200010 · Cited by 32 · see citing works
How Does AI Improve Human Decision-Making? Evidence from the AI-Powered Go Program
Choi, S., Kim, N., Kim, J., & Kang, H. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3893835 · Cited by 30 · see citing works
von Schenk, A., Klockmann, V., & Köbis, N. (2025) · Perspectives on Psychological Science · doi:10.1177/17456916231194949 · Cited by 30 · see citing works
The Fairness of Credit Scoring Models
Hurlin, C., Perignon, C., & Saurin, S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3785882 · Cited by 26 · see citing works
Toward a science of human–AI teaming for decision making: a complementarity framework
Gonzalez, C., Donahue, K., Goldstein, D. G., Heidari, H., Jalali, M. S., Schelble, B., Singh, A., & Woolley, A. W. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag030 · Cited by 33 · see citing works
Heterogeneity in Algorithm-Assisted Decision-Making: A Case Study in Child Abuse Hotline Screening
Cheng, L., & Chouldechova, A. (2022) · arXiv · doi:10.48550/arxiv.2204.05478 · Cited by 20 · see citing works
Fair governance with humans and machines.
Hermstrüwer, Y., & Langenbach, P. (2023) · Psychology, Public Policy, and Law · doi:10.1037/law0000381 · Cited by 19 · see citing works
Explainability Pitfalls: Beyond Dark Patterns in Explainable AI
Ehsan, U., & Riedl, M. O. (2021) · arXiv · doi:10.48550/arxiv.2109.12480 · Cited by 99 · see citing works
Quantifying machine influence over human forecasters
Abeliuk, A., Benjamin, D. M., Morstatter, F., & Galstyan, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-72690-4 · Cited by 18 · see citing works
Predicting and explaining with machine learning models: Social science as a touchstone
Buchholz, O., & Grote, T. (2023) · Studies in History and Philosophy of Science · doi:10.1016/j.shpsa.2023.10.004 · Cited by 15 · see citing works
Gemalmaz, M. A., & Yin, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534201 · Cited by 10 · see citing works
Gules‐Guctas, E. (2026) · Public Administration Review · doi:10.1111/puar.70043 · Cited by 7 · see citing works
Al Khaldy, M., Aldweesh, A., & al-Qerem, A. (2025) · Advances in Computational Intelligence and Robotics · doi:10.4018/979-8-3373-3760-9.ch003
راشد, ز. ح. (2026) · مجلة الدراسات والبحوث القانونية · doi:10.67603/jlsr.v11i2.10268
Algorithmic monoculture and social welfare
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603202
- Authors
- Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S.
- Affiliations
- Cornell University (US)
- Stanford University (US)
- National Bureau of Economic Research (US)
- Venue
- The Quarterly Journal of Economics
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-08-26
- Volume
- 133
- Issue
- 1
- Pages
- 237-293
- ISSN
- 0033-5533 (ISSN-L)
- 1531-4650
- PMC
- PMC5947971
- OpenAlex
- W2551317447
- Cited by
- 1,040
- References
- 50
- Citations by year
- 2026: 130 · 2025: 140 · 2024: 108 · 2023: 197 · 2022: 91 · 2021: 140 · 2020: 100 · 2019: 89 · 2018: 30 · 2017: 12 · 2016: 2 · 2015: 1
- FWCI
- 163.32
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- The Quarterly Journal of Economics (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Institute of Biomedical Imaging and Bioengineering (U54EB020405)
- Topics
- Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Law, Economics, and Judicial Systems
- Keywords
- bail decisions, pretrial detention, algorithmic prediction, machine learning, judicial decision-making, counterfactual evaluation, quasi-random assignment, racial disparities, violent crime, crime reduction, welfare gains, policy simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
