The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studies
Abstract
As algorithmic risk assessment instruments (RAIs) are increasingly adopted to assist decision makers, their predictive performance and potential to promote inequity have come under scrutiny. However, while most studies examine these tools in isolation, researchers have come to recognize that assessing their impact requires understanding the behavior of their human interactants. In this paper, building off of several recent crowdsourcing works focused on criminal justice, we conduct a vignette study in which laypersons are tasked with predicting future re-arrests. Our key findings are as follows: (1) Participants often predict that an offender will be rearrested even when they deem the likelihood of re-arrest to be well below 50%; (2) Participants do not anchor on the RAI's predictions; (3) The time spent on the survey varies widely across participants and most cases are assessed in less than 10 seconds; (4) Judicial decisions, unlike participants' predictions, depend in part on factors that are orthogonal to the likelihood of re-arrest. These results highlight the influence of several crucial but often overlooked design decisions and concerns around generalizability when constructing crowdsourcing studies to analyze the impacts of RAI
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Fogliato et al. [12], there is no deterministic decision rule that describes how participants' probability estimates map to their binary decisions.
“there is no determinstic decision rule that describes how the participants' probability estimates map to their binary decisions.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Section 4.1 reports that most participants did not use a fixed probability threshold and that a uniform 50% threshold does not reproduce their binary predictions. But 32.4% consistently followed the 50% rule, and the paper does not rule out every possible deterministic mapping; the claim is broader than its finding. (LLM, conf=0.85)
Experimental — accuracy not yet measuredIn Fogliato et al. [12], people are better at the probability task than the decision task.
“we find that people are better at the probability task than the decision task.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Section 4.1 (page unavailable) reports that participants’ binary decisions often diverged from what their probability estimates would imply at a 50% threshold. That supports a difference between the tasks, but not the broader claim that people performed better at the probability task: the paper says the overall accuracy of decisions derived from probability estimates was similar to that of their actual binary decisions. (LLM, conf=0.78)
Experimental — accuracy not yet measuredIn Fogliato et al. [12], behavioral decision-makers' reliance is closer to the appropriate reliance under the non-anchoring condition than the anchoring condition in both tasks.
“we find that the behavioral decision-makers' reliance is closer to the appropriate reliance under the non-anchoring condition than the anchoring condition in both tasks.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Section 4.2 (page unavailable) reports that revised risk estimates in the non-anchoring condition were closer to the RAI’s estimates than those in the anchoring condition. But binary predictions matched the RAI at the same rate in both conditions. The paper therefore does not support the claim that reliance was closer to appropriate reliance in both tasks; closeness to the RAI is not itself established as appropriate reliance. (LLM, conf=0.87)
Experimental — accuracy not yet measuredLetting behavioral decision-makers make a decision by themselves first can improve their reliance, but not necessarily help them distinguish between signals where the AI recommendation is expected to outperform the human recommendation and signals where the human recommendation is expected to outperform the AI recommendation.
“This suggests that letting the behavioral decision-makers make a decision by themselves first (a.k.a., the non-anchoring effects) can improve their reliance, but not necessarily help them distinguish between the signals where the AI recommendation is expected to outperform the human recommendation and the signals where the human recommendation is expected to outperform the AI recommendation.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredApplying the framework to three well-regarded AI-advised decision making experiments from literature, in all three cases, examining results against the baseline and benchmark for complementary performance better reveals the limits of human behavioral performance.
“We apply the framework to three well-regarded AI-advised decision making experiments from literature [3, 12, 21]. In all three cases, we show 1) that examining the results against the baseline and benchmark for complementary performance better reveals the limits of human behavioral performance”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
In Fogliato et al. [12], behavioral performance is similar to the human-only baseline in the decision task.
“while they perform similarly in the decision task”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
In Fogliato et al. [12], rational baselines and rational benchmarks differ for each task between the anchoring and no anchoring conditions.
“the rational baselines and the rational benchmarks differ for each task between the anchoring and the no anchoring conditions”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Section 3.2 describes the anchoring and non-anchoring tasks, and §4.2 compares participants’ predictions and performance across those conditions. Neither passage addresses rational baselines or rational benchmarks for each task; searches for those terms found no passage that establishes the claimed difference. (LLM, conf=0.72)
Experimental — accuracy not yet measuredApplying the framework to the three experiments identifies specific sources of behavioral loss that help explain the experiment results but were not accounted for by the original interpretations.
“and 2) specific sources of behavioral loss that help explain the experiment results but were not accounted for by the original interpretations of the results.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Section 4 and §5.2 discuss this paper’s experiment and offer possible explanations for participants’ predictions, but they do not describe a framework applied to three experiments or identify sources of “behavioral loss” omitted from earlier interpretations. Searches for those terms and paraphrases found no passage addressing that claim; the indexed passages provide section labels but no page numbers. (LLM, conf=0.70)
Experimental — accuracy not yet measuredIn Fogliato et al. [12], discrimination loss is slightly higher for the no anchoring condition.
“while the discrimination loss ( R m -B R -R ∅ ) is slightly higher.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Section 4.2 compares anchoring and non-anchoring conditions, but reports only that AUC and other performance measures did not differ significantly; it does not give the direction of an AUC difference or report discrimination loss. Searches for “discrimination loss” and the anchoring comparison found no passage establishing that discrimination loss was slightly higher in the non-anchoring condition. (LLM, conf=0.72)
Experimental — accuracy not yet measuredIn Fogliato et al. [12], behavioral performance is higher than the human-only baseline in the probabilistic task.
“the behavioral performance is higher than the performance of the human only baseline in the probabilistic task”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
In Fogliato et al. [12], the rational baseline in the anchoring condition is slightly higher than in the non-anchoring condition.
“the rational baseline in the anchoring condition is slightly higher than in the non-achoring condition.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
In Fogliato et al. [12], reliance loss is lower for the no anchoring condition.
“the reliance loss ( R -R m R -R ∅ ) is lower for the no anchoring condition”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
In Fogliato et al. [12], despite similar absolute behavioral performance across conditions, behavioral decision-makers have better relative performance in the non-anchoring condition than the anchoring condition when compared to the rational baseline and rational benchmark.
“Despite the behavioral performance being similar across the conditions in terms of absolute values, the behavioral decision-makers have better relative performance in the non-anchoring condition than the anchoring condition when compared to the rational baseline and the rational benchmark.”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision Under Risk
Kahneman, D., & Tversky, A. (2013) · World Scientific Handbook in Financial Economics Series · doi:10.1142/9789814417358_0006 · Cited by 3,091
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 1×
“Risk assessment instruments (RAIs) are increasingly deployed in critical domains, including finance, education, criminal justice, child welfare, hiring, and healthcare [8, 21, 63, 68, 71, 74].”
BackgroundZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1×
“In order to overcome the difficulties of characterizing decision-making and the impact of RAIs on systems where the human is in-the-loop, many researchers have turned to lab experiments [6, 65, 85, 86].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
Englich, B., Mussweiler, T., & Strack, F. (2006) · Personality and Social Psychology Bulletin · doi:10.1177/0146167205282152 · Cited by 554
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 2×
“These measures include analogs to measures that were adopted by prior work, such as deviation [65], influence [38, 39, 65], agreement fraction, and switch fraction [85].”
BackgroundMethodThe Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 7×
“In §??, we show that, in contrast with past work [38, 39], we did not find evidence of disparate-interactions in the uptake of RAI’s predictions.”
BackgroundMethodResultInfluentialLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
RISK, RACE, AND RECIDIVISM: PREDICTIVE BIAS AND DISPARATE IMPACT*
SKEEM, J. L., & LOWENKAMP, C. T. (2016) · Criminology · doi:10.1111/1745-9125.12123 · Cited by 317
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221
Self-citationAssessing Risk Assessment in Action
Stevenson, M. (2018) · Minnesota Law Review · doi:10.24926/265535.958 · Cited by 193
An impact assessment of machine learning risk forecasts on parole board decisions and recidivism
Berk, R. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9286-2 · Cited by 169
The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098066 · Cited by 138
A Human-Centered Review of Algorithms used within the U.S. Child Welfare System
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376229 · Cited by 116
Human Decision Making with Machine Assistance: An Experiment on Bailing and Jailing
Grgić-Hlača, N., Engel, C., & Gummadi, K. P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359280 · Cited by 113
The effects of mixing machine learning and human judgment
Vaccaro, M., & Waldo, J. (2019) · Communications of the ACM · doi:10.1145/3359338 · Cited by 49
Do I Look Like a Criminal? Examining how Race Presentation Impacts Human Judgement of Recidivism
Mallari, K., Inkpen, K., Johns, P., Tan, S., Ramesh, D., & Kamar, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376257 · Cited by 24
Flagged first, then by citations
Research works printed but not linked: 54
- 10.5555/3122009.3290419 · DOI
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica. See https://www. propublica. org/article/machine-bias-risk-assessments-in-criminal-sentencing (2016).
- James Austin and Wendy Ware. 2020. Why Bail Reform is Safe and Effective: The Case of Cook County. Available at SSRN 3599410 (2020).
- 10.1145/3378393.3402507 · DOI
- 10.1093/jnci/87.10.720 · DOI
- Leo Breiman. 2001. Random forests. Machine learning, Vol. 45, 1 (2001), 5--32.
- 10.1111/j.1740-1461.2012.01254.x · DOI
- 10.1145/2939672.2939785 · DOI · Paper page
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148.
- Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. 2021. Characterizing fairness over the set of good models under selective labels. arXiv preprint arXiv:2101.00352 (2021). · arXiv
- Mick P Couper. 2008. Designing effective web surveys .Cambridge University Press.
- 10.1016/j.dss.2010.06.003 · DOI
- Matthew DeMichele, Peter Baumgartner, Michael Wenger, Kelle Barrick, Megan Comfort, and Shilpi Misra. 2018a. The public safety assessment: A re-validation and assessment of predictive utility and differential prediction by race and gender in kentucky. Available at SSRN 3168452 (2018).
- Matthew DeMichele, Megan Comfort, Shilpi Misra, Kelle Barrick, and Peter Baumgartner. 2018b. The intuitive-override model: Nudging judges toward pretrial risk assessment instruments. Available at SSRN 3168500 (2018).
- Sarah L Desmarais, Samantha A Zottola, Sarah E Duhart Clarke, and Evan M Lowder. 2020. Predictive validity of pretrial risk assessments: A systematic review of the literature. Criminal Justice and Behavior (2020), 0093854820932959.
- 10.1093/biomet/58.3.403 · DOI · Paper page
- 10.1007/s10979-005-8380-7 · DOI
- 10.1111/j.1559-1816.2001.tb02687.x · DOI
- 10.1002/bdm.495 · DOI
- 10.1093/acprof:oso/9780198525134.001.0001 · DOI
- Riccardo Fogliato, Alexandra Chouldechova, and Max G'Sell. 2020. Fairness Evaluation in Presence of Biased Noisy Labels. In International Conference on Artificial Intelligence and Statistics. PMLR, 2325--2336.
- 10.1162/COLI_a_00057 · DOI
- 10.1002/bdm.539 · DOI
- 10.1037/0096-3445.117.3.227 · DOI
- 10.1145/3287560.3287563 · DOI · Paper page
- Chris Guthrie, Jeffrey J Rachlinski, and Andrew J Wistrich. 2000. Inside the judicial mind. Cornell L. Rev., Vol. 86 (2000), 777.
- Jordan Hyatt and Steven L Chanenson. 2016. The use of risk assessment at sentencing: Implications for research and policy. Villanova Law/Public Policy Research Paper 2017--1040 (2016).
- Jongbin Jung Sharad Goel Jennifer Skeem et al. 2020. The limits of human predictions of recidivism. Science advances Vol. 6 7 (2020) eaaz0652. · DOI
- 10.1109/TVCG.2020.3030335 · DOI · Paper page
- Nathan Kallus and Angela Zhou. 2019. Assessing Disparate Impact of Personalized Interventions: Identifiability and Bounds. In Advances in Neural Information Processing Systems. 3426--3437.
- 10.3386/w23180 · DOI · Paper page
- Jonathan J Koehler. 1996. The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behavioral and brain sciences, Vol. 19, 1 (1996), 1--17.
- Isaac M Lipkus, Greg Samsa, and Barbara K Rimer. 2001. General performance on a numeracy scale among highly educated samples. Medical decision making, Vol. 21, 1 (2001), 37--44.
- 10.1111/j.1468-0262.2004.00537.x · DOI
- Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods, Vol. 44, 1 (2012), 1--23.
- 10.1006/jesp.1999.1414 · DOI
- 10.1146/annurev-economics-072412-131310 · DOI
- Richard E Nisbett and Timothy D Wilson. 1977. Telling more than we can know: verbal reports on mental processes. Psychological review, Vol. 84, 3 (1977), 231.
- Anthony O'Hagan, Caitlin E Buck, Alireza Daneshkhah, J Richard Eiser, Paul H Garthwaite, David J Jenkinson, Jeremy E Oakley, and Tim Rakow. 2006. Uncertain judgements: eliciting experts' probabilities .John Wiley & Sons.
- Daniel Paravisini and Antoinette Schoar. 2013. The incentive effect of scores: Randomized evidence from credit committees. Technical Report. National Bureau of Economic Research. · DOI
- Drazen Prelec. 1998. The probability weighting function. Econometrica (1998), 497--527.
- 10.1109/SP.2019.00014 · DOI
- Catherine A Roster, Lorenzo Lucianetti, and Gerald Albaum. 2015. Exploring slider vs. categorical response formats in web-based surveys. Journal of Research Practice, Vol. 11, 1 (2015), D1--D1.
- 10.7326/0003-4819-127-11-199712010-00003 · DOI
- Nicholas Scurich and Richard S John. 2011. The effect of framing actuarial risk probabilities on involuntary civil commitment decisions. Law and human behavior, Vol. 35, 2 (2011), 83--91.
- 10.1145/3351095.3372827 · DOI · Paper page
- Jennifer Skeem and Christopher Lowenkamp. 2020. Using algorithms to address trade-offs inherent in predicting recidivism. Behavioral Sciences & the Law (2020).
- Sarah Tan, Julius Adebayo, Kori Inkpen, and Ece Kamar. 2018. Investigating Human+Machine Complementarity for Recidivism Predictions. arXiv preprint arXiv:1808.09123 (2018). · arXiv
- 10.1111/j.2517-6161.1996.tb02080.x · DOI · Paper page
- Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. science, Vol. 185, 4157 (1974), 1124--1131.
- 10.1145/3358955.3363293 · DOI
- 10.1111/j.1539-6924.2008.01137.x · DOI
- 10.21105/joss.01686 · DOI · Paper page
- Timothy D Wilson. 2004. Strangers to ourselves. Harvard University Press.
Cited by
Sorted by citations
Hadi Mogavi, R., Deng, C., Juho Kim, J., Zhou, P., D. Kwon, Y., Hosny Saleh Metwally, A., Tlili, A., Bassanelli, S., Bucchiarone, A., Gujar, S., et al. (2024) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100027 · Cited by 369
Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
“Topics like performance feedback [2, 55], risk perception [27, 34], uncertainty [77] and confidence [10, 79, 87] of machine learning models, impact of explanations [3, 46] have been extensively studied in human-AI decision making.”
BackgroundA Decision Theoretic Framework for Measuring AI Reliance
Guo, Z., Wu, Y., Hartline, J. D., & Hullman, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658901 · Cited by 27
He, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23
Metadata
- Authors
- Fogliato, R., Chouldechova, A., & Lipton, Z.
- Affiliations
- Carnegie Mellon University, Pittsburgh, PA, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-24
- ISSN
- 2573-0142
- Cited by
- 23
- References
- 22
- Language
- en
- Funders
- Digital Transformation and Innovation Center sponsored by PwC (A026204)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
