CoChat
Sign Up
Cited by 116Ethics and Social Impacts of AIDecision-Making and Behavioral EconomicsArtificial Intelligence in Law

Human Decision Making with Machine Assistance: An Experiment on Bailing and Jailing

Open access
Grgić-Hlača, Nina, Engel, Christoph, Gummadi, Krishna P.·2019·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3359280
Open this paper in CoChat — free accountOpen PDF

Abstract

Much of political debate focuses on the concern that machines might take over. Yet in many domains it is much more plausible that the ultimate choice and responsibility remain with a human decision-maker, but that she is provided with machine advice. A quintessential illustration is the decision of a judge to bail or jail a defendant. In multiple jurisdictions in the US, judges have access to a machine prediction about a defendant's recidivism risk. In our study, we explore how receiving machine advice influences people's bail decisions. We run a vignette experiment with laypersons whom we test on a subsample of cases from the database of this prediction tool. In study 1, we ask them to predict whether defendants will recidivate before tried, and manipulate whether they have access to machine advice. We find that receiving machine advice has a small effect, which is biased in the direction of predicting no recidivism. In the field, human decision makers sometimes have a chance, after the fact, to learn whether the machine has given good advice. In study 2, after each trial we inform participants of ground truth. This does not make it more likely that they follow the advice, despite the fact that the machine is (on average) slightly more accurate than real judges. This also holds if initially the advice is mostly correct, or if it initially is mostly to predict (no) recidivism. Real judges know that their decisions affect defendants' lives. They may also be concerned about reelection or promotion. Hence a lot is at stake. In study 3 we emulate high stakes by giving participants a financial incentive. An incentive to find the ground truth, or to avoid false positive or false negatives, does not make participants more sensitive to machine advice. But an incentive to follow the advice is effective.

AI snapshot

Written from the paper’s full text.

TL;DR

In three vignette experiments using COMPAS/ProPublica bail cases, machine advice had only a small effect on lay participants’ recidivism predictions, and the effect was biased toward predicting no recidivism. Feedback about ground truth did not increase reliance on advice, while only incentives tied to following advice increased sensitivity to it.

Key findings

  • Study 1: participants changed their prediction after receiving machine advice in 390 of 5,150 cases; of those changes, 340 followed the advice and 50 opposed it. The estimated population change rate was 7.6% (p < .001).
  • Advice was asymmetric: the probability of shifting toward “no recidivism” when advised to do so was 8.45%, versus 5.7% for shifting toward “recidivism” when advised to do so. Participants were less accurate without advice (60.2%) than with advice (61.6%), and both were below the decision aid’s 68% accuracy.
  • Study 2: giving feedback about ground truth had no discernible effect on predictions or propensity to change after advice; varying whether advice was initially mostly correct/incorrect or mostly bail/jail advice also produced no significant effects.
  • Study 3: incentives to match ground truth or avoid false positives/false negatives did not increase reliance on advice. Incentives to align with machine advice did increase changing decisions after advice, but stronger incentives were not more effective than weak ones (Wald test p = .495); strong alignment incentives raised accuracy from 61.6% to 64.5% and lowered false positive rate from .348 to .296.

Methods

  • Three online vignette experiments on Prolific with U.S. participants who self-reported jury service. Study 1 had 103 participants who passed attention checks; Studies 2 and 3 each had 100, with 20 participants per treatment.
  • Stimuli were 50 randomly selected cases from the ProPublica COMPAS dataset of 7,214 defendants in Broward County, Florida, 2013–2014. The authors trained a logistic regression classifier to predict two-year recidivism; its accuracy was 68%, close to COMPAS.
  • Study 1 used a within-subjects design comparing pre-advice and post-advice predictions. Study 2 added between-subjects feedback and case-order treatments. Study 3 added between-subjects financial incentive treatments. Analyses used linear probability models with participant random effects.

Limitations

  • The paper states the main limitation is the subject pool: for ethical and control reasons, the authors could not study real judges deciding real cases, and testing real judges on hypothetical cases would be confounded by varied experience. They used laypersons eligible for jury duty, most of whom had served.
  • The paper states that feedback and incentives only emulate courtroom conditions: real judges do not receive feedback for every case, never immediately after deciding, and normally do not experience immediate financial consequences. Thus the manipulations estimate upper bounds.
  • Evident from the text: samples were more Caucasian, more educated, and more liberal than the U.S. population; only 74% (Study 1), 79% (Study 2), and 72% (Study 3) reported actual jury service despite being recruited from a jury-service pool.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3359280

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 23 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In Study 3, an incentive to find the ground truth, or to avoid false positives or false negatives, does not make participants more sensitive to machine advice.

    “An incentive to find the ground truth, or to avoid false positive or false negatives, does not make participants more sensitive to machine advice.”
    Quote in the paperSubstantial claim
  • Findingown data

    In Study 3, an incentive to follow the advice is effective.

    “But an incentive to follow the advice is effective.”
    • Also stated in: § 1 INTRODUCTION · plain
      “It only proves effective if participants gain money by following the advice.”
    Quote in the paperSubstantial claim
  • Findingown data

    In Study 2, informing participants of ground truth after each trial does not make it more likely that they follow the machine's advice.

    “In study 2, after each trial we inform participants of ground truth.This does not make it more likely that they follow the advice, despite the fact that the machine is (on average) slightly more accurate than real judges.”
    • Also stated in: § 1 INTRODUCTION · plain
      “In Study 2 we test whether giving human decision makers feedback about the performance of the machine moderates the effect; it does not.”
    Quote in the paperSubstantial claim
  • Findingown data

    Upon receiving advice, in 390 of 5150 cases the participants change their prediction.

    “Upon receiving advice, in 390 of 5150 cases the participants change their prediction.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the population, 7.6% will change their decision in the light of machine advice (p < .001).

    “This regression estimates that, in the population, 7.6% will change their decision in the light of machine advice (p < .001).”
    Quote in the paperSubstantial claim
  • May combine two claims. 21
  • Classified as probably not a substantial claim. 26
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Receiving machine advice has a small effect on participants' predictions, biased in the direction of predicting no recidivism.

    “We find that receiving machine advice has a small effect, which is biased in the direction of predicting no recidivism.”
    • Also stated in: § 1 INTRODUCTION · plain
      “We find a small effect, which is biased in the direction of predicting no recidivism.”
    • Also stated in: § 2.3 Results · plain
      “Descriptively, machine advice does not have a strong effect.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The finding that feedback does not increase advice-following also holds if initially the advice is mostly correct, or if it initially is mostly to predict (no) recidivism.

    “This also holds if initially the advice is mostly correct, or if it initially is mostly to predict (no) recidivism.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Our participants are more highly educated, with close to 60% holding at least a bachelor's degree, compared to 30% in the 2016 U.S. Census.

    “Our participants are also more highly educated, with close to 60% holding at least a bachelor's degree, compared to 30% in the 2016 U.S. Census.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Our 3 samples consist of more Caucasians than the U.S. population (82%, 68% and 68%, compared to 61%).

    “Our 3 samples consists of more Caucasians than the U.S. population (82%, 68% and 68%, compared to 61%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The skews in education and ethnicity are consistent with the findings from prior studies on crowdworking platform demographics.

    “The skews in education and ethnicity are consistent with the findings from prior studies on crowdworking platform demographics [41,54,57].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [41]

      Not checked in the free run.

    • cites [54]

      Not checked in the free run.

    • cites [57]

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    For the 50 defendants considered, the reconstructed classifier predicts 18 defendants to reoffend when put on bail, while police records show 24 defendants actually reoffended.

    “For the 50 defendants we consider in our experiments, the reconstructed classifier predicts that 18 defendants to reoffend when put on bail, while police records show that 24 defendants actually reoffended.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Our participants are more liberal leaning (54%, 50% and 47% compared to 33%).

    “Also, our participants are more liberal leaning (54%, 50% and 47% compared to 33%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Even though we targeted only participants that have self-reported that they have served on a jury in their Prolific profiles, only 74% (study 1), 79% (study 2) and 72% (study 3) of them reported that this is the case in our demographic survey.

    “Even though we targeted only participants that have self-reported that they have served on a jury in their Prolific profiles, only 74% (study 1), 79% (study 2) and 72% (study 3) of them reported that this is the case in our demographic survey.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    While for certain defendants only very few participants predict that the defendant will commit a new crime, in other cases all of the participants predict recidivism, and decisions in the intermediate cases are well balanced.

    “While for certain defendants only very few participants predict that the defendant will commit a new crime, in other cases all of the participants predict recidivism, and decisions in the intermediate cases are well balanced.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Participants' decisions are affected by ground truth and machine confidence.

    “We conclude Result 4: Participants' decisions are affected by ground truth and machine confidence.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.

  • May combine two claims.

    Findingown data

    Before observing machine advice, 71.7% of responses concur with the advice; after observing machine advice, 77.3% concur.

    “Namely, before observing machine advice, 3692 out of the 5150 responses (71.7%) concur with the advice, while 1458 differ.After observing machine advice, 3982 out of the 5150 responses (77.3%) concur with the advice.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.

  • May combine two claims.

    Findingown data

    If respondents receive advice that concurs with their pre-advice prediction, they become more confident; receiving opposing advice leads to a decrease in confidence.

    “Result 5: If respondents receive advice which concurs with their pre-advice prediction, they become more confident, while receiving opposing advice leads to a decrease in confidence.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Giving participants feedback about ground truth has no discernible effect on their predictions or their propensity to change prediction upon receiving advice.

    “Figure 2 shows that giving participants feedback has no discernible effect.”
    • Also stated in: § 3.3 Results · plain
      “We also do not find a significant effect of the feedback treatment on participants' propensity to change their prediction upon receiving advice.7”
    • Also stated in: § 3.3 Results · plain
      “We thus have no support for Hypothesis 6.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.

  • May combine two claims.

    Findingown data

    Strong alignment incentives lead to an increase in accuracy (from 61.6% to 64.5%), with most of this increase visible in lower false positive rates (from .348 to .296).

    “Strong alignment incentives lead to an increase in accuracy (from 61.6% to 64.5%), with most of this increase in performance visible in lower false positive rates (from .348 to .296).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.
    • Rests on an assumption: The financial incentives used in Study 3 emulate higher stakes in the field.

  • May combine two claims.

    Findingown data

    The effects of ground truth and its interaction with advice to jail are insignificant when a case random effect is added to the specification, while all other effects remain significant with p < .015.

    “The effects of ground truth, and its interaction with advice to jail, are insignificant if we add a case random effect to the specification.All other effects remain significant, with p < .015.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In a statistical model with a participant random effect the effect is significant at p < .001, and adding a random effect for cases gives p = .089.

    “In a statistical model with a participant random effect, p < .001;if we further add a random effect for cases, p = .089.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    On average, COMPAS makes correct predictions in only slightly more than 65% of all cases.

    “Yet earlier work has shown that, on average, COMPAS makes correct predictions in only slightly more than 65% of all cases [4,24].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [4]

      Not checked in the free run.

    • cites [24]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The vignettes are shown in random order, to mitigate the effects of order bias.

    “The vignettes are shown in random order, to mitigate the effects of order bias [61].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [61]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    In order to make sure that participants take the task seriously, they have to answer 2 attention check questions, and we discard the answers of all respondents who did not successfully answer both.

    “In order to make sure that participants take the task seriously, they have to answer 2 attention check questions.We discard the answers of all of the respondents who did not successfully answer both attention check questions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Supreme Court of the State of Wisconsin ruled that judges looking at COMPAS scores during sentencing must get warnings about the tool's performance.

    “Based on such considerations, the Supreme Court of the State of Wisconsin ruled that judges looking at COMPAS scores during sentencing must get warnings about the tool's performance [67].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Real judges are not making predictions; they decide whether to put the defendant on bail, but the law wants this decision to be based on the risk of recidivism, and the machine advice exclusively covers this risk.

    “Real judges are not making predictions; they decide whether to put the defendant on bail.However, the law wants this decision to be based on the risk of recidivism, and the machine advice exclusively covers this risk.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Unlike MTurk, Prolific was explicitly designed for online subject recruitment for the scientific community.

    “Unlike MTurk, Prolific was explicitly designed for online subject recruitment for the scientific community [53].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    We test laypersons, who are likely to vary in their attitudes towards crime and incapacitation, and we do not know these attitudes; to overcome this potential confound we directly ask respondents to predict recidivism risk.

    “We test laypersons, who are likely to vary in their attitudes towards crime, and incapacitation in particular.We do not know these attitudes.To overcome this potential confound and directly ask respondents to predict recidivism risk.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    We provide causal evidence, by a series of experiments.

    “We provide causal evidence, by a series of experiments.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The way in which we add feedback and incentives to the design of our experiment only emulates feedback and incentives as they occur in the courtroom.

    “We do, however, readily admit that the way in which we add these elements to the design of our experiment only emulates feedback and incentives as they occur in the courtroom.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductreasoning

    For the reasons just explained, giving feedback, and adding incentives, increases ecological validity.

    “For the reasons just explained, giving feedback, and adding incentives, increases ecological validity.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We do not know whether defendants who have been incarcerated would have committed new crimes when on bail.

    “We do of course not know whether defendants who have been incarcerated would have committed new crimes when on bail.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    We estimate linear probability models, as coefficients can then be directly interpreted, and as we keep the ability to add interaction terms.

    “We estimate linear probability models, as coefficients can then be directly interpreted, and as we keep the ability to add interaction terms [2].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    We replace the COMPAS risk score with the reconstructed classifier, which we know to predict precisely what we ask our participants to predict, namely the risk that defendant recidivates within two years.

    “We replace it with the reconstructed classifier, which we know to predict precisely what we ask our participants to predict, namely the risk that defendant recidivates within two years.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    We estimate a model with a random effect for participants because each participant makes 50 predictions.

    “For statistical analysis, we take into account the fact that each participant makes 50 predictions.This is why we estimate a model with a random effect for participants.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Study design and conductreasoning

    The gap between our manipulations and the decisions in the field that we want to understand is the inevitable price we have to pay for identifying causal effects.

    “This gap between our manipulations and the decisions in the field that we want to understand is the inevitable price we have to pay for identifying causal effects.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    With this design choice, we test machine advice as it truly performs in the field.

    “Hence with this design choice, we test machine advice as it truly performs in the field.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    COMPAS does not reveal how it calculates the risk score, so it is not clear what exactly the predicted risk score represents.

    “However, COMPAS does not reveal how it calculates the risk score.It is therefore not clear what exactly the predicted risk score represents.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Feedback is not critical for the effect of machine advice; human agents do not have to learn when to trust machine advice.

    “Feedback is not critical for the effect of machine advice.Human agents do not have to learn (when) to trust machine advice.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.

  • May combine two claims.

    Interpretationown data

    A financial incentive only matters if it is tied to following the advice, not if it is tied to the quality of the advice or to its direction.

    “A financial incentive (meant to emulate higher stakes) only matters if it is tied to following the advice, not if it is tied to the quality of the advice, or to its direction.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.
    • Rests on an assumption: The financial incentives used in Study 3 emulate higher stakes in the field.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The learning opportunity in Study 2 is much more powerful than it could ever be in the field.

    “As explained, the learning opportunity is much more powerful than it could ever be in the field.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Fresh participants judge individual cases in very similar ways.

    “It is also reassuring that fresh participants judge individual cases in very similar ways.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.
    • Rests on an assumption: The ground truth labels in the COMPAS dataset are accurate.
    • Rests on an assumption: The decision aid used in the study has an average accuracy of 68%.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Experimental evidence shows that judges are predominantly motivated intrinsically.

    “That this expectation is not unfounded can even be shown experimentally [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    The study ensured that participants were eligible for jury duty and that most of them had actually served as jury members.

    “Yet we have made sure that participants are eligible for jury duty, and that most of them have actually served as jury members.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Focusing on the cases studied by Dressel and Farid makes it possible to compare the performance of the study's respondents with the performance of their respondents on the same set of cases.

    “Focusing on the cases studied by Dressel and Farid[24] makes it possible to compare the performance of our respondents with the performance of their respondents, on the same set of cases.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Dressel and Farid[24]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Adding a case random effect does not change the coefficients and p-values.

    “Coefficient and p-value stay the same when adding a case random effect to the specification.”
    • Also stated in: § 4.3 Results · plain
      “All coefficients and all p-values stay the same if we add a case random effect to the specification.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Testing real judges on hypothetical cases would have been problematic because the varied experiences of different judges could not be controlled for.

    “It would also have been problematic to test real judges on hypothetical cases, as we could not control for the varied experiences of the different judges.”
    Quote in the paperSubstantial claim
42 of 42

  • The paper relies on this without support.

    There is no literature that systematically investigates the effect of machine advice on the accuracy of decisions made by humans.

    “To the best of our knowledge, there is no literature that systematically investigates the effect of machine advice on the accuracy of decisions made by humans.”

  • The paper relies on this without support.

    Real judges do not receive feedback for each and every case, and they never receive feedback immediately after making a decision.

    “Real judges do not receive feedback for each and every case, and they never receive feedback immediately after making a decision.”

  • The paper relies on this without support.

    Real judges normally do not experience immediate financial consequences of their decision to bail or jail a defendant.

    “Furthermore, real judges normally do not experience immediate financial consequences of their decision to bail or jail a defendant.”

  • The paper relies on this without support.

    COMPAS produces a risk score on a scale from 0 to 10.

    “COMPAS produces a”

  • The paper relies on this without support.

    The COMPAS tool is not hypothetical, but actually used for consequential human decisions.

    “The tool is not hypothetical, but actually used for consequential human decisions.”

  • The paper relies on this without support.

    The COMPAS tool is managed by a commercial provider that has every interest in improving accuracy.

    “The tool is managed by a commercial provider that has every interest in improving accuracy.”

  • The paper relies on this without support.

    The law wants the bail decision to be based on the risk of recidivism, and the machine advice exclusively covers this risk.

    “However, the law wants this decision to be based on the risk of recidivism, and the machine advice exclusively covers this risk.”

  • The paper relies on this without support.

    Laypersons' decisions on the vignettes are a valid proxy for judges' decisions in the judicial domain.

    “In Study 1, we test laypersons on these vignettes, both without and with the benefit of advice.”
    “Using this approach, we are the first to provide causal evidence about the effect of machine advice on the accuracy of human decision making, in the judicial domain.”

  • The paper relies on this without support.

    The ground truth labels in the COMPAS dataset are accurate.

    “For all cases we know ground truth.”

  • The paper relies on this without support.

    The decision aid used in the study has an average accuracy of 68%.

    “our decision aid is only mildly reliable, with average accuracy of 68%, close to the accuracy of the COMPAS tool [4,24].”

  • The paper relies on this without support.

    Multiple US States give judges access to COMPAS predictions of recidivism risk for bail decisions.

    “This is not a contrived example; multiple US States give judges access to precisely this prediction, based on the commercial product COMPAS, sold by Northpointe.”

  • The authors state this as an assumption.

    The paper is the first to provide causal evidence about the effect of machine advice on the accuracy of human decision making in the judicial domain.

    “Using this approach, we are the first to provide causal evidence about the effect of machine advice on the accuracy of human decision making, in the judicial domain.”

  • The paper relies on this without support.

    The financial incentives used in Study 3 emulate higher stakes in the field.

    “A financial incentive (meant to emulate higher stakes) only matters if it is tied to following the advice, not if it is tied to the quality of the advice, or to its direction.”

  • The paper relies on this without support.

    How participants say they would decide, were they given authority, reflects how they would actually decide.

    “we could only ask how they would decide, were they to be given authority.”

  • The paper relies on this without support.

    Laypersons randomly drawn from the general population are an adequate substitute for real judges in studying decision making with machine advice.

    “This is why we turn to laypersons who are randomly drawn from the general population.”
15 of 15

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even though algorithmic advice can improve the accuracy of human predictions, people's judgments about when and how to diverge from algorithmic recommendations are typically incorrect.

    “even though algorithmic advice can improve the accuracy of human predictions, people's judgments about when and how to diverge from algorithmic recommendations are typically incorrect”

    From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as Grgić-Hlača et al., 2019

    In Study 1 Results (section 2.3; page unavailable), advice raised lay participants’ prediction accuracy from 60.2% to 61.6%, but the decision aid remained more accurate at 68%. This supports the claim that advice can help while participants’ departures from it did not improve overall accuracy. The broader assertion that people’s judgments about when and how to diverge are typically incorrect goes beyond this experiment with laypersons making recidivism predictions. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Human overrides are unlikely to be effective as quality control because substantial evidence demonstrates that humans tend to override algorithms in detrimental rather than beneficial ways.

    “this remedy is unlikely to be effective: substantial evidence demonstrates that humans tend to override algorithms in detrimental rather than beneficial ways”

    From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as Grgić-Hlača et al., 2019

    The Introduction (page unavailable) reports that machine advice had little effect on participants’ decisions, which lends some support to skepticism about human oversight as quality control. But the paper’s experiment concerns laypersons making bail-related predictions; the cited passages do not establish substantial evidence that humans generally override algorithms more often in detrimental than beneficial ways. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval, recidivism prediction, house price estimation, and student admission.

    “Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”

    From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [47]

  • Several mechanisms have been proposed to optimize the combination of human and AI decisions and facilitate appropriate reliance, including interpretable explanations, cognitive forcing functions, and feedback loops.

    “These mechanisms include providing interpretable explanations for AI advice [21, 72, 123], using cognitive forcing functions[17, 47, 99], and incorporating feedback loops to enhance the interaction between humans and AI systems [10, 11, 139].”

    From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [47]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Net agreement gain is described as resolved disagreements

    Minor

    The 290 figure is the net gain in agreement: 340 participants switched to the advice, while 50 switched away from it. Thus 340 of the original disagreements were resolved, not 290.

  • The machine predicts recidivism in 16 cases, not 18

    Minor

    Table 3 records 1,339 + 309 = 1,648 pre-advice responses for cases where the machine predicts recidivism. With 103 participants responding to each case, that represents 16 cases, contradicting the stated 18.

2 of 2

Cites

Flagged first, then by citations

1 of the 72 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

  1. Sam Corbett-Davies and Sharad Goel . 2018. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. arXiv preprint arXiv:1808.00023 ( 2018 ). Sam Corbett-Davies and Sharad Goel. 2018. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. arXiv preprint arXiv:1808.00023 (2018). · arXiv
  2. DRI. 2012 . DRI 2012 National Poll on the Civil Justice System . tempurlhttp://www.dri.org/advocacy/center-for-law-and-public-policy/poll/2012-poll/showURL DRI. 2012. DRI 2012 National Poll on the Civil Justice System . tempurlhttp://www.dri.org/advocacy/center-for-law-and-public-policy/poll/2012-poll/showURL
  3. Brandon L Garrett and John Monahan . 2019. Judging Risk . California Law Review ( 2019 ). Brandon L Garrett and John Monahan. 2019. Judging Risk. California Law Review (2019).
  4. Gabriele Paolacci , Jesse Chandler , and Panagiotis G Ipeirotis . 2010. Running Experiments on Amazon Mechanical Turk . Judgment and Decision Making ( 2010 ). Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running Experiments on Amazon Mechanical Turk . Judgment and Decision Making (2010).
  5. Pew Research Center. 2016. 2016 Party Identification Detailed Tables . tempurlhttp://www.people-press.org/2016/09/13/2016-party-identification-detailed-tables/showURL Pew Research Center. 2016. 2016 Party Identification Detailed Tables . tempurlhttp://www.people-press.org/2016/09/13/2016-party-identification-detailed-tables/showURL

Cited by

Sorted by citations

Related

Authors
Grgić-Hlača, N., Engel, C., & Gummadi, K. P.
Affiliations
  • Max Planck Institute for Software Systems (DE)
  • Max Planck Institute for Behavioral Economics (DE)
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2019-11-07
Volume
3
Issue
CSCW
Pages
1-25
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3359280
OpenAlex
W2974075815
Cited by
113
References
72
Citations by year
2026: 14 · 2025: 12 · 2024: 20 · 2023: 16 · 2022: 25 · 2021: 18 · 2020: 7 · 2019: 1
FWCI
23.16
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
  • SSRN Electronic Journal (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
Funders
European Research Council (789373)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
Keywords
human decision-making, bail decisions, recidivism prediction, algorithmic decision support, judicial decision-making, financial incentives, trust in algorithms, human-machine interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review