CoChat

The accuracy, fairness, and limits of predicting recidivism

Open access
Dressel, Julia, Farid, Hany · 2018 · Science Advances

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.

Abstract

Should we trust computers to make life-altering decisions in the criminal justice system?

Overview

CoChat Summary

TL;DR

In a Broward County sample, nonexpert Amazon Mechanical Turk participants predicted recidivism about as accurately and with similar racial disparities as COMPAS, and COMPAS’s proprietary 137-feature score could be matched by simple linear classifiers using as few as two features. The authors conclude that algorithmic recidivism prediction, at least as represented by COMPAS, is no better than untrained human judgment and reaches a performance ceiling around 65% accuracy.

Key findings

  • A crowd of nonexperts was statistically indistinguishable from COMPAS: crowd accuracy was 67.0% versus COMPAS’s 65.2% on the 1000-defendant subset (P = 0.85), and AUC-ROC was 0.71 ± 0.03 for participants versus 0.70 ± 0.04 for COMPAS.
  • Both humans and COMPAS showed similar racial disparities in error rates: participants’ false-positive rate was 37.1% for black defendants versus 27.2% for white defendants (P = 0.027), while COMPAS’s was 40.4% versus 25.4% (P = 0.002); participants’ false-negative rate was 29.2% for black defendants versus 40.3% for white defendants (P = 0.034), while COMPAS’s was 30.9% versus 47.9% (P = 0.003).
  • COMPAS’s accuracy was reproducible with simpler models: logistic regression with 7 features achieved 66.6% accuracy [64.4, 68.9], logistic regression with only 2 features—age and total number of previous convictions—achieved 66.8% [64.3, 69.2], and a nonlinear SVM achieved 65.2% [63.0, 67.2], all comparable to COMPAS’s 65.4% [64.3, 66.5].
  • Including race in the human prediction task did not significantly change overall accuracy or fairness: crowd accuracy was 66.5% with race versus 67.0% without race (P = 0.66), while false-positive rates for black versus white defendants remained significantly different (40.0% vs. 26.2%, P = 0.001).

Methods

  • Data came from 7214 pretrial defendants in Broward County, Florida, from 2013–2014, including demographics, criminal history, COMPAS recidivism risk scores, and 2-year rearrest records; human assessment used a fixed random subset of 1000 defendants.
  • Human participants were recruited via Amazon’s Mechanical Turk: 462 in the no-race condition and 449 in the race condition, with catch trials removing inattentive respondents; each participant judged one of 20 subsets of 50 defendants, and crowd predictions used a majority-rules criterion.
  • Algorithmic assessment trained logistic regression and nonlinear SVM classifiers on 7 features over 1000 random 80%/20% training/testing splits, and also tested all subsets of the 7 features, including a 2-feature logistic regression.

Limitations

  • Stated in the paper: recidivism is operationalized as rearrest, not direct reoffending, and differences in arrest rates across race complicate direct comparison of false-positive and false-negative rates.
  • Stated in the paper: participants were not instructed how to weight features; it remains unknown whether guidelines, dynamic risk factors, larger crowds, or criminal justice expertise would improve accuracy.
  • Evident from the text: the human sample was drawn from Mechanical Turk and assumed to have little to no criminal justice expertise, there were not enough nonwhite participants to reliably measure differences across participant race, and COMPAS’s proprietary inner workings were not revealed, limiting independent verification.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1126/sciadv.aao5580

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Human (no-race condition) versus COMPAS algorithmic predictions (see also Table 1).
Fig. 1 Human (no-race condition) versus COMPAS algorithmic predictions (see also Table 1).CC BY-NC · PubMed Central

Verification

Claims the paper makes

0 Valid · 1 Not valid · 8 Not checked · 47 Own results and interpretations

20 more claims

Create a free account to see all 56 claims and to verify the 6 references still open.

See all claims — free account
  • It may state two separate claims as one. 28
  • Probably not a substantial claim. 6

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 18 references

Cited by

Items 101–200 of 1,260

Metadata

Authors
Dressel, J., & Farid, H.
Affiliations
Dartmouth College (US)
Corresponding author
Farid, Hany
Venue
Science Advances
Published
2018-01-05
Volume
4
Issue
1
Pages
eaao5580-eaao5580
ISSN
2375-2548 (ISSN-L)
OpenAlex
W2785011159
Cited by
1,071
References
17
Citations by year
2026: 138 · 2025: 124 · 2024: 149 · 2023: 153 · 2022: 126 · 2021: 127 · 2020: 148 · 2019: 76 · 2018: 29
FWCI
188.76
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Science Advances (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
recidivism prediction, COMPAS, risk assessment software, algorithmic fairness, algorithmic accuracy, criminal justice, parole decisions, sentencing, linear predictor, algorithmic bias
MeSH terms
Recidivism, Algorithms, Humans, ROC Curve, Area Under Curve

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.