CoChat
Sign Up
Cited by 178Ethics and Social Impacts of AIArtificial Intelligence in LawCriminal Justice and Corrections Analysis

The age of secrecy and unfairness in recidivism prediction

Open access
Rudin, Cynthia, Wang, Caroline, Coker, Beau·2020·Harvard Data Science Review
DOI 10.1162/99608f92.6ed64b30
Open this paper in CoChat — free account

Abstract

In our current society, secret algorithms make important decisions about individuals. There has been substantial discussion about whether these algorithms are unfair to groups of individuals. While noble, this pursuit is complex and ultimately stagnating because there is no clear definition of fairness and competing definitions are largely incompatible. We argue that the focus on the question of fairness is misplaced, as these algorithms fail to meet a more important and yet readily obtainable goal: transparency. As a result, creators of secret algorithms can provide incomplete or misleading descriptions about how their models work, and various other kinds of errors can easily go unnoticed. By trying to partially reconstruct the COMPAS model—a recidivism risk-scoring model used throughout the criminal justice system—we show that it does not seem to depend linearly on the defendant’s age, despite statements to the contrary by the model’s creator. This observation has not been made before despite many recently published papers on COMPAS. Furthermore, by subtracting from COMPAS its (hypothesized) nonlinear age component, we show that COMPAS does not necessarily depend on race other than through age and criminal history. This contradicts ProPublica’s analysis, which made assumptions about age that disagree with what we observe in the data. In other words, faulty assumptions about a proprietary model led to faulty conclusions that went unchecked until now. Were the model transparent in the first place, this likely would not have occurred. We demonstrate other issues with definitions of fairness and lack of transparency in the context of COMPAS, including that a simple model based entirely on a defendant’s age is as ‘unfair’ as COMPAS by ProPublica’s chosen definition. We find that there are many defendants with low risk scores but long criminal histories, suggesting that data inconsistencies occur frequently in criminal justice databases. We argue that transparency satisfies a different notion of procedural fairness by providing both the defendants and the public with the opportunity to scrutinize the methodology and calculations behind risk scores for recidivism.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/99608f92.6ed64b30

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Basic typographical errors in inputs to black box recidivism prediction models have led to catastrophic errors deeply affecting people's lives.

    “Basic typographical errors in inputs to black box recidivism prediction models have led to catastrophic errors deeply affecting people’s lives (29).”

    From Interpretable algorithmic forensics (2023) · cites this paper as [29]

    Section 3 (pp. 20–21) describes a data-input error involving the proprietary COMPAS model and a prisoner whose parole was denied. But the paper’s example of a mistyped input preceding a murder involved a simple, transparent scoring system, not a black-box model (Introduction, p. 2; Section 3, p. 19). The claim conflates these cases when it attributes that catastrophic outcome to black-box models. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

3 of the 22 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. [21]21. J. Zeng, B. Ustun, C. Rudin, Interpretable classification models for recidivism prediction, Journal of the Royal Statistical Society: Series A (Statistics in Society) 180 , 689 (2017).
  2. [23]23. E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, C. Rudin, Learning certifiably optimal rule lists for categorical data, ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (2017).
  3. [24]24. E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, C. Rudin, Certifiably optimal rule lists for categorical data, Journal of Machine Learning Research 19 , 1 (2018).
  4. [26]26. B. Ustun, C. Rudin, Learning optimized risk scores from large-scale datasets, Journal of Machine Learning Research (2019). Accepted.
  5. [28]28. S. Tan, R. Caruana, G. Hooker, Y. Lou, Distill-and-compare: Auditing black-box models using transparent model distillation, AAAI/ACM Conference on AI, Ethics, and Society (2018).
  6. [32]32. T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, 22nd SIGKDD Conference on Knowledge Discovery and Data Mining (2016).

Cited by

Sorted by citations

Related

Authors
Rudin, C., Wang, C., & Coker, B.
Affiliations
  • Duke University (US)
  • Department of Computer Science, Trinity College of Arts and Sciences, Duke University, Durham, North Carolina, United States of America; Department of Mathematics, Trinity College of Arts and Sciences, Duke University, Durham, North Carolina, United States of America
  • Department of Biostatistics, T.H. Chan School of Public Health, Harvard University, Cambridge, Massachusetts, United States of America
ORCID
Venue
Harvard Data Science Review
Publisher
MIT Press
Published
2020-03-31
Volume
2
Issue
1
ISSN
  • 2644-2353 (ISSN-L)
  • 2688-8513
DOI
10.1162/99608f92.6ed64b30
OpenAlex
W2899360465
Cited by
177
References
22
Citations by year
2026: 17 · 2025: 26 · 2024: 30 · 2023: 23 · 2022: 31 · 2021: 25 · 2020: 14 · 2019: 9 · 2018: 2
FWCI
31.69
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Harvard Data Science Review (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
  • Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
Keywords
COMPAS, algorithmic transparency, algorithmic fairness, criminal justice, ProPublica, model interpretability, procedural justice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review