The DOI matches its Crossref record; 19 of the 22 references it cites were found in a registry or an index.
The Age of Secrecy and Unfairness in Recidivism Prediction
Open accessAbstract
In our current society, secret algorithms make important decisions about individuals. There has been substantial discussion about whether these algorithms are unfair to groups of individuals. While noble, this pursuit is complex and ultimately stagnating because there is no clear definition of fairness and competing definitions are largely incompatible. We argue that the focus on the question of fairness is misplaced, as these algorithms fail to meet a more important and yet readily obtainable goal: transparency. As a result, creators of secret algorithms can provide incomplete or misleading descriptions about how their models work, and various other kinds of errors can easily go unnoticed. By trying to partially reconstruct the COMPAS model—a recidivism risk-scoring model used throughout the criminal justice system—we show that it does not seem to depend linearly on the defendant’s age, despite statements to the contrary by the model’s creator. This observation has not been made before despite many recently published papers on COMPAS. Furthermore, by subtracting from COMPAS its (hypothesized) nonlinear age component, we show that COMPAS does not necessarily depend on race other than through age and criminal history. This contradicts ProPublica’s analysis, which made assumptions about age that disagree with what we observe in the data. In other words, faulty assumptions about a proprietary model led to faulty conclusions that went unchecked until now. Were the model transparent in the first place, this likely would not have occurred. We demonstrate other issues with definitions of fairness and lack of transparency in the context of COMPAS, including that a simple model based entirely on a defendant’s age is as ‘unfair’ as COMPAS by ProPublica’s chosen definition. We find that there are many defendants with low risk scores but long criminal histories, suggesting that data inconsistencies occur frequently in criminal justice databases. We argue that transparency satisfies a different notion of procedural fairness by providing both the defendants and the public with the opportunity to scrutinize the methodology and calculations behind risk scores for recidivism.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Basic typographical errors in inputs to black box recidivism prediction models have led to catastrophic errors deeply affecting people’s lives (29).”
From Interpretable algorithmic forensics (2023) · cites this paper as [29]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Classification and Regression by randomForest
Liaw, A. I., & Wiener, M. C. (2007) · Cited by 18,594
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Self-citationFisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165
Self-citationFairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
Improving the Predictive Accuracy of Static-99 and Static-2002 With Older Sex Offenders
Helmus, L., Thornton, D., Hanson, R. K., & Babchishin, K. (2012) · Sexual abuse. A Journal of Research and Treatment · doi:10.1177/1079063211409951 · Cited by 349
Evaluating the predictive validity of the COMPAS risk and needs assessment system
Brennan, T., Dieterich, W., & Ehret, B. (2009) · Criminal Justice and Behavior · doi:10.1177/0093854808326545 · Cited by 272
Marmarelis, V. Z. (2004) · doi:10.1109/9780471679370.ch5 · Cited by 152
Tollenaar, N., & van der Heijden, P. G. M. (2012) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2012.01056.x · Cited by 133
Actuarial Assessment of Risk for Reoffense Among Adult Sex Offenders
Langton, C. M., Barbaree, H. E., Seto, M. C., Peacock, E. J., Harkins, L., & Hansen, K. T. (2007) · Criminal Justice and Behavior · doi:10.1177/0093854806291157 · Cited by 129
Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice
Rudin, C. D., & Ustun, B. (2018) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.2018.0957 · Cited by 114
Self-citation
Research works printed but not linked: 6
- [21]21. J. Zeng, B. Ustun, C. Rudin, Interpretable classification models for recidivism prediction, Journal of the Royal Statistical Society: Series A (Statistics in Society) 180 , 689 (2017).
- [23]23. E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, C. Rudin, Learning certifiably optimal rule lists for categorical data, ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (2017).
- [24]24. E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, C. Rudin, Certifiably optimal rule lists for categorical data, Journal of Machine Learning Research 19 , 1 (2018).
- [26]26. B. Ustun, C. Rudin, Learning optimized risk scores from large-scale datasets, Journal of Machine Learning Research (2019). Accepted.
- [28]28. S. Tan, R. Caruana, G. Hooker, Y. Lou, Distill-and-compare: Auditing black-box models using transparent model distillation, AAAI/ACM Conference on AI, Ethics, and Society (2018).
- [32]32. T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, 22nd SIGKDD Conference on Knowledge Discovery and Data Mining (2016).
Cited by
Sorted by citations
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Self-citationFisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165
Self-citationInterpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958
Self-citationTransparency and the black box problem: why we do not trust AI
von Eschenbach, W. J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00477-0 · Cited by 786
Chun, W. H. K., & Barnett, A. (2021) · The MIT Press eBooks · doi:10.7551/mitpress/14050.001.0001 · Cited by 456
Metadata
- Authors
- Rudin, C., Wang, C., & Coker, B.
- Affiliations
- Duke University (US)
- Department of Computer Science, Trinity College of Arts and Sciences, Duke University, Durham, North Carolina, United States of America; Department of Mathematics, Trinity College of Arts and Sciences, Duke University, Durham, North Carolina, United States of America
- Department of Biostatistics, T.H. Chan School of Public Health, Harvard University, Cambridge, Massachusetts, United States of America
- Venue
- Harvard Data Science Review
- Publisher
- MIT Press
- Published
- 2020-03-31
- Volume
- 2
- Issue
- 1
- ISSN
- 2644-2353 (ISSN-L)
- 2688-8513
- OpenAlex
- W2899360465
- Cited by
- 177
- References
- 22
- Citations by year
- 2026: 17 · 2025: 26 · 2024: 30 · 2023: 23 · 2022: 31 · 2021: 25 · 2020: 14
- FWCI
- 31.69
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Harvard Data Science Review (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
- Keywords
- COMPAS, algorithmic transparency, algorithmic fairness, criminal justice, ProPublica, model interpretability, procedural justice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
