Layers of Bias: A Unified Approach for Understanding Problems With Risk Assessment
The DOI matches its Crossref record; 57 of the 62 references it cites were found in a registry or an index.
Abstract
Scholars in several fields, including quantitative methodologists, legal scholars, and theoretically oriented criminologists, have launched robust debates about the fairness of quantitative risk assessment. As the Supreme Court considers addressing constitutional questions on the issue, we propose a framework for understanding the relationships among these debates: layers of bias. In the top layer, we identify challenges to fairness within the risk-assessment models themselves. We explain types of statistical fairness and the tradeoffs between them. The second layer covers biases embedded in data. Using data from a racially biased criminal justice system can lead to unmeasurable biases in both risk scores and outcome measures. The final layer engages conceptual problems with risk models: Is it fair to make criminal justice decisions about individuals based on groups? We show that each layer depends on the layers below it: Without assurances about the foundational layers, the fairness of the top layers is irrelevant.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While both [501, 470] showed that the momentum variants of local SGD can converge to stationary points of non-convex objective functions at the same rate as synchronous mini-batch SGD, it”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [470]
In the citing paper
“This framework generalizes the server momentum framework proposed by Hsu et al. [237], Wang et al. [470], which allows momentum without increasing communication costs.”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [470]
In the citing paper
“Similar dynamics have been found in credit rating agencies [20,40], supply chain audits [69], and accounting audits [58].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [20]
In the citing paper
“Criminal justice researchers have also established that racial bias permeates arrest records, as Black Americans and other minority groups are disproportionately stopped, searched, and arrested by police for the same behaviors exhibited by White Americans [71][72][73].”
From Efficiency vs. Equity: A Structured Interdisciplinary Review of AI in Criminal Justice Risk Assessments (2026) · cites this paper as [71]
In the citing paper
“Over the past few decades, there has been a push for sentencing guidelines and other tools that reduce judicial discretion when making decisions about pre-trial detention and custodial sentences [71].”
From Efficiency vs. Equity: A Structured Interdisciplinary Review of AI in Criminal Justice Risk Assessments (2026) · cites this paper as [71]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
The Politics of Blame Avoidance
Weaver, R. K. (1986) · Journal of Public Policy · doi:10.1017/s0143814x00004219 · Cited by 2,145
Punishment and inequality in America
(2007) · Choice Reviews Online · doi:10.5860/choice.44-4178 · Cited by 2,064
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Clinical versus mechanical prediction: a meta-analysis
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000) · Psychological Assessment · doi:10.1037/1040-3590.12.1.19 · Cited by 1,640
STEFFENSMEIER, D., ULMER, J., & KRAMER, J. (1998) · Criminology · doi:10.1111/j.1745-9125.1998.tb01265.x · Cited by 1,592
The Recent Past and Near Future of Risk and/or Need Assessment
Andrews, D. A., Bonta, J., & Wormith, J. S. (2006) · Crime & Delinquency · doi:10.1177/0011128705281756 · Cited by 1,507
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Gelman, A., Fagan, J., & Kiss, A. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001040 · Cited by 873
Dong, X. (2004) · Current Opinion in Plant Biology · doi:10.1016/j.pbi.2004.07.005 · Cited by 845
Eberhardt, J. L., Davies, P. G., Purdie-Vaughns, V. J., & Johnson, S. L. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01716.x · Cited by 818
Andrews, D. A., Bonta, J., & Wormith, J. S. (2011) · Criminal Justice and Behavior · doi:10.1177/0093854811406356 · Cited by 742
On the Run: Fugitive Life in an American City
Goffman, A. (2014) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226136851.001.0001 · Cited by 723
Pulled Over: How Police Stops Define Race and Citizenship
Epp, C. R., Maynard-Moody, S., & Haider-Markel, D. P. (2014) · doi:10.7208/chicago/9780226114040.001.0001 · Cited by 687
Ethnicity and Sentencing Outcomes in U.S. Federal Courts: Who is Punished More Harshly?
Steffensmeier, D., & Demuth, S. (2000) · American Sociological Review · doi:10.1177/000312240006500505 · Cited by 650
Soss, J., & Weaver, V. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-060415-093825 · Cited by 591
The efficacy of violence prediction: a meta-analytic comparison of nine risk assessment tools.
Yang, M., Wong, S. C. P., & Coid, J. (2010) · Psychological Bulletin · doi:10.1037/a0020473 · Cited by 525
CUMULATIVE DISADVANTAGE: EXAMINING RACIAL AND ETHNIC DISPARITY IN PROSECUTION AND SENTENCING
KUTATELADZE, B. L., ANDILORO, N. R., JOHNSON, B. D., & SPOHN, C. C. (2014) · Criminology · doi:10.1111/1745-9125.12047 · Cited by 478
Arresting Citizenship: The Democratic Consequences of American Crime Control
Lerman, A. E., & Weaver, V. M. (2014) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226137971.001.0001 · Cited by 459
Invisible Men: Mass Incarceration and the Myth of Black Progress
(2013) · Choice Reviews Online · doi:10.5860/choice.50-2982 · Cited by 401
DEMUTH, S. (2003) · Criminology · doi:10.1111/j.1745-9125.2003.tb01007.x · Cited by 295
Precinct or prejudice? Understanding racial disparities in New York City’s stop-and-frisk policy
Goel, S., Rao, J. M., & Shroff, R. (2016) · The Annals of Applied Statistics · doi:10.1214/15-aoas897 · Cited by 285
Polaschek, D. L. L. (2012) · Legal and Criminological Psychology · doi:10.1111/j.2044-8333.2011.02038.x · Cited by 273
Evaluating the predictive validity of the COMPAS risk and needs assessment system
Brennan, T., Dieterich, W., & Ehret, B. (2009) · Criminal Justice and Behavior · doi:10.1177/0093854808326545 · Cited by 272
Risk Assessment in Criminal Sentencing
Monahan, J., & Skeem, J. L. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-092945 · Cited by 256
Flores, A. W., Bechtel, K., & Lowenkamp, C. T. (2016) · Federal probation · Cited by 240
Ariel, B., Sutherland, A., Henstock, D., Young, J., Drover, P., Sykes, J., Megicks, S., & Henderson, R. (2016) · European Journal of Criminology · doi:10.1177/1477370816643734 · Cited by 216
Harcourt, B. E. (2015) · Federal Sentencing Reporter · doi:10.1525/fsr.2015.27.4.237 · Cited by 210
Wooldredge, J., Frank, J., Goulette, N., & Travis, L. (2015) · Criminology & Public Policy · doi:10.1111/1745-9133.12124 · Cited by 201
Life, Liberty, and Trade Secrets: Intellectual Property in the Criminal Justice System
Wexler, R. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2920883 · Cited by 200
Assessing Risk Assessment in Action
Stevenson, M. (2018) · Minnesota Law Review · doi:10.24926/265535.958 · Cited by 193
(2014) · Cited by 190
Spencer, K. B., Charbonneau, A. K., & Glaser, J. (2016) · Social and Personality Psychology Compass · doi:10.1111/spc3.12210 · Cited by 167
Actuarial Sentencing: An “Unsettled” Proposition
Hannah-Moffat, K. (2013) · Justice Quarterly · doi:10.1080/07418825.2012.682603 · Cited by 165
Race and Cumulative Discrimination in the Prosecution of Criminal Defendants
Stolzenberg, L., D’Alessio, S. J., & Eitle, D. (2013) · Race and Justice · doi:10.1177/2153368713500317 · Cited by 164
Performance of recidivism risk assessment instruments in U.S. correctional settings.
Desmarais, S. L., Johnson, K. L., & Singh, J. P. (2016) · Psychological Services · doi:10.1037/ser0000075 · Cited by 143
Discrimination and Implicit Bias in a Racially Unequal Society
Banks, R. R., Eberhardt, J. L., & Ross, L. D. (2006) · California Law Review · doi:10.2307/20439061 · Cited by 140
Goddard, T., & Myers, R. R. (2016) · Theoretical Criminology · doi:10.1177/1362480616645172 · Cited by 123
Schwalbe, C. S., Fraser, M. W., Day, S. H., & Cooley, V. (2006) · Criminal Justice and Behavior · doi:10.1177/0093854806286451 · Cited by 114
van Eijk, G. (2016) · Punishment & Society · doi:10.1177/1462474516666282 · Cited by 107
The slow violence of state organized race crime
Ward, G. (2015) · Theoretical Criminology · doi:10.1177/1362480614550119 · Cited by 102
Implementing Anti-discrimination Policies in Statistical Profiling Models
Pope, D. G., & Sydnor, J. R. (2011) · American Economic Journal: Economic Policy · doi:10.1257/pol.3.3.206 · Cited by 87
Perkins, K. L., & Sampson, R. J. (2015) · RSF The Russell Sage Foundation Journal of the Social Sciences · doi:10.7758/rsf.2015.1.1.03 · Cited by 85
Amphetamine attenuation of tonic immobility in chickens
Boren, J. L., & Gallup, G. G. (1976) · Physiological Psychology · doi:10.3758/bf03326589 · Cited by 84
The War on Marijuana in Black and WhitePreprint
Bunting, W. C., Garcia, L., & Edwards, E. (2013) · SSRN Electronic Journal · Cited by 75
The role of bail and pretrial detention in the reproduction of racial inequalities
Menefee, M. R. (2018) · Sociology Compass · doi:10.1111/soc4.12576 · Cited by 52
DeMichele, M. T., Baumgartner, P., Wenger, M., Barrick, K., Comfort, M. L., & Misra, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3168452 · Cited by 52
Schlesinger, T. (2013) · Race and Justice · doi:10.1177/2153368713483320 · Cited by 50
Starr, S. B. (2015) · Federal Sentencing Reporter · doi:10.1525/fsr.2015.27.4.229 · Cited by 29
Milgram, A., Holsinger, A. M., Vannostrand, M., & Alsdorf, M. W. (2015) · Federal Sentencing Reporter · doi:10.1525/fsr.2015.27.4.216 · Cited by 21
Risk in Sentencing: Constitutionally Suspect Variables and Evidence-Based Sentencing
Oleson, J. C. (2011) · SMU Law Review · doi:10.25172/smulr.64.4.9 · Cited by 20
Race, Drugs, and Law Enforcement: Toward Equitable Policing
Beckett, K. (2012) · Criminology & Public Policy · doi:10.1111/j.1745-9133.2012.00844.x · Cited by 19
Disentangling Flight Risk from Dangerousness
Gouldin, L. P. (2016) · BYU Law Library (Brigham Young University) · Cited by 14
The Problem of Infra-Marginality in Outcome Tests for DiscriminationPreprint
Simoiu, C., Corbett‐Davies, S., & Goel, S. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2811449 · Cited by 6
Research works printed but not linked: 28
- Austin A., 2017, Federal Probation, 81
- Ayoub L. H., 2015, Coming home to Harlem: A randomized controlled trial of the Harlem Parole Reentry Court
- Barocas S., 2016, California Law Review, 104, 721
- Blackmon D. A., 2008, Slavery by another name: The re-enslavement of Black people in America from the Civil War to World War II
- Bonta J., 2007, Rehabilitation, 6, 1
- Chouldechova A., G’Sell M. (2017). Fairer and more accurate, but for whom? arXiv, 1707.00046. Retrieved from http://arxiv.org/abs/1707.00046 · arXiv
- Cooprider K., 2009, Federal Probation, 73, 12
- Dieterich W., 2016, Northpointe, Inc
- Flores A. W., 2016, Federal Probation, 80, 38
- Gouldin L., 2016, Brigham Young University Law Review, 2016, 837
- Kang J., 2011, UCLA Law Review, 59, 1126
- Kleinberg J., 2016, arXiv, 1609, 05807
- Kroll J. A., 2016, University of Pennsylvania Law Review, 165, 633
- Laura and John Arnold Foundation, Pretrial justice
- Levinson J. D., 2007, Duke Law Journal, 57, 345
- Levinson J. D., 2010, Ohio State Journal of Criminal Law, 8, 187
- Lowenkamp C. T., 2008, Federal Probation, 72, 2
- Lowenkamp C. T., 2009, Federal Probation, 73, 33
- Mayson S. G., 2017, The Yale Law Journal, 127, 490
- Mayson S. G. (2018). Bias in, bias out (University of Georgia School of Law Legal Studies Research Paper No. 2018–35). The Yale Law Journal. Advance online publication. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3257004
- Oleson J. C., 2011, SMU Law Review, 64, 1329
- Richardson L. S., 2012, The Yale Law Journal, 122, 2626
- Starr S. B., 2014, Stanford Law Review, 66, 803
- Summers C., 2010, Pretrial risk assessment
- VanNostrand M., 2009, Federal Probation, 73, 5
- Westervelt E., 2017, NPR: All Things Considered
- Wexler R., 2017, Stanford Law Review, 70, 1343
- Zliobaite I. (2017). Fairness-aware machine learning: A perspective. arXiv: 1708.00754. Retrieved from http://arxiv.org/abs/1708.00754 · arXiv
Cited by
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
Self-citationJacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362
“[19] speak to “layers of bias” in risk assessment models: “we show that each layer depends on the layers below it: Without assurances about the foundational layers, the fairness of the top layers is irrelevant.”
BackgroundRoles for computing in social change
Abebe, R., Barocas, S., Kleinberg, J., Levy, K., Raghavan, M., & Robinson, D. G. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372871 · Cited by 219
“Some advocates and scholars have argued that any feasible formalization of the problem, given the available data, will have drawbacks so serious as to draw into question the use of actuarial methods at all [39, 65, 67, 71].”
BackgroundUnderstanding artificial intelligence ethics and safety
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 134
Metadata
- Authors
- Eckhouse, L., Lum, K., Conti-Cook, C., & Ciccolini, J.
- Affiliations
- University of Denver (US)
- Human Rights Data Analysis Group
- The Legal Aid Society (New York)
- Corresponding author
- Eckhouse, Laurel
- Venue
- Criminal Justice and Behavior
- Publisher
- SAGE Publications
- Published
- 2018-11-23
- Volume
- 46
- Issue
- 2
- Pages
- 185-209
- ISSN
- 0093-8548 (ISSN-L)
- 1552-3594
- OpenAlex
- W2901856814
- Cited by
- 156
- References
- 62
- Citations by year
- 2026: 16 · 2025: 16 · 2024: 18 · 2023: 19 · 2022: 26 · 2021: 29 · 2020: 20 · 2019: 12
- FWCI
- 30.30
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-12-14)
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Criminal Justice and Behavior (publishedVersion)
- Funders
- Human Rights Data Analysis Group
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- risk assessment, algorithmic fairness, data bias, racial bias in criminal justice, predictive policing, Supreme Court, constitutional question
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.