CoChat

Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments

Green, Ben, Chen, Yiling · 2019 · Proceedings of the Conference on Fairness, Accountability, and Transparency

The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.

Abstract

Despite vigorous debates about the technical characteristics of risk assessments being deployed in the U.S. criminal justice system, remarkably little research has studied how these tools affect actual decision-making processes. After all, risk assessments do not make definitive decisions---they inform judges, who are the final arbiters. It is therefore essential that considerations of risk assessments be informed by rigorous studies of how judges actually interpret and use them. This paper takes a first step toward such research on human interactions with risk assessments through a controlled experimental study on Amazon Mechanical Turk. We found several behaviors that call into question the supposed efficacy and fairness of risk assessments: our study participants 1) underperformed the risk assessment even when presented with its predictions, 2) could not effectively evaluate the accuracy of their own or the risk assessment's predictions, and 3) exhibited behaviors fraught with "disparate interactions," whereby the use of risk assessments led to higher risk predictions about black defendants and lower risk predictions about white defendants. These results suggest the need for a new "algorithm-in-the-loop" framework that places machine learning decision-making aids into the sociotechnical context of improving human decisions rather than the technical context of generating the best prediction in the abstract. If risk assessments are to be used at all, they must be grounded in rigorous evaluations of their real-world impacts instead of in their theoretical potential.

Overview

Integrity signals

Registry

Crossref · 10.1145/3287560.3287563

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 62 references

Research works printed but not linked: 11

  1. Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016. Machine Bias. ProPublica ( 2016 ). https:/www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias. ProPublica (2016). https:/www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
  2. Mary L. Cummings . 2006. Automation and Accountability in Decision Support System Interface Design. Journal of Technology Studies ( 2006 ). Mary L. Cummings. 2006. Automation and Accountability in Decision Support System Interface Design. Journal of Technology Studies (2006).
  3. Mona J.E. Danner , Marie VanNostrand , and Lisa M . Spruance . 2015 . Risk-Based Pretrial Release Recommendation and Supervision Guidelines . Mona J.E. Danner, Marie VanNostrand, and Lisa M. Spruance. 2015. Risk-Based Pretrial Release Recommendation and Supervision Guidelines.
  4. Sarah L. Desmarais and Jay P . Singh . 2013 . Risk Assessment Instruments Validated and Implemented in Correctional Settings in the United States . (2013). Sarah L. Desmarais and Jay P. Singh. 2013. Risk Assessment Instruments Validated and Implemented in Correctional Settings in the United States. (2013).
  5. William Dieterich , Christina Mendoza , and Tim Brennan . 2016. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity . Northpointe Inc . ( 2016 ). William Dieterich, Christina Mendoza, and Tim Brennan. 2016. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. Northpointe Inc. (2016).
  6. Gideon's Promise , The National Legal Aid and Defenders Association , The National Association for Public Defense, and The National Association of Criminal Defense Lawyers . 2017 . Joint Statement in Support of the Use of Pretrial Risk Assessment Instruments. (2017). http://www.publicdefenders.us/files/Defenders%20Statement%20on%20Pretrial%20RAI%20May%202017.pdf Gideon's Promise, The National Legal Aid and Defenders Association, The National Association for Public Defense, and The National Association of Criminal Defense Lawyers. 2017. Joint Statement in Support of the Use of Pretrial Risk Assessment Instruments. (2017). http://www.publicdefenders.us/files/Defenders%20Statement%20on%20Pretrial%20RAI%20May%202017.pdf
  7. John S. Goldkamp and Michael R . Gottfredson . 1984 . Judicial Decision Guidelines for Bail: The Philadelphia Experiment. National Institute of Justice Research Report ( 1984). John S. Goldkamp and Michael R. Gottfredson. 1984. Judicial Decision Guidelines for Bail: The Philadelphia Experiment. National Institute of Justice Research Report (1984).
  8. Kamala Harris and Rand Paul . 2017 . Pretrial Integrity and Safety Act of 2017 . 115th Congress (2017). Kamala Harris and Rand Paul. 2017. Pretrial Integrity and Safety Act of 2017. 115th Congress (2017).
  9. Jeff Larson , Surya Mattu , Lauren Kirchner , and Julia Angwin . 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica ( 2016 ). https:/www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (2016). https:/www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
  10. Josh Salman , Emily Le Coz, and Elizabeth Johnson . 2016 . Florida's broken sentencing system. Sarasota Herald-Tribune ( 2016). http://projects.heraldtribune.com/bias/sentencing/ Josh Salman, Emily Le Coz, and Elizabeth Johnson. 2016. Florida's broken sentencing system. Sarasota Herald-Tribune (2016). http://projects.heraldtribune.com/bias/sentencing/
  11. Tom Upchurch . 2018. To work for society, data scientists need a hippocratic oath with teeth. Wired ( 2018 ). https:/www.wired.co.uk/article/data-ai-ethics-hippocratic-oath-cathy-o-neil-weapons-of-math-destruction Tom Upchurch. 2018. To work for society, data scientists need a hippocratic oath with teeth. Wired (2018). https:/www.wired.co.uk/article/data-ai-ethics-hippocratic-oath-cathy-o-neil-weapons-of-math-destruction

Cited by

See all 336

Metadata

Authors
Green, B., & Chen, Y.
Affiliations
Harvard University (US)
Venue
Proceedings of the Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2019-01-29
Pages
90-99
OpenAlex
W2901481055
Cited by
275
References
61
Citations by year
2026: 25 · 2025: 37 · 2024: 32 · 2023: 59 · 2022: 44 · 2021: 29 · 2020: 30 · 2019: 19
FWCI
31.82
Type
conference paper
Language
en
Open access
closed
Locations
ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Artificial Intelligence in Law
  • Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
Keywords
risk assessment tools, criminal justice decision-making, algorithmic fairness, indirect discrimination, human-machine interaction, forecast evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. https://app.cochat.ai/papers/10.1145/3287560.3287563
The paper
Green, B., & Chen, Y. (2019). Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. Proceedings of the Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3287560.3287563

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.