Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
Despite vigorous debates about the technical characteristics of risk assessments being deployed in the U.S. criminal justice system, remarkably little research has studied how these tools affect actual decision-making processes. After all, risk assessments do not make definitive decisions---they inform judges, who are the final arbiters. It is therefore essential that considerations of risk assessments be informed by rigorous studies of how judges actually interpret and use them. This paper takes a first step toward such research on human interactions with risk assessments through a controlled experimental study on Amazon Mechanical Turk. We found several behaviors that call into question the supposed efficacy and fairness of risk assessments: our study participants 1) underperformed the risk assessment even when presented with its predictions, 2) could not effectively evaluate the accuracy of their own or the risk assessment's predictions, and 3) exhibited behaviors fraught with "disparate interactions," whereby the use of risk assessments led to higher risk predictions about black defendants and lower risk predictions about white defendants. These results suggest the need for a new "algorithm-in-the-loop" framework that places machine learning decision-making aids into the sociotechnical context of improving human decisions rather than the technical context of generating the best prediction in the abstract. If risk assessments are to be used at all, they must be grounded in rigorous evaluations of their real-world impacts instead of in their theoretical potential.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“What is more, the authors also report important limitations in participants’ ability to evaluate the algorithm's performance (97).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Green and Chen 2019
In the citing paper
“They document a propensity for selective adherence to the algorithm by participants, resulting in discriminatory outcomes.”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Green and Chen 2019
In the citing paper
“However, such audits tend to be conducted after model deployment, when the system has already negatively impacted users [26,51].”
From Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing (2020) · cites this paper as [26]
In the citing paper
“In their experimental study (with lay participants) about receptiveness to algorithmic advice in the context of criminal justice, the authors find evidence of “disparate interactions” with the risk assessment algorithm, leading to higher risk predictions for black defendants and lower risk predictions for white defendants.”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Green and Chen 2019
In the citing paper
“Early empirical evidence does seem to suggest that far from being used as “neutral” devices, algorithmic outputs can accentuate disparate and discriminatory outcomes in human decision‐making (Green and Chen 2019).”
From Accountable Artificial Intelligence: Holding Algorithms to Account (2021) · cites this paper as Green and Chen 2019
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“, the risk assessment) using gradient boosted trees [26].”
BackgroundCrenshaw, K. (2018) · Feminist Legal Theory · doi:10.4324/9780429500480-5 · Cited by 9,510
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793 · cited 1×
“[28], participants were incentivized to report their true estimates of crime risk.”
BackgroundCrenshaw, K. (1998) · Feminism And Politics · doi:10.1093/oso/9780198782063.003.0016 · Cited by 5,672
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 1×
“algorithms than errors made by other people [17].”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 1×
“Recent work related to interpretability provides one important direction where progress is already being made [20, 52, 53].”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“Prompted by these concerns, machine learning researchers have developed a rapidly-growing body of technical work focused on topics such as characterizing the incompatibility of dierent fairness metrics [6, 44] and developing new algorithms to reduce bias [24, 33].”
BackgroundClinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,171 · cited 1×
“Judges may also be more reluctant to rely on risk assessments, believing that their own judgment is superior: previous research has shown that people with more expertise are less willing to take advice [38, 48, 50], and a recent survey found that less than 10%”
BackgroundFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 2×
“Prompted by these concerns, machine learning researchers have developed a rapidly-growing body of technical work focused on topics such as characterizing the incompatibility of dierent fairness metrics [6, 44] and developing new algorithms to reduce bias [24, 33].”
Background
Research works printed but not linked: 11
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016. Machine Bias. ProPublica ( 2016 ). https:/www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias. ProPublica (2016). https:/www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
- Mary L. Cummings . 2006. Automation and Accountability in Decision Support System Interface Design. Journal of Technology Studies ( 2006 ). Mary L. Cummings. 2006. Automation and Accountability in Decision Support System Interface Design. Journal of Technology Studies (2006).
- Mona J.E. Danner , Marie VanNostrand , and Lisa M . Spruance . 2015 . Risk-Based Pretrial Release Recommendation and Supervision Guidelines . Mona J.E. Danner, Marie VanNostrand, and Lisa M. Spruance. 2015. Risk-Based Pretrial Release Recommendation and Supervision Guidelines.
- Sarah L. Desmarais and Jay P . Singh . 2013 . Risk Assessment Instruments Validated and Implemented in Correctional Settings in the United States . (2013). Sarah L. Desmarais and Jay P. Singh. 2013. Risk Assessment Instruments Validated and Implemented in Correctional Settings in the United States. (2013).
- William Dieterich , Christina Mendoza , and Tim Brennan . 2016. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity . Northpointe Inc . ( 2016 ). William Dieterich, Christina Mendoza, and Tim Brennan. 2016. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. Northpointe Inc. (2016).
- Gideon's Promise , The National Legal Aid and Defenders Association , The National Association for Public Defense, and The National Association of Criminal Defense Lawyers . 2017 . Joint Statement in Support of the Use of Pretrial Risk Assessment Instruments. (2017). http://www.publicdefenders.us/files/Defenders%20Statement%20on%20Pretrial%20RAI%20May%202017.pdf Gideon's Promise, The National Legal Aid and Defenders Association, The National Association for Public Defense, and The National Association of Criminal Defense Lawyers. 2017. Joint Statement in Support of the Use of Pretrial Risk Assessment Instruments. (2017). http://www.publicdefenders.us/files/Defenders%20Statement%20on%20Pretrial%20RAI%20May%202017.pdf
- John S. Goldkamp and Michael R . Gottfredson . 1984 . Judicial Decision Guidelines for Bail: The Philadelphia Experiment. National Institute of Justice Research Report ( 1984). John S. Goldkamp and Michael R. Gottfredson. 1984. Judicial Decision Guidelines for Bail: The Philadelphia Experiment. National Institute of Justice Research Report (1984).
- Kamala Harris and Rand Paul . 2017 . Pretrial Integrity and Safety Act of 2017 . 115th Congress (2017). Kamala Harris and Rand Paul. 2017. Pretrial Integrity and Safety Act of 2017. 115th Congress (2017).
- Jeff Larson , Surya Mattu , Lauren Kirchner , and Julia Angwin . 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica ( 2016 ). https:/www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (2016). https:/www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
- Josh Salman , Emily Le Coz, and Elizabeth Johnson . 2016 . Florida's broken sentencing system. Sarasota Herald-Tribune ( 2016). http://projects.heraldtribune.com/bias/sentencing/ Josh Salman, Emily Le Coz, and Elizabeth Johnson. 2016. Florida's broken sentencing system. Sarasota Herald-Tribune (2016). http://projects.heraldtribune.com/bias/sentencing/
- Tom Upchurch . 2018. To work for society, data scientists need a hippocratic oath with teeth. Wired ( 2018 ). https:/www.wired.co.uk/article/data-ai-ethics-hippocratic-oath-cathy-o-neil-weapons-of-math-destruction Tom Upchurch. 2018. To work for society, data scientists need a hippocratic oath with teeth. Wired (2018). https:/www.wired.co.uk/article/data-ai-ethics-hippocratic-oath-cathy-o-neil-weapons-of-math-destruction
Cited by
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Accountable Artificial Intelligence: Holding Algorithms to Account
Busuioc, M. (2021) · Public Administration Review · doi:10.1111/puar.13293 · Cited by 652
“Early empirical evidence does seem to suggest that far from being used as “neutral” devices, algorithmic outputs can accentuate disparate and discriminatory outcomes in human decision-making (Green and Chen 2019).”
BackgroundThe Ethics of Artificial Intelligence: Principles, Challenges, and Opportunities
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 507
Alon-Barkat, S., & Busuioc, M. (2023) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muac007 · Cited by 469
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392
“In the context of algorithm-assisted human decision making, prior research has explored topics such as what interactions can facilitate the development of machine learning models [9, 29, 47], how to improve human performance with an algorithm’s assistance [12, 37, 48], and the ways in which laypeople perceive algorith…”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- Green, B., & Chen, Y.
- Affiliations
- Harvard University (US)
- Venue
- Proceedings of the Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2019-01-29
- Pages
- 90-99
- OpenAlex
- W2901481055
- Cited by
- 275
- References
- 61
- Citations by year
- 2026: 25 · 2025: 37 · 2024: 32 · 2023: 59 · 2022: 44 · 2021: 29 · 2020: 30 · 2019: 19
- FWCI
- 31.82
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Sociology and Political Science › Criminal Justice and Corrections Analysis
- Keywords
- risk assessment tools, criminal justice decision-making, algorithmic fairness, indirect discrimination, human-machine interaction, forecast evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. https://app.cochat.ai/papers/10.1145/3287560.3287563
- The paper
- Green, B., & Chen, Y. (2019). Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. Proceedings of the Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3287560.3287563
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.