How Child Welfare Workers Reduce Racial Disparities in Algorithmic Decisions
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Machine learning tools have been deployed in various contexts to support human decision-making, in the hope that human-algorithm collaboration can improve decision quality. However, the question of whether such collaborations reduce or exacerbate biases in decision-making remains underexplored. In this work, we conducted a mixed-methods study, analyzing child welfare call screen workers’ decision-making over a span of four years, and interviewing them on how they incorporate algorithmic predictions into their decision-making process. Our data analysis shows that, compared to the algorithm alone, workers reduced the disparity in screen-in rate between Black and white children from 20% to 9%. Our qualitative data show that workers achieved this by making holistic risk assessments and adjusting for the algorithm’s limitations. Our analyses also show more nuanced results about how human-algorithm collaboration affects prediction accuracy, and how to measure these effects. These results shed light on potential mechanisms for improving human-algorithm collaboration in high-risk decision-making contexts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 3×
“In some contexts, human-AI collaboration has demonstrated potential to improve fairness and efectiveness of decision-making, compared with human or AI decision-making alone, e.g. [6 , 23, 43, 52, 71].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 3×
“However, given recent empirical fndings demonstrating ways such local explanations can backfre in ways that harm human-AI decision-making (e.g., [6, 74]), further research is needed to understand how these can be designed and presented efectively.”
BackgroundMethodManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations
Perry, W., McInnis, B., Price, C., Smith, S., & Hollywood, J. (2013) · doi:10.7249/rr233 · Cited by 693
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Lifetime Prevalence of Investigating Child Maltreatment Among US Children
Kim, H., Wildeman, C., Jonson-Reid, M., & Drake, B. (2017) · American Journal of Public Health · doi:10.2105/ajph.2016.303545 · Cited by 640 · cited 1×
“Since child welfare workers themselves are known to make racially disparate decisions [24, 25, 27, 40, 54], it is unclear whether adding them back “in the loop” will do any good in this regard.”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“Recent years have seen the deployment of AI-based tools either to augment or replace human judgments across a growing range of high-impact decision-making contexts, such as social work, criminal justice, hiring, healthcare, and education [11, 15, 38, 43, 44, 58, 72, 95, 98].”
BackgroundTschandl, P., Codella, N., Akay, B. N., Argenziano, G., Braun, R. P., Cabo, H., Gutman, D., Halpern, A., Helba, B., Hofmann-Wellenhof, R., et al. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30333-x · Cited by 555 · cited 1×
“Prior work suggests that on various predictive tasks, AI systems are more accurate than human decision makers [28, 30, 55, 89].”
BackgroundBuilding Classifiers with Independency Constraints
Calders, T., Kamiran, F., & Pechenizkiy, M. (2009) · 2009 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2009.83 · Cited by 468
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 3×
“2 RELATED WORK 2.1 Algorithm-in-the-loop decision-making “Algorithm-in-the-loop” decision-making is commonly characterized by frst having an algorithm-produced prediction or classif-cation, with a human making the fnal decision after considering an algorithmic recommendation [38, 85].”
BackgroundMethodInfluentialLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 2×
“Some studies have demonstrated that combinations of human and algorithmic judgment can improve prediction and/or decision-making (e.g., [6, 23, 43, 52, 57, 71]).”
BackgroundProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307 · cited 2×
“27 Second, as discussed, measuring prediction accuracy is limited by the use of proxy outcomes (re-referral or placement) that do not align with the outcomes that humans are predicting [8, 23].”
BackgroundDisparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275
The Flaws of Policies Requiring Human Oversight of Government Algorithms
Green, B. (2021) · doi:10.2139/ssrn.3921216 · Cited by 259
Human–machine partnership with artificial intelligence for chest radiograph diagnosis
Patel, B. N., Rosenberg, L., Willcox, G., Baltaxe, D., Lyons, M., Irvin, J., Rajpurkar, P., Amrhein, T., Gupta, R., Halabi, S., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0189-7 · Cited by 254
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247 · cited 1×
“Recent years have seen the deployment of AI-based tools either to augment or replace human judgments across a growing range of high-impact decision-making contexts, such as social work, criminal justice, hiring, healthcare, and education [11, 15, 38, 43, 44, 58, 72, 95, 98].”
BackgroundStudent Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235 · cited 3×
“In some contexts, human-AI collaboration has demonstrated potential to improve fairness and efectiveness of decision-making, compared with human or AI decision-making alone, e.g. [6 , 23, 43, 52, 71].”
BackgroundSelf-citationA Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 10×
“Another possible approach to improving fairness in decision-making may be through human-AI collaborations , aimed at combining strengths and mitigating limitations in both AI-based and human decisions [23, 42, 60, 86, 97].”
BackgroundResultInfluentialValue-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Self-citationWang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 1×
“Recent years have seen the deployment of AI-based tools either to augment or replace human judgments across a growing range of high-impact decision-making contexts, such as social work, criminal justice, hiring, healthcare, and education [11, 15, 38, 43, 44, 58, 72, 95, 98].”
BackgroundSelf-citationAssessing Risk Assessment in Action
Stevenson, M. (2018) · Minnesota Law Review · doi:10.24926/265535.958 · Cited by 193 · cited 2×
“For example, in a real-world pretrial criminal context, human-AI collaboration was shown to exacerbate discriminatory decision-making [3, 84].”
BackgroundStreet-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
Alkhatib, A., & Bernstein, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300760 · Cited by 188 · cited 1×
“Workers conduct holistic, contextual assessments when they consider any piece of information in a report to be relevant only when it’s relevant to risk and safety in the context of all other information in the report (cf. [4]).”
BackgroundDesigning for human–AI complementarity in K‐12 education
Holstein, K., & Aleven, V. (2022) · AI Magazine · doi:10.1002/aaai.12058 · Cited by 156
Self-citationSaxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476089 · Cited by 134 · cited 1×
“For a case study of other kinds of RAIs and algorithms used in child welfare, see [80].”
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123 · cited 1×
“Many studies have focused on prediction accuracy, fnding that on many tasks, algorithms can outperform humans in terms of prediction accuracy [8, 37, 39, 56, 57, 99].”
BackgroundWilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118
A Human-Centered Review of Algorithms used within the U.S. Child Welfare System
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376229 · Cited by 116
Human Decision Making with Machine Assistance: An Experiment on Bailing and Jailing
Grgić-Hlača, N., Engel, C., & Gummadi, K. P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359280 · Cited by 113 · cited 1×
“Many studies have focused on prediction accuracy, fnding that on many tasks, algorithms can outperform humans in terms of prediction accuracy [8, 37, 39, 56, 57, 99].”
BackgroundShifting Concepts of Value: Designing Algorithmic Decision-Support Systems for Public Services
Holten Møller, N., Shklovski, I., & Hildebrandt, T. T. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420149 · Cited by 97
What can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92
Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer
Madras, D., Pitassi, T., & Zemel, R. (2017) · arXiv · doi:10.48550/arxiv.1711.06664 · Cited by 90
Tolan, S., Miron, M., Gómez, E., & Castillo, C. (2019) · Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law · doi:10.1145/3322640.3326705 · Cited by 75
Cited by
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“Design probes: [279, 303] Participatory design: [165] Field study: [114, 145, 148, 321] [26, 28, 44, 53, 62, 72, 103, 104, Design workshops: Co-design: [15, 19, 31, 117] Narrative study: [292] 111, 156, 162, 164, 175, 180, 187, [11, 216, 277, 282] Research through design: Mixed methods: 194, 231, 248, 253, 254, 261, 2…”
BackgroundInfluentialGhosting the Machine: Judicial Resistance to a Recidivism Risk Assessment Instrument
Pruss, D. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3593999 · Cited by 31
Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making
Guerdan, L., Coston, A., Wu, Z. S., & Holstein, K. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594036 · Cited by 28
Self-citationBuilding, Shifting, & Employing Power: A Taxonomy of Responses From Below to Algorithmic Harm
DeVrio, A., Eslami, M., & Holstein, K. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658958 · Cited by 21
Self-citationHeterogeneity in Algorithm-Assisted Decision-Making: A Case Study in Child Abuse Hotline Screening
Cheng, L., & Chouldechova, A. (2022) · arXiv · doi:10.48550/arxiv.2204.05478 · Cited by 20
Metadata
- Authors
- Cheng, H.-F., Stapleton, L., Kawakami, A., Sivaraman, V., Cheng, Y., Qing, D., Perer, A., Holstein, K., Wu, Z. S., & Zhu, H.
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-27
- Pages
- 1-22
- OpenAlex
- W4225087473
- Cited by
- 85
- References
- 40
- FWCI
- 17.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.