Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Child welfare agencies across the United States are turning to data-driven predictive technologies (commonly called predictive analytics) which use government administrative data to assist workers’ decision-making. While some prior work has explored impacted stakeholders’ concerns with current uses of data-driven predictive risk models (PRMs), less work has asked stakeholders whether such tools ought to be used in the first place. In this work, we conducted a set of seven design workshops with 35 stakeholders who have been impacted by the child welfare system or who work in it to understand their beliefs and concerns around PRMs, and to engage them in imagining new uses of data and technologies in the child welfare system. We found that participants worried current PRMs perpetuate or exacerbate existing problems in child welfare. Participants suggested new ways to use data and data-driven tools to better support impacted communities and suggested paths to mitigate possible harms of these tools. Participants also suggested low-tech or no-tech alternatives to PRMs to address problems in child welfare. Our study sheds light on how researchers and designers can work in solidarity with impacted communities, possibly to circumvent or oppose child welfare agencies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Posner, K., Brown, G. K., Stanley, B., Brent, D. A., Yershova, K. V., Oquendo, M. A., Currier, G. W., Melvin, G. A., Greenhill, L., Shen, S., et al. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.10111704 · Cited by 5,157 · cited 2×
“P17 praised the ColumbiaSuicide Severity Rating Scale (C-SSRS) [97, 98]: “There are some yes or no answers and it’s not about ‘Oh, I want to get this kid 5150ed,’20it’s”
MethodDwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018) · Nature · doi:10.1038/s41586-018-0637-6 · Cited by 1,859
Muller, M. J., & Kuhn, S. (1993) · Communications of the ACM · doi:10.1145/153571.255960 · Cited by 909
Participatory design and "democratizing innovation"
Björgvinsson, E., Ehn, P., & Hillgren, P.-A. (2010) · Proceedings of the 11th Biennial Participatory Design Conference · doi:10.1145/1900441.1900448 · Cited by 709
Lifetime Prevalence of Investigating Child Maltreatment Among US Children
Kim, H., Wildeman, C., Jonson-Reid, M., & Drake, B. (2017) · American Journal of Public Health · doi:10.2105/ajph.2016.303545 · Cited by 640 · cited 1×
“PRMs have been deployed in response to racial biases and disparities [38, 70], inaccurate and inconsistent decisions, child fatalities [73], etc.”
BackgroundHarrington, C., Erete, S., & Piper, A. M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359318 · Cited by 612
Moral crumple zones: cautionary tales in human–robot interaction
Elish, M. C. (2019) · Engaging Science, Technology, and Society · doi:10.17351/ests2019.260 · Cited by 321
The Flaws of Policies Requiring Human Oversight of Government Algorithms
Green, B. (2021) · doi:10.2139/ssrn.3921216 · Cited by 259
Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235 · cited 1×
“abuse? AI-based predictive algorithms are being used or are being considered for use across all of these everyday public sector decisions [19, 28, 56, 93, 126].”
MethodSelf-citationA Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 2×
“Others argue more automation can worsen disparities and decision quality [26, 35, 45].”
BackgroundSelf-citationValue-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Self-citationBrown, A., Chouldechova, A., Putnam-Hornstein, E., Tobin, A., & Vaithianathan, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300271 · Cited by 207
Self-citationAssessing Risk Assessment in Action
Stevenson, M. (2018) · Minnesota Law Review · doi:10.24926/265535.958 · Cited by 193
Democratizing Algorithmic Fairness
Wong, P.-H. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00355-w · Cited by 182
Kawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · cited 8×
“Others note that automation is a spectrum: CPS agencies can pressure workers to conform to PRMs’ recommendations in some cases more than others [26, 67].”
BackgroundMethodResultInfluentialSelf-citationSaxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476089 · Cited by 134 · cited 3×
“However, prior work may have been limited in what kinds of choices they put “on the table” for stakeholders, because they worked with CPS agencies, which are either mandated to use, or have already chosen to use, algorithms [36, 114].”
BackgroundA Human-Centered Review of Algorithms used within the U.S. Child Welfare System
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376229 · Cited by 116
Shifting Concepts of Value: Designing Algorithmic Decision-Support Systems for Public Services
Holten Møller, N., Shklovski, I., & Hildebrandt, T. T. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420149 · Cited by 97
ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia
Halfaker, A., & Geiger, R. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415219 · Cited by 82
Smith, C. E., Yu, B., Srivastava, A., Halfaker, A., Terveen, L., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376783 · Cited by 77 · cited 1×
“Others intentionally work with specific groups who are most impacted by these technologies, yet still do not empower impacted stakeholders to engage in broader design decisions [20, 23, 27, 52, 57, 115, 119, 119].”
BackgroundSelf-citationIdentifying Police Officers at Risk of Adverse Events
Carton, S., Helsby, J., Joseph, K., Mahmud, A., Park, Y., Walsh, J., Cody, C., Patterson, C. E., Haynes, L., & Ghani, R. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939698 · Cited by 64
Algorithmic Risk Assessment in the Hands of Humans
Stevenson, M., & Doleac, J. L. (2020) · doi:10.2139/ssrn.3513695 · Cited by 42
Conducting Participatory Design to Improve Algorithms in Public Services: Lessons and Challenges
Saxena, D., & Guha, S. (2020) · Companion Publication of the 2020 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3406865.3418331 · Cited by 30
Cited by
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“Design probes: [279, 303] Participatory design: [165] Field study: [114, 145, 148, 321] [26, 28, 44, 53, 62, 72, 103, 104, Design workshops: Co-design: [15, 19, 31, 117] Narrative study: [292] 111, 156, 162, 164, 175, 180, 187, [11, 216, 277, 282] Research through design: Mixed methods: 194, 231, 248, 253, 254, 261, 2…”
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
“The term “accountability”, for example, is mentioned in over 33% of ecosystem audit abstracts, the highest of any category. is goal, These studies often utilise a wider range of methods, including qualitative interviews, surveys, workshops, and literature reviews [68, 69, 70].”
BackgroundHarms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149
“2022], and public services [Black et al. 2022; Loi and Spielkamp 2021; Stapleton et al. 2022], ADM can often exhibit similar kinds of diffuse and long-term harms to those we discuss coming from increased agency.”
BackgroundKawakami, A., Coston, A., Zhu, H., Heidari, H., & Holstein, K. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642849 · Cited by 54
“The Situate AI Guidebook synthesizes these key considerations into a toolkit to scaffold early-stage deliberations around whether and under what conditions to move forward with developing or deploying a proposed public sector AI tool.”
BackgroundMethodResultInfluentialSelf-citation
Metadata
- Authors
- Stapleton, L., Lee, M. H., Qing, D., Wright, M., Chouldechova, A., Holstein, K., Wu, Z. S., & Zhu, H.
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 1162-1177
- OpenAlex
- W4280569846
- Cited by
- 76
- References
- 25
- FWCI
- 15.92
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.