Rethinking "Risk" in Algorithmic Systems Through A Computational Narrative Analysis of Casenotes in Child-Welfare
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Risk assessment algorithms are being adopted by public sector agencies to make high-stakes decisions about human lives. Algorithms model “risk” based on individual client characteristics to identify clients most in need. However, this understanding of risk is primarily based on easily quantifiable risk factors that present an incomplete and biased perspective of clients. We conducted a computational narrative analysis of child-welfare casenotes and draw attention to deeper systemic risk factors that are hard to quantify but directly impact families and street-level decision-making. We found that beyond individual risk factors, the system itself poses a significant amount of risk where parents are over-surveilled by caseworkers and lack agency in decision-making processes. We also problematize the notion of risk as a static construct by highlighting the temporality and mediating effects of different risk, protective, systemic, and procedural factors. Finally, we draw caution against using casenotes in NLP-based systems by unpacking their limitations and biases embedded within them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 2×
“[126] refer to this as the portability trap – “Failure to understand how repurposing algorithmic solutions designed for one social context may be misleading, inaccurate, or otherwise do harmwhen applied to a different context”.”
BackgroundThe Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility
Ackerman, M. S. (2000) · Human–Computer Interaction · doi:10.1207/s15327051hci1523_5 · Cited by 892
Mitigating Gender Bias in Natural Language Processing: Literature Review
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1159 · Cited by 492 · cited 1×
“Recent studies in NLP have examined text datasets and seed lexicons and found that social hierarchies and racial, cultural, and cognitive biases can become embedded in and amplifed by NLP systems and lead to signifcant disparities in downstream tasks [12, 24, 30, 50, 134].”
BackgroundVeale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2021) · arXiv · doi:10.48550/arxiv.2106.15590 · Cited by 317
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533083 · Cited by 317
Machine Learning for Sociology
Molina, M., & Garip, F. (2019) · Annual Review of Sociology · doi:10.1146/annurev-soc-073117-041106 · Cited by 295 · cited 1×
“This study responds to calls within SIGCHI research to investigate complex sociotechnical systems from both a qualitative and quantitative lens to understand the opportunities and limitations of computational research towards highlighting social problems and addressing injustices [3, 15, 99].”
BackgroundDigital discretion: A systematic literature review of ICT and street-level discretion
Busch, P. A., & Henriksen, H. Z. (2018) · Information Polity · doi:10.3233/ip-170050 · Cited by 267
Young, M. M., Bullock, J. B., & Lecy, J. D. (2019) · Perspectives on Public Management and Governance · doi:10.1093/ppmgov/gvz014 · Cited by 255
Brown, A., Chouldechova, A., Putnam-Hornstein, E., Tobin, A., & Vaithianathan, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300271 · Cited by 207
Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
Alkhatib, A., & Bernstein, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300760 · Cited by 188
Gambrill, E., & Shlonsky, A. (2000) · Children and Youth Services Review · doi:10.1016/s0190-7409(00)00123-7 · Cited by 187
Zhang, A., Boltz, A., Wang, C. W., & Lee, M. K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501866 · Cited by 166 · cited 1×
“scholars have studied issues of citizen engagement in the public space [46, 85], citizen activism [95], empowerment of affected communities [27, 107], centering worker well-being in gig work [118, 139, 140, 145], and engaging in action research with community partners [38].”
BackgroundKawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · cited 4×
“[78] on AFST showed that there were misalignments between AFST’s predictive target and call screeners’ decision-making objective where call screeners relied more on their contextual understanding of the family and risk factors to make decisions rather than empirical risk as predicted by AFST.”
BackgroundResultInfluentialAndrejevic, M. (2019) · Surveillance & Society · doi:10.24908/ss.v17i1/2.12930 · Cited by 142 · cited 2×
““Risk” has been one of the core organizing principles in this economic shift in governance where administrative data accumulated by government agencies about citizens purportedly allows them to preemptively recognize clients in the riskiest circumstances [10, 31], i.”
BackgroundSaxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476089 · Cited by 134 · cited 3×
“We partnered with a child-welfare agency in a metropolitan area in a Midwestern U.S. state that is part of the broader child-welfare system that was recently investigated by Saxena et al. (2021) [120].”
BackgroundSelf-citationCharting the Landscape of Online Cryptocurrency Manipulation
Nizzoli, L., Tardelli, S., Avvenuti, M., Cresci, S., Tesconi, M., & Ferrara, E. (2020) · IEEE Access · doi:10.1109/access.2020.3003370 · Cited by 132
Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry
Benjamin, J. J., Berger, A., Merrill, N., & Pierce, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445481 · Cited by 130
A Human-Centered Review of Algorithms used within the U.S. Child Welfare System
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376229 · Cited by 116
Self-citationThe need for comprehensive risk management systems in child welfare
Gambrill, E., & Shlonsky, A. (2001) · Children and Youth Services Review · doi:10.1016/s0190-7409(00)00124-9 · Cited by 103 · cited 1×
“cessive investigations, and 3) prior work has established that empirical knowledge in child-welfare is quite limited and there is still a signifcant amount of debate regarding which risk factors (when taken together) lead to the accumulation of risk and which protective factors help mediate these risks [58, 121, 128].”
BackgroundShifting Concepts of Value: Designing Algorithmic Decision-Support Systems for Public Services
Holten Møller, N., Shklovski, I., & Hildebrandt, T. T. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420149 · Cited by 97
Street-Level Algorithms and AI in Bureaucratic Decision-Making: A Caseworker Perspective
Ammitzbøll Flügge, A., Hildebrandt, T., & Møller, N. H. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449114 · Cited by 83
Robertson, S., Nguyen, T., & Salehi, N. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445748 · Cited by 67
Carnochan, S., Moore, M., & Austin, M. J. (2013) · Journal of Evidence-Based Social Work · doi:10.1080/15433714.2013.788953 · Cited by 50 · cited 25×
“The exemplar casenote below depicts how caseworkers and parents work together in resolving such risks within the restrictive framework of CWS. Topic 2 example: MS [parent] stated she lost her job and will be without a job by the end of the month.”
BackgroundInfluentialBadillo-Urquiola, K., Page, X., & Wisniewski, P. J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300497 · Cited by 49
Bureaucracy as a Lens for Analyzing and Designing Algorithmic Systems
Pääkkönen, J., Nelimarkka, M., Haapoja, J., & Lampinen, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376780 · Cited by 35 · cited 3×
“On the other hand, HCI scholars have recommended that we engage with uncertainties as opportunities for human-centered design rather than treat them as obstacles [22, 105, 131].”
BackgroundConducting Participatory Design to Improve Algorithms in Public Services: Lessons and Challenges
Saxena, D., & Guha, S. (2020) · Companion Publication of the 2020 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3406865.3418331 · Cited by 30
Self-citationFedoravicius, N., McMillen, J. C., Rowe, J. E., Kagotho, N., & Ware, N. C. (2008) · Social Service Review · doi:10.1086/589780 · Cited by 27
Child Welfare System: Interaction of Policy, Practice and Algorithms
Saxena, D., Badillo-Urquiola, K., Wisniewski, P., & Guha, S. (2020) · Companion of the 2020 ACM International Conference on Supporting Group Work · doi:10.1145/3323994.3369888 · Cited by 16
Self-citationForks over knives: Predictive inconsistency in criminal justice algorithmic risk assessment tools
Greene, T., Shmueli, G., Fell, J., Lin, C.-F., & Liu, H.-W. (2022) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12966 · Cited by 15
Forks Over Knives: Predictive Inconsistency in Criminal Justice Algorithmic Risk Assessment Tools
Greene, T., Shmueli, G., Fell, J., Lin, C.-F., & Liu, H.-W. (2020) · arXiv · doi:10.48550/arxiv.2012.00289 · Cited by 8
Cited by
Kawakami, A., Coston, A., Zhu, H., Heidari, H., & Holstein, K. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642849 · Cited by 54
Examining risks of racial biases in NLP tools for child protective services
Field, A., Coston, A., Gandhi, N., Chouldechova, A., Putnam-Hornstein, E., Steier, D., & Tsvetkov, Y. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594094 · Cited by 18
“While much literature has critiqued risk assessment systems [18, 53], little work has explored the impact of NLP in these tools, nor potential biases in assumedly more benign information extraction systems.”
BackgroundMethodInfluentialTang, X., Lima, G., Jiang, L., Simko, L., & Zou, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710935 · Cited by 17
Oguine, O. C., Park, J. K., Akter, M., Olesk, J., Alluhidan, A., Wisniewski, P., & Badillo-Urquiola, K. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710995 · Cited by 8
Metadata
- Authors
- Saxena, D., Moon, E. S.-Y., Chaurasia, A., Guan, Y., & Guha, S.
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-19
- OpenAlex
- W4366547917
- Cited by
- 34
- References
- 32
- FWCI
- 8.77
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.