Cited by 21Ethics and Social Impacts of AIArtificial Intelligence in LawExplainable Artificial Intelligence (XAI)

Examining risks of racial biases in NLP tools for child protective services

Open access
Field, Anjalie, Coston, Amanda, Gandhi, Nupoor, Chouldechova, Alexandra, Putnam-Hornstein, Emily, Steier, David, Tsvetkov, Yulia·2023·2023 ACM Conference on Fairness, Accountability, and Transparency

Abstract

Although much literature has established the presence of demographic bias in natural language processing (NLP) models, most work relies on curated bias metrics that may not be reflective of real-world applications. At the same time, practitioners are increasingly using algorithmic tools in high-stakes settings, with particular recent interest in NLP. In this work, we focus on one such setting: child protective services (CPS). CPS workers often write copious free-form text notes about families they are working with, and CPS agencies are actively seeking to deploy NLP models to leverage these data. Given well-established racial bias in this setting, we investigate possible ways deployed NLP is liable to increase racial disparities. We specifically examine word statistics within notes and algorithmic fairness in risk prediction, coreference resolution, and named entity recognition (NER). We document consistent algorithmic unfairness in NER models, possible algorithmic unfairness in coreference resolution models, and little evidence of exacerbated racial bias in risk prediction. While there is existing pronounced criticism of risk prediction, our results expose previously undocumented risks of racial bias in realistic information extraction systems, highlighting potential concerns in deploying them, even though they may appear more benign. Our work serves as a rare realistic examination of NLP algorithmic fairness in a potential deployed setting and a timely investigation of a specific risk associated with deploying NLP in CPS settings.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3593013.3594094

Cites

56 of 57 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generative AI outputs tend to reproduce normative views and mirror societal biases for marginalized groups, including people with disabilities.

    “Generative AI outputs tend to reproduce normative views and mirror societal biases for marginalized groups [15, 21, 36, 41, 42, 45, 53, 66, 72, 91], including PwD [10, 12, 39, 48, 54, 62, 77, 113].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [36]

    The Abstract says NLP models can absorb and amplify data biases, including biases related to race and disability. But the Discussion does not establish that generative AI outputs tend to reproduce normative views or societal biases; it calls for evaluating generative models for racial bias if they are deployed. The source therefore supports the broader bias concern, not the claim specifically about generative outputs. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 58 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% Solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers. Association for Computational Linguistics, New York City, USA, 57-60. https: //aclanthology.org/N06-2015

Cited by

Sorted by citations

Authors
Field, A., Coston, A., Gandhi, N., Chouldechova, A., Putnam-Hornstein, E., Steier, D., & Tsvetkov, Y.
Affiliations
  • Johns Hopkins University (US)
  • Carnegie Mellon University (US)
  • School of Social Work, University of North Carolina at Chapel Hill, USA
  • University of Washington (US)
Venue
2023 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2023-06-12
Pages
1479-1492
OpenAlex
W4379087118
Cited by
18
References
58
Citations by year
2026: 3 · 2025: 6 · 2024: 8 · 2023: 1
FWCI
4.51
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • Allegheny County Department of Human Services
  • NSF (National Science Foundation) (IIS2142739, DGE1745016)
  • Alfred P. Sloan Foundation
  • Block Center for Technology and Society at Carnegie Mellon University
  • Google (PhD Fellowship)
  • Meta (PhD Fellowship)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
racial bias, algorithmic fairness, child protection services, natural language processing, named entity recognition, coreference resolution, risk prediction, racial disparities, information extraction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review