CoChat
Sign Up
PreprintCited by 23Ethics and Social Impacts of AIAdversarial Robustness in Machine LearningLaw, AI, and Intellectual Property

AI red-teaming is a sociotechnical problem: on values, labor, and harms

Open access
Gillespie, Tarleton, Shaw, Ryland, Gray, Mary L., Suh, Jina·2024·arXiv
DOI 10.48550/arxiv.2412.09751arXiv 2412.09751

Preprint — published as 10.1145/3731657

Open this paper in CoChat — free account

Abstract

As generative AI technologies find more and more real-world applications, the importance of testing their performance and safety seems paramount. "Red-teaming" has quickly become the primary approach to test AI models--prioritized by AI companies, and enshrined in AI policy and regulation. Members of red teams act as adversaries, probing AI systems to test their safety mechanisms and uncover vulnerabilities. Yet we know far too little about this work or its implications. This essay calls for collaboration between computer scientists and social scientists to study the sociotechnical systems surrounding AI technologies, including the work of red-teaming, to avoid repeating the mistakes of the recent past. We highlight the importance of understanding the values and assumptions behind red-teaming, the labor arrangements involved, and the psychological impacts on red-teamers, drawing insights from the lessons learned around the work of content moderation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.09751

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Researchers caution that involving marginalized communities in elaborating on their marginalization for red-teaming can become transactional, extractive and exploitative, and warn of emotional and mental health costs of this type of human labor.

    “However, researchers caution that involving marginalized communities in elaborating on their marginalization for red-teaming, can become transactional, extractive and exploitative (cf. [29, 46]); and warn of the emotional and mental health costs of this type of human 'labor'.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [46]

  • Other trends in AI evaluation include red-teaming, whereby image generation models or systems are deliberately tested with adversarial prompts to uncover vulnerabilities and bias, surfacing offensive or harmful content for error detection or removal.

    “Other trends in AI evaluation include red-teaming, whereby image generation models or systems are deliberately tested with adversarial prompts to uncover their 'vulnerabilities' and 'bias'; surfacing offensive or harmful content for purposes of error detection, or removal [46].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [46]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 20

  1. [1][1] Ajunwa, I., Crawford, K., and Schultz, J. 2017. Limitless worker surveillance. California Law Review (2017). -
  2. [2][2] Arsht, A. and Etcovitch, D. 2018. The human cost of online content moderation. Harvard Journal of Law and Technology 2 (2018). -
  3. [3][3] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. 2022. Constitutional AI: Harmlessness from AI Feedback. http://arxiv.org/abs/2212.08073 arXiv:2212.08073 [cs]. - · arXiv
  4. [6][6] Carmi, E. 2019. The Hidden Listeners: Regulating the Line from Telephone Operators to Content Moderators. International Journal of Communication 13 (2019), 440-458. -
  5. [10][10] Deng, W. H., Yurrita, M., Díaz, M., Suh, J., Judd, N., Groves, L., Shen, H., Eslami, M., and Holstein, K. 2024. Responsible Crowdsourcing for Responsible Generative AI: Engaging Crowds in AI Auditing and Evaluation. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 12 (Oct. 2024), 148-150. https://doi.org/10.1609/hcomp.v12i1.31609 - · DOI
  6. [11][11] Feffer, M., Sinha, A., Lipton, Z. C., and Heidari, H. 2024. Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv preprint arXiv:2401.15897 (2024). - · arXiv
  7. [13][13] Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., DasSarma, N., Drain, D., Elhage, N., El-Showk, S., Fort, S., Hatfield-Dodds, Z., Henighan, T., Hernandez, D., Hume, T., Jacobson, J., Johnston, S., Kravec, S., Olsson, C., Ringer, S., Tran-Johnson, E., Amodei, D., Brown, T., Joseph, N., McCandlish, S., Olah, C., Kaplan, J., and Clark, J. 2022. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. http://arxiv.org/abs/2209.07858 arXiv:2209.07858 [cs]. - · arXiv
  8. [15][15] Gillespie, T. 2018. Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media . Yale University Press, New Haven. -
  9. [16][16] Gillespie, T. 2024. Generative AI and the politics of visibility. Big Data & Society 11, 2 (June 2024), 20539517241252131. https://doi.org/10.1177/ 20539517241252131 Publisher: SAGE Publications Ltd. -
  10. [18][18] Hao, S., Kumar, P., Laszlo, S., Poddar, S., Radharapu, B., and Shelby, R. 2023. Safety and Fairness for Content Moderation in Generative Models. http://arxiv.org/abs/2306.06135 arXiv:2306.06135 [cs]. - · arXiv
  11. [19][19] Inie, N., Stray, J., and Derczynski, L. 2023. Summon a demon and bind it: A grounded theory of llm red teaming in the wild. arXiv preprint arXiv:2311.06237 (2023). - · arXiv
  12. [20][20] Joh, E. E. 2009. Breaking the Law to Enforce it: Undercover Police Participation in Crime. Stanford Law Review 62 (2009), 155. https://api. semanticscholar.org/CorpusID:152308574 -
  13. [22][22] Klonick, K. 2018. The New Governors: The People, Rules and Processes Governing Online Speech. Harvard Law Review 131 (2018), 73. -
  14. [26][26] Metcalf, J. and Singh, R. 2023. Scaling Up Mischief: Red-Teaming AI and Distributing Governance. Harvard Data Science Review Special Issue 4 (Dec. 2023). https://doi.org/10.1162/99608f92.ff6335af - · DOI
  15. [31][31] Roberts, S. T. 2016. Commercial Content Moderation: Digital Laborers' Dirty Work. In Intersectional Internet: Race, Sex, Class and Culture Online , Safiya Umoja Noble and Brendesha Tynes (Eds.). Peter Lang Publishing Inc., New York, 147-159. -
  16. [32][32] Roberts, S. T. 2019. Behind the Screen: Content Moderation in the Shadows of Social Media . Yale University Press, New Haven. -
  17. [35][35] Shay, J. 2014. Moral injury. Psychoanalytic psychology 31, 2 (2014), 182. -
  18. [37][37] Steiger, M., Bharucha, T. J., Venkatagiri, S., Riedl, M. J., and Lease, M. 2021. The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021). -
  19. [39][39] Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., Gabriel, I., Rieser, V., and Isaac, W. 2023. Sociotechnical Safety Evaluation of Generative AI Systems. http://arxiv.org/abs/2310.11986 arXiv:2310.11986 [cs]. - · arXiv
  20. [41][41] Zhang, A. Q., Amores, J., Gray, M. L., Czerwinski, M., and Suh, J. 2024. AURA: Amplifying Understanding, Resilience, and Awareness for Responsible AI Content Work. arXiv preprint arXiv:2411.01426 (2024). - · arXiv

Cited by

Sorted by citations

Related

Authors
Gillespie, T., Shaw, R., Gray, M. L., & Suh, J.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
69
Issue
2
Pages
88-95
ISSN
0001-0782 (ISSN-L)
DOI
10.48550/arxiv.2412.09751
arXiv
2412.09751
OpenAlex
W4405432888
Cited by
6
References
0
Citations by year
2026: 5 · 2025: 1
FWCI
28.86
Type
journal article
Language
en
Open access
Licence
Locations
  • Communications of the ACM (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Law, AI, and Intellectual Property
Keywords
AI red teaming, generative AI safety, socio-technical systems, content moderation, AI policy and regulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review