CoChat
Sign Up
PreprintCited by 130Adversarial Robustness in Machine LearningEthics and Social Impacts of AIHate Speech and Cyberbullying Detection

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Open access
Ganguli, Deep, Lovitt, Liane, Kernion, Jackson, Askell, Amanda, Bai, Yuntao, Kadavath, Saurav, Mann, Ben, Perez, Ethan +28 more·2022·arXiv
DOI 10.48550/arxiv.2209.07858arXiv 2209.07858
Open this paper in CoChat — free accountOpen PDF

Abstract

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2209.07858

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Red-teaming remains inconsistently defined and often lacks transparency about scope, methods and coverage.

    “Recent work has noted that red-teaming remains inconsistently defined and often lacks transparency about scope, methods and coverage (Ganguli et al. 2022; Feffer et al. 2024).”

    From Generative AI in child sexual exploitation and abuse: views from UK law enforcement (2026) · cites this paper as Ganguli et al. 2022

    Section 5.2 (pp. 14–15) says organizations have incentives not to share red-teaming findings and calls for more detailed public discussion of methods and shared practices. This supports the concern about transparency, but the paper does not establish that red-teaming is inconsistently defined or that reporting commonly omits its scope and coverage. Section 5.1 (p. 14) discusses incomplete coverage of the authors’ own attacks, a narrower point. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Jailbreaking cannot cover all, or even most, instances of harm, and the input data can be misused by malicious users.

    “While jailbreaking is commonly used in the task of red teaming language models to identify, measure, and reduce harmful output, the method cannot cover all, or even most, instances of harm, and the input data can be misused by malicious users (Ganguli et al., 2022).”

    From Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback (2025) · cites this paper as Ganguli et al., 2022

    The dataset-use discussion (p. 25) says the red-team dataset is not comprehensive of all possible harms or attacks and could be used to train models that produce more harmful responses. The Limitations section (p. 14) also describes missed attack types. Neither passage establishes that jailbreaking misses “even most” harms; the paper discusses red teaming and its dataset rather than measuring jailbreaking’s coverage. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The methodology is well grounded in established risk foresight and validation practice.

    “This methodology is well grounded in established risk foresight and validation practice (Perez et al. 2022;Ganguli et al. 2022;Fröhling et al. 2026;Buçinca et al. 2023;Rao et al. 2025;Herdel et al. 2025;Wang et al. 2024b)”

    From Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents (2026) · cites this paper as Ganguli et al. 2022

    The Introduction (p. 1) describes red teaming as adversarially probing models for harmful outputs to measure and improve safety, and §3.1 (p. 4) says the authors incorporated Trust & Safety best practices into the task design. But the paper calls this an early approach and says shared red-teaming norms remain to be developed (§5.2, pp. 14–15). It does not establish that the unspecified methodology is well grounded in established risk-foresight and validation practice. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Current approaches to validating frontier AI models prior to deployment rely on independent testing and red-teaming.

    “Current approaches to validating frontier AI models prior to deployment lean on independent testing and red-teaming (as e.g.Ziegler et al., 2022;Perez et al., 2022;Ganguli et al., 2022;Ouyang et al., 2022;Khlaaf, 2023).”

    From Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems (2024) · cites this paper as Ganguli et al., 2022

    The Introduction (p. 1) describes manual red-teaming used to measure language-model safety, supporting red-teaming as a validation approach. The paper does not establish that frontier-model validation generally relies on independent testing before deployment; Limitations and Future Work (p. 14) presents third-party red-teaming as a possibility rather than a practice in this study. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Another way to build challenging benchmarks is to gather questions that are more likely to lead to hallucinations.

    “In addition to long-tail knowledge, another way to build challenging benchmarks is to gather questions that are more likely to lead to hallucinations - a methodology similar to adversarial prompting [129, 329] and red teaming [76].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [76]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 130

Research works printed but not linked: 55

  1. [1][1] A. Abid, M. Farooqi, and J. Zou. Large language models associate Muslims with violence. Nature Machine Intelligence , 3(6):461-463, June 2021. Number: 6 Publisher: Nature Publishing Group. -
  2. [2][2] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan. A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861 [cs] , Dec. 2021. arXiv: 2112.00861. - · arXiv
  3. [4][4] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. HatfieldDodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Apr. 2022. Number: arXiv:2204.05862 arXiv:2204.05862 [cs]. - · arXiv
  4. [6][6] M. Bartolo, T. Thrush, R. Jia, S. Riedel, P. Stenetorp, and D. Kiela. Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 8830-8848, 2021. arXiv:2104.08678 [cs]. - · arXiv
  5. [7][7] C. Basta, M. R. Costa-jussà, and N. Casas. Evaluating the Underlying Gender Bias in Contextualized Word Embeddings. arXiv:1904.08783 [cs] , Apr. 2019. arXiv: 1904.08783. - · arXiv
  6. [8][8] D. Bates, M. Mächler, B. Bolker, and S. Walker. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software , 67(1):1-48, 2015. -
  7. [9][9] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, pages 610-623, New York, NY, USA, Mar. 2021. Association for Computing Machinery. -
  8. [10][10] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang. On the Opportunities and Risks of Foundation Models. arXiv:2108.07258 [cs] , Aug. 2021. arXiv: 2108.07258. - · arXiv
  9. [11][11] M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong, T. Maharaj, P. W. Koh, S. Hooker, J. Leung, A. Trask, E. Bluemke, J. Lebensold, C. O'Keefe, M. Koren, T. Ryffel, J. B. Rubinovitz, T. Besiroglu, F. Carugati, J. Clark, P. Eckersley, S. de Haas, M. Johnson, B. Laurie, A. Ingerman, I. Krawczuk, A. Askell, R. Cammarota, A. Lohn, D. Krueger, C. Stix, P. Henderson, L. Graham, C. Prunkl, B. Martin, E. Seger, N. Zilberman, S. O. hÉigeartaigh, F. Kroeger, G. Sastry, R. Kagan, A. Weller, B. Tse, E. Barnes, A. Dafoe, P. Scharre, A. Herbert-Voss, M. Rasser, S. Sodhani, C. Flynn, T. K. Gilbert, L. Dyer, S. Khan, Y. Bengio, and M. Anderljung. Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. arXiv:2004.07213 [cs] , Apr. 2020. arXiv: 2004.07213. - · arXiv
  10. [13][13] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting Training Data from Large Language Models. arXiv:2012.07805 [cs] , June 2021. arXiv: 2012.07805. - · arXiv
  11. [14][14] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep Reinforcement Learning from Human Preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. -
  12. [15][15] B. Dang, M. J. Riedl, and M. Lease. But Who Protects the Moderators? The Case of Crowdsourced Image Moderation, Jan. 2020. arXiv:1804.10999 [cs]. - · arXiv
  13. [16][16] A. Das, B. Dang, and M. Lease. Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , 8(1):33-42, Oct. 2020. -
  14. [17][17] E. Diener, D. Wirtz, W. Tov, C. Kim-Prieto, D.-w. Choi, S. Oishi, and R. Biswas-Diener. New Wellbeing Measures: Short Scales to Assess Flourishing and Positive and Negative Feelings. Social Indicators Research , 97(2):143-156, June 2010. -
  15. [18][18] E. Dinan, G. Abercrombie, A. S. Bergman, S. Spruit, D. Hovy, Y.-L. Boureau, and V. Rieser. Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling. arXiv:2107.03451 [cs] , July 2021. arXiv: 2107.03451. - · arXiv
  16. [19][19] E. Dinan, A. Fan, A. Williams, J. Urbanek, D. Kiela, and J. Weston. Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 8173-8188, Online, Nov. 2020. Association for Computational Linguistics. -
  17. [20][20] E. Dinan, S. Humeau, B. Chintagunta, and J. Weston. Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack, Aug. 2019. arXiv:1908.06083 [cs]. - · arXiv
  18. [21][21] L. Dixon, J. Li, J. Sorensen, N. Thain, and L. Vasserman. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , AIES '18, pages 67-73, New York, NY, USA, Dec. 2018. Association for Computing Machinery. -
  19. [22][22] D. Ganguli, D. Hernandez, L. Lovitt, N. DasSarma, T. Henighan, A. Jones, N. Joseph, J. Kernion, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, N. Elhage, S. E. Showk, S. Fort, Z. HatfieldDodds, S. Johnston, S. Kravec, N. Nanda, K. Ndousse, C. Olsson, D. Amodei, D. Amodei, T. Brown, J. Kaplan, S. McCandlish, C. Olah, and J. Clark. Predictability and Surprise in Large Generative Models. arXiv:2202.07785 [cs] , Feb. 2022. arXiv: 2202.07785. - · arXiv
  20. [23][23] S. Garg, V. Perot, N. Limtiaco, A. Taly, E. H. Chi, and A. Beutel. Counterfactual Fairness in Text Classification through Robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , AIES '19, pages 219-226, New York, NY, USA, Jan. 2019. Association for Computing Machinery. -
  21. [24][24] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford. Datasheets for Datasets. arXiv:1803.09010 [cs] , Dec. 2021. arXiv: 1803.09010. - · arXiv
  22. [25][25] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. ArXiv , abs/2009.11462, 2020. - · arXiv
  23. [27][27] E. A. Holmes, E. L. James, T. Coode-Bate, and C. Deeprose. Can Playing the Computer Game 'Tetris' Reduce the Build-Up of Flashbacks for Trauma? A Proposal from Cognitive Science. PLOS ONE , 4(1):e4153, Jan. 2009. Publisher: Public Library of Science. -
  24. [28][28] B. Hutchinson, V. Prabhakaran, E. Denton, K. Webster, Y. Zhong, and S. Denuyl. Social Biases in NLP Models as Barriers for Persons with Disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5491-5501, Online, July 2020. Association for Computational Linguistics. -
  25. [29][29] R. Jia and P. Liang. Adversarial Examples for Evaluating Reading Comprehension Systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 2021-2031, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. -
  26. [30][30] Y. Jiang and M. Bansal. Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2726-2736, Florence, Italy, July 2019. Association for Computational Linguistics. -
  27. [31][31] S. Karunakaran and R. Ramakrishan. Testing Stylistic Interventions to Reduce Emotional Impact of Content Moderation Workers. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , 7:50-58, Oct. 2019. -
  28. [32][32] D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4110-4124, Online, June 2021. Association for Computational Linguistics. -
  29. [33][33] K. Kurita, N. Vyas, A. Pareek, A. W. Black, and Y. Tsvetkov. Measuring Bias in Contextualized Word Representations. arXiv:1906.07337 [cs] , June 2019. arXiv: 1906.07337. - · arXiv
  30. [34][34] P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov. Towards Understanding and Mitigating Social Biases in Language Models. arXiv:2106.13219 [cs] , June 2021. arXiv: 2106.13219. - · arXiv
  31. [35][35] S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] , Sept. 2021. arXiv: 2109.07958. - · arXiv
  32. [36][36] A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi. DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts. arXiv:2105.03023 [cs] , June 2021. arXiv: 2105.03023. - · arXiv
  33. [37][37] K. McGuffie and A. Newhouse. The Radicalization Risks of GPT-3 and Advanced Neural Language Models. arXiv:2009.06807 [cs] , Sept. 2020. arXiv: 2009.06807. - · arXiv
  34. [38][38] L. McInnes, J. Healy, and J. Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, Sept. 2020. arXiv:1802.03426 [cs, stat]. - · arXiv
  35. [40][40] Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial NLI: A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4885-4901, Online, July 2020. Association for Computational Linguistics. -
  36. [41][41] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, Mar. 2022. arXiv:2203.02155 [cs]. - · arXiv
  37. [42][42] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red Teaming Language Models with Language Models. arXiv:2202.03286 [cs] , Feb. 2022. arXiv: 2202.03286. - · arXiv
  38. [43][43] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. v. d. Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, S. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J.-B. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. d. M. d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. d. L. Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. Hechtman, L. Weidinger, I. Gabriel, W. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv:2112.11446 [cs] , Dec. 2021. arXiv: 2112.11446. - · arXiv
  39. [44][44] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents, Apr. 2022. arXiv:2204.06125 [cs]. - · arXiv
  40. [45][45] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4902-4912, Online, July 2020. Association for Computational Linguistics. -
  41. [46][46] P. Röttger, B. Vidgen, D. Nguyen, Z. Waseem, H. Margetts, and J. Pierrehumbert. HateCheck: Functional Tests for Hate Speech Detection Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 41-58, Online, Aug. 2021. Association for Computational Linguistics. -
  42. [47][47] M. Sap, S. Gabriel, L. Qin, D. Jurafsky, N. A. Smith, and Y. Choi. Social Bias Frames: Reasoning about Social and Power Implications of Language. arXiv:1911.03891 [cs] , Apr. 2020. arXiv: 1911.03891. - · arXiv
  43. [48][48] I. Solaiman and C. Dennison. Process for Adapting Language Models to Society (PALMS) with ValuesTargeted Datasets. arXiv:2106.10328 [cs] , Nov. 2021. arXiv: 2106.10328. - · arXiv
  44. [49][49] M. Steiger, T. J. Bharucha, S. Venkatagiri, M. J. Riedl, and M. Lease. The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , CHI '21, pages 1-14, New York, NY, USA, May 2021. Association for Computing Machinery. -
  45. [50][50] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks, Feb. 2014. arXiv:1312.6199 [cs]. - · arXiv
  46. [51][51] A. Tamkin, M. Brundage, J. Clark, and D. Ganguli. Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models. arXiv:2102.02503 [cs] , Feb. 2021. arXiv: 2102.02503. - · arXiv
  47. [52][52] E. R. Thompson. Development and Validation of an Internationally Reliable Short-Form of the Positive and Negative Affect Schedule (PANAS). Journal of Cross-Cultural Psychology , 38(2):227-242, Mar. 2007. Publisher: SAGE Publications Inc. -
  48. [53][53] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C.-C. Chang, I. Krivokon, W. Rusch, M. Pickett, K. Meier-Hellstern, M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera-Arcas, C. Cui, M. Croak, E. Chi, and Q. Le. LaMDA: Language Models for Dialog Applications. arXiv:2201.08239 [cs] , Jan. 2022. arXiv: 2201.08239. - · arXiv
  49. [55][55] E. Wallace, A. Williams, R. Jia, and D. Kiela. Analyzing Dynamic Adversarial Training Data in the Limit, Oct. 2021. arXiv:2110.08514 [cs]. - · arXiv
  50. [56][56] D. Watson, L. A. Clark, and A. Tellegen. Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology , 54(6):1063-1070, June 1988. -
  51. [57][57] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social risks of harm from Language Models. arXiv:2112.04359 [cs] , Dec. 2021. arXiv: 2112.04359. - · arXiv
  52. [58][58] J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang. Challenges in Detoxifying Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2447-2469, Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. -
  53. [59][59] H. Xu, Y. Ma, H. Liu, D. Deb, H. Liu, J. Tang, and A. K. Jain. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review, Oct. 2019. arXiv:1909.08072 [cs, stat]. - · arXiv
  54. [60][60] J. Xu, D. Ju, M. Li, Y.-L. Boureau, J. Weston, and E. Dinan. Bot-Adversarial Dialogue for Safe Conversational Agents. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tür, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021 , pages 2950-2968. Association for Computational Linguistics, 2021. -
  55. [61][61] D. M. Ziegler, S. Nix, L. Chan, T. Bauman, P. Schmidt-Nielsen, T. Lin, A. Scherlis, N. Nabeshima, B. Weinstein-Raun, D. de Haas, B. Shlegeris, and N. Thomas. Adversarial Training for High-Stakes Reliability, May 2022. arXiv:2205.01663 [cs]. · arXiv

Authors
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., DasSarma, N., Drain, D., Elhage, N., El-Showk, S., Fort, S., Hatfield-Dodds, Z., Henighan, T., Hernandez, D., Hume, T., Jacobson, J., Johnston, S., Kravec, S., Olsson, C., Ringer, S., Tran-Johnson, E., Amodei, D., Brown, T., Joseph, N., McCandlish, S., Olah, C., Kaplan, J., & Clark, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2209.07858
arXiv
2209.07858
OpenAlex
W4296413526
Cited by
130
References
0
Citations by year
2026: 33 · 2025: 20 · 2024: 38 · 2023: 39
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
red teaming, language models, scale effects, reinforcement learning from human feedback, rejection sampling, offensive language, model size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review