Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Open accessThe DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
⚠ This paper contains prompts and model outputs that are offensive in nature. When trained on large, unfiltered crawls from the Internet, language models pick up and reproduce all kinds of undesirable biases that can be found in the data: They often generate racist, sexist, violent, or otherwise toxic language. As large models require millions of training examples to achieve good performance, it is difficult to completely prevent them from being exposed to such content. In this paper, we first demonstrate a surprising finding: Pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce. We refer to this capability as self-diagnosis. Based on this finding, we then propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. We refer to this approach as self-debiasing. Self-debiasing does not rely on manually curated word lists, nor does it require any training data or changes to the model’s parameters. While we by no means eliminate the issue of language models generating biased text, we believe our approach to be an important step in this direction.1
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, it has limitations as it cannot be applied to downstream tasks.”
From Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models (2024) · cites this paper as Schick et al. (2021)
In the citing paper
“Importantly, this method does not compromise the language model and requires no additional training.”
From Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models (2024) · cites this paper as Schick et al. (2021)
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 16×
“…illustrating that both static (e.g., Mikolov et al., 2013; Bojanowski et al., 2017) and contextualized word embeddings (e.g., Peters et al., 2018; Devlin et al., 2019) pretrained in a self-supervised fashion exhibit all kinds of unfair and discriminative biases (Bolukbasi et al., 2016; Caliskan…”
BackgroundMethodInfluentialEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“There is a large body of work illustrating that both static (e.g., Mikolov et al., 2013; Bojanowski et al., 2017) and contextualized word embeddings (e.g., Peters et al., 2018; Devlin et al., 2019) pretrained in a self-supervised fashion exhibit all kinds of unfair and discriminative biases…”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“…a large body of work illustrating that both static (e.g., Mikolov et al., 2013; Bojanowski et al., 2017) and contextualized word embeddings (e.g., Peters et al., 2018; Devlin et al., 2019) pretrained in a self-supervised fashion exhibit all kinds of unfair and discriminative biases (Bolukbasi et…”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 13×
“Figure 1: Most probable continuations according to T5-3B (Raffel et al., 2020) and GPT2-XL (Radford et al.”
BackgroundMethodInfluentialEnriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 1×
“There is a large body of work illustrating that both static (e.g., Mikolov et al., 2013; Bojanowski et al., 2017) and contextualized word embeddings (e.g., Peters et al., 2018; Devlin et al., 2019) pretrained in a self-supervised fashion exhibit all kinds of unfair and discriminative biases…”
BackgroundTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Building training datasets with more care and deliberation, an alternative solution discussed by Bender et al. (2021), is important, especially for improving linguistic and cultural diversity in online and other forms of communication.”
Background
Research works printed but not linked: 1
- Wang, 2019, BERT has a mouth, and it must speak: BERT as a Markov random field language model, 30, 10.18653/v1/W19-2304 · doi:10.18653/v1/w19-2304
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
“[168] showed that PLMs are able to recognize the toxicity of the text they produce (self-diagnosis).”
BackgroundBias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369
“For example, researchers are developing approaches for LLMs to self-diagnose 59 and self-criticize based on principles written by people 60 to de-bias their output.”
BackgroundRed Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363
Erasing Concepts from Diffusion Models
Gandikota, R., Materzyńska, J., Fiotto-Kaufman, J., & Bau, D. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00230 · Cited by 227
“Like previous works, we treat “A and not B” as the difference between log probability densities for A and B; a similar observation has been used to reduce the undesirable output of both language models [37] and vision generators [38].”
MethodBias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
“Self-Debias is another work that leverages model’s internal knowledge to generate unbiased prompts (Schick et al., 2021).”
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219
“[32] demonstrated that language models can self-debias the text they produce, specifically regarding toxic output.”
BackgroundInfluentialPre-Trained Language Models for Text Generation: A Survey
Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., & Wen, J.-R. (2024) · ACM Computing Surveys · doi:10.1145/3649449 · Cited by 195
“Counterfactual data augmentation [287], dropout regularization [247], prompt-based self-debias [83, 208].”
BackgroundMethodToxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2304.05335 · Cited by 178
“Similarly, (Schick et al., 2021) propose a decoding algorithm, which given the description of the offending output, reduces the probability of generating the offensive text.”
BackgroundGPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
Yoo, K. M., Park, D., Kang, J., Lee, S.-W., & Park, W. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.192 · Cited by 151
Defending ChatGPT against jailbreak attack via self-reminders
Xie, Y., Yi, J., Shao, J., Curl, J., Lyu, L., Chen, Q., Xie, X., & Wu, F. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00765-8 · Cited by 149
“Recent studies have explored the capacity of LLMs to validate and correct their own claims 31–33 .”
Visual Adversarial Examples Jailbreak Aligned Large Language Models
Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., & Mittal, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30150 · Cited by 150
“Following practices of Schick, Udupa, and Sch¨utze (2021); Mehrabi et al. (2022), we use the challenging subset of Real-ToxicityPrompts, which contains 1225 text prompts for triggering toxic continuations.”
MethodTowards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131
“…et al., 2019; Sheng et al., 2019, 2020; Shwartz et al., 2020; Vig et al., 2020; Yeo and Chen, 2020; Huang et al., 2020; Dhamala et al., 2021; Schick et al., 2021), GPT-3 (Brown et al., 2020), CTRL (Dhamala et al., 2021), TransformerXL (Shwartz et al., 2020; Vig et al., 2020; Huang et al.,…”
BackgroundGenerating Datasets with Pretrained Language Models
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.555 · Cited by 122
“We achieve this using Schick et al. (2021)’s self-debiasing algorithm: When sampling the token xk, we consider not just py = pM (xk | Iy(x1), x1, . . . , xk−1) [xk’s probability given Iy(x1)], but also py′ [xk’s probability given Iy′(x1)], for all y′ ∈ CL(y).”
BackgroundMethodInfluentialSelf-citationAuto-Debias: Debiasing Masked Language Models with Automated Biased Prompts
Guo, Y., Yang, Y., & Abbasi, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.72 · Cited by 113
“…also reported similar human-like biases and stereotypes in contextual embedding PLMs (May et al., 2019; Kurita et al., 2019; Tan and Celis, 2019; Hutchinson et al., 2020; Guo and Caliskan, 2021; Wolfe and Caliskan, 2021) or in the text generation tasks (Schick et al., 2021; Sheng et al., 2019).”
BackgroundMarked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Cheng, M., Durmus, E., & Jurafsky, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.84 · Cited by 115
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
Geva, M., Caciularu, A., Wang, K., & Goldberg, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.3 · Cited by 117
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models
Meade, N., Poole-Dayan, E., & Reddy, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.132 · Cited by 110
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
ZeroGen: Efficient Zero-shot Learning via Dataset Generation
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, C., & Kong, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.801 · Cited by 340
COLD: A Benchmark for Chinese Offensive Language Detection
Deng, J., Zhou, J., Sun, H., Zheng, C., Mi, F., Meng, H. M. L., & Huang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.796 · Cited by 226
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97
“Some bias measurement approaches, such as selfdebiasing (Schick et al., 2021), do not require a list of terms at all.”
BackgroundAddressing bias in generative AI: Challenges and research opportunities in information management
Wei, X., Kumar, N., & Zhang, H. (2025) · Information & Management · doi:10.1016/j.im.2025.104103 · Cited by 174
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 200
“…translation (Chen et al., 2023b; Raunak et al., 2023; Ki and Carpuat, 2024), information retrieval (Gero et al., 2023), vision language tasks (Yin et al., 2023; Ge et al., 2023; Zhou et al., 2024; Lee et al., 2024; Huang et al., 2024b; Liu et al., 2024), and prompt optimization (Pryzant et al.,…”
A survey on multilingual large language models: corpora, alignment, and bias
Xu, Y., Hu, L., Zhao, J., Qiu, Z., Xu, K., Ye, Y., & Gu, H. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40579-4 · Cited by 223
“…a method involving chain-of-thought prompting through SHAP analysis can efficiently mitigate biases against queer people in the output of LLMs. Schick et al. [172] introduced a debiasing technique named Self-Debias which uses a model’s internal knowledge to discourage biased text generation.”
Language models, like humans, show content effects on reasoning tasks
Lampinen, A. K., Dasgupta, I., Chan, S. C. Y., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., & Hill, F. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae233 · Cited by 69
“In some cases language models can either be prompted or can learn to verify, correct, or debias their own out - puts (79).”
Irfan, B., Kuoppamäki, S., & Skantze, G. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1363713 · Cited by 97
True Few-Shot Learning with Prompts—A Real-World Perspective
Schick, T., & Schütze, H. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00485 · Cited by 111
“This idea has since been applied to solve a wide range of different NLP tasks without any task-specific training data (Puri and Catanzaro, 2019; Opitz, 2019; Davison et al., 2019; Schick et al., 2021; Schick and Schütze, 2021; Wei et al., 2021; Sanh et al., 2021).”
BackgroundMethodSelf-citationSchramowski, P., Tauchmann, C., & Kersting, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533192 · Cited by 217
“[41] demonstrated that language models can self-debias the text they produce, specifically regarding toxic output.”
BackgroundResultDetecting Hate Speech with GPT-3Preprint
Chiu, K.-L., Collins, A., & Alexander, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12407 · Cited by 188
LLMaAA: Making Large Language Models as Active Annotators
Zhang, R., Li, Y., Ma, Y., Zhou, M. Q., & Zou, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.872 · Cited by 199
The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems
Ziems, C., Yu, J., Wang, Y.-C., Halevy, A., & Yang, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.261 · Cited by 58
Fairness in Deep Learning: A Survey on Vision and Language Research
Parraga, O., Móre, M. D., Mattjie, C., Gavenski, N. S., Kupssinskü, L. S., Medronha, A., de Moura, L. V., Simões, G. S., & Barros, R. C. (2023) · ACM Computing Surveys · doi:10.1145/3637549 · Cited by 123
“For these reasons, the most promising research directions regarding bias mitigation in FMs are fine-tuning and prompting approaches [13, 63, 65, 170, 175, 178] (Sections 5.3.3 and 5.4.1).”
InfluentialSclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023) · arXiv · doi:10.48550/arxiv.2310.11324 · Cited by 52
“This StereoSet-inspired task (Nadeem et al., 2021) requires the model to, given a short passage, classify it into one of four types of stereotype or anti-stereotype (gender, profession, race, and religion). mance being significantly different (Schick et al., 2021).”
MethodChanging the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51
“The social biases in NLP models are also attracting more attention (Sheng et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020).”
BackgroundUnmasking the Mask – Evaluating Social Biases in Masked Language Models
Kaneko, M., & Bollegala, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21453 · Cited by 46
Unlearning Bias in Language Models by Partitioning Gradients
Yu, C., Jeoung, S., Kasi, A., Yu, P., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.375 · Cited by 158
“In a different vein, methods like Schick et al. (2021) utilize multiple iterative prompts to debias generations only.”
MethodRespectful or Toxic? Using Zero-Shot Learning with Language Models to Detect Hate Speech
Plaza-del-Arco, F. M., Nozza, D., & Hovy, D. (2023) · doi:10.18653/v1/2023.woah-1.6 · Cited by 114
“Schick et al. (2021) explore toxicity in LLMs using comparable prompts to self-diagnose toxicity during the decoding.”
BackgroundAnticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44
“In the context of safety, Schick et al. (2021) find that LLMs show an ability to self-identify and mitigate toxic generations using prompt manipulation.”
BackgroundCognitive Biases and Artificial Intelligence
Wang, J., & Redelmeier, D. A. (2024) · NEJM AI · doi:10.1056/aics2400639 · Cited by 56
He, X., Zannettou, S., Shen, Y., & Zhang, Y. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00061 · Cited by 35
“Similarly, Schick et al. [48] find that language models can identify toxic content and whether the generated text contains undesirable biases, all using prompt learning techniques.”
BackgroundMethodInfluentialLanguage Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
Generative Language Models for Paragraph-Level Question Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.42 · Cited by 70
Generative Language Models for Paragraph-Level Question Generation
Ushio, A., Alva-Manchego, F., & Camacho-Collados, J. (2022) · arXiv · doi:10.48550/arxiv.2210.03992 · Cited by 38
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems
Dinan, E., Abercrombie, G., Bergman, A., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.284 · Cited by 32
“Schick et al. (2021) propose something similar, but use instead the language model’s own knowledge of toxic content to detect toxic generations in zero-shot manner.”
BackgroundEthical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs
Rao, A., Khandelwal, A., Tanmay, K., Agarwal, U., & Choudhury, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.892 · Cited by 135
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark
Zhou, J., Deng, J., Mi, F., Li, Y., Wang, Y., Huang, M., Jiang, X., Liu, Q., & Meng, H. M. L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.262 · Cited by 78
“As a foundation of the strategies for above tasks, the social bias detection task is usually formalized as a binary classification task (i.e., biased or not) (Founta et al., 2018; Dinan et al., 2019, 2021; Schick et al., 2021).”
BackgroundMethodADEPT: A DEbiasing PrompT Framework
Yang, K., Yu, C., Fung, Y. R., Li, M., & Ji, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26279 · Cited by 87
RL4F: Generating natural language feedback with reinforcement learning for repairing model outputs
Akyurek, A. F., Akyurek, E., Kalyan, A., Clark, P., Wijaya, D. T., & Tandon, N. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.427 · Cited by 29
An Empirical Comparison of LM-based Question and Answer Generation Methods
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.899 · Cited by 56
Gupta, B. B., Gaurav, A., Arya, V., Alhalabi, W., Alsalman, D., & Vijayakumar, P. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109215 · Cited by 25
Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions
Thakur, H., Jain, A., Vaddamanu, P., Liang, P. P., & Morency, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.30 · Cited by 66
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models
Xie, Z., & Lukasiewicz, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.876 · Cited by 57
White-box Multimodal Jailbreaks Against Large Vision-Language Models
Wang, R., Ma, X., Zhou, H., Ji, C., Ye, G., & Jiang, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681092 · Cited by 113
“To further broaden our evaluation scope, we also employ an automated testing procedure using the RealToxicityPrompts benchmark [10].”
InfluentialReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval
Yu, Y., Zhuang, Y., Zhang, R., Yu, M., Shen, J., & Zhang, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.748 · Cited by 42
Specification overfitting in artificial intelligence
Roth, B., de Araujo, P. H. L., Xia, Y., Kaltenbrunner, S., & Korab, C. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11040-6 · Cited by 26
“Limitations of these systems include the use of non-generalizable heuristics (Tu et al, 2020), generating texts that are biased, hateful, and toxic (Schick et al, 2021), and texts that look fluent and plausible but contain falsehoods and misinformation (Ji et al, 2023; Lin et al, 2022).”
Potential Pitfalls With Automatic Sentiment Analysis: The Example of Queerphobic Bias
Ungless, E. L., Roß, B., & Belle, V. (2023) · Social Science Computer Review · doi:10.1177/08944393231152946 · Cited by 24
“There is a signi fi cant body of work looking to develop less biased language models for a range of tasks (Dixon et al., 2018; Liang et al., 2020; Schick et al., 2021; Ungless et al., 2022; Webster et al., 2021; Zhao et al., 2018), for example, using counterfactually augmented data (Sen et al.,…”
BackgroundThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
“Moreover, prompt has also been applied to data augmentation (Schick and Schütze, 2021), domain adaptation (Ben-David et al., 2021), debiasing (Schick et al., 2021) and so on.”
MethodAI Chatbots and Linguistic Injustice
Park, S. (2024) · Journal of Universal Language · doi:10.22425/jul.2024.25.1.99 · Cited by 26
“Typical methods for mitigating bias involve the use of counterfactual data augmentation (Zmigrod et al. 2019, Dinan et al. 2020, Webster et al. 2020, Barikeri et al. 2021), dropout regularization (Webster et al. 2020), and self-debias (Schick et al. 2021).”
MethodThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
“Moreover, prompt has also been applied to data augmentation (Schick and Schütze, 2021), domain adaptation (Ben-David et al., 2021), debiasing (Schick et al., 2021) and so on.”
MethodGender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models
Ghanbarzadeh, S., Huang, Y., Palangi, H., Cruz Moreno, R., & Khanpour, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.336 · Cited by 17
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A. S., & Viviano, J. D. (2021) · arXiv · doi:10.48550/arxiv.2105.02732 · Cited by 17
“, 2020), and develop methods for evaluating the bias in trained models (Schick et al., 2021).”
MethodLahoti, P., Blumm, N., Ma, X., Kotikalapudi, R., Potluri, S., Tan, Q., Srinivasan, H., Packer, B., Beirami, A., Beutel, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.643 · Cited by 71
Gender Bias in Natural Language Processing and Computer Vision: A Comparative Survey
Bartl, M., Mandal, A., Leavy, S., & Little, S. (2024) · ACM Computing Surveys · doi:10.1145/3700438 · Cited by 48
“It is similar to Schick et al.’s [105] SelfDebias, however, instead of crafting prompts to elicit biased text, Guo et al. [54] automatically found prompts that can be used for de-biasing a masked language model (MLM) , thereby reducing the reliance on external corpora.”
InfluentialDifferentially Private Language Models for Secure Data Sharing
Mattern, J., Jin, Z., Weggenmann, B., Schölkopf, B., & Sachan, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.323 · Cited by 99
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design
Bergman, A. S., Abercrombie, G., Spruit, S. L., Hovy, D., Dinan, E., Boureau, Y.-L., & Rieser, V. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.4 · Cited by 33
“In addition, work on building safer LLMs explores fine-tuning on curated data (Solaimon and Dennison, 2021) or directly controlling the generations of the model (Dathathri et al., 2019; Liu et al., 2021; Schick et al., 2021; Xu et al., 2020).”
BackgroundMitigating harm in language models with conditional-likelihood filtrationPreprint
Ngo, H., Raterink, C., Araújo, J. G. M., Zhang, I., Chen, C., Morisot, A., & Frosst, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07790 · Cited by 74
“Self-debiasing [Schick et al., 2021] mitigates corpus-based bias in language models during generation by using the learned knowledge of a language model to identify biases in text with a zero-shot prompt-based approach.”
Anoop, K. R., Gangan, M. P., Deepak, P., & Lajish, V. L. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-4453-6_2 · Cited by 21
“[98] GPT-2, T5 Occupation, Nationality, Religion, Gender, Age, Race, Disability Self-Diagnosing Self-Debiasing”
BackgroundA Practical Toolkit for Multilingual Question and Answer Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.8 · Cited by 31
The Turing Quest: Can Transformers Make Good NPCs?
Gao, Q., & Emami, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.17 · Cited by 16
Self-Detoxifying Language Models via Toxification Reversal
Leong, C. T., Cheng, Y., Wang, J., Wang, J., & Li, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.269 · Cited by 81
Advancements in AI-Generated Content Forensics: A Systematic Literature Review
Xu, Q., Mu, W., Li, J., Sun, T., & Jiang, X. (2025) · ACM Computing Surveys · doi:10.1145/3760526 · Cited by 26
“On the algorithmic side, some methods [232] focus on controlling the generative output, using decoding algorithms to curtail the production of problematic text.”
A Robust Bias Mitigation Procedure Based on the Stereotype Content Model
Ungless, E. L., Rafferty, A., Nag, H., & Roß, B. (2022) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/2022.nlpcss-1.23 · Cited by 41
On Measuring Social Biases in Prompt-Based Multi-Task Learning
Akyürek, A. F., Paik, S., Kocyigit, M. Y., Akbıyık, S., Runyun, Ş. L., & Wijaya, D. T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.42 · Cited by 33
Debiasing pretrained text encoders by paying attention to paying attention
Gaci, Y., Benatallah, B., Casati, F., & Benabdeslem, K. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.651 · Cited by 14
“, 2020), zero-shot learning (Schick et al., 2021), dropout (Meade et al.”
Wang, F., Lin, M., Ma, Y., Liu, H., He, Q., Tang, X., Tang, J., Pei, J., & Wang, S. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736563 · Cited by 43
PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives
Gao, S., Borges, B., Oh, S., Bayazit, D., Kanno, S., Wakaki, H., Mitsufuji, Y., & Bosselut, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.362 · Cited by 43
Using In-Context Learning to Improve Dialogue Safety
Meade, N., Gella, S., Hazarika, D., Gupta, P., Jin, D., Reddy, S., Liu, Y., & Hakkani-Tur, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.796 · Cited by 13
Demartini, G., Roitero, K., & Mizzaro, S. (2023) · Communications of the ACM · doi:10.1145/3611641 · Cited by 13
“[22] proposed an algorithm that reduces the probability of a language model generating problematic text.”
BackgroundCross-Domain Fake News Detection Using a Prompt-Based Approach
Alghamdi, J., Lin, Y., & Luo, S. (2024) · Future Internet · doi:10.3390/fi16080286 · Cited by 16
“Template formatting: We explore two modifications to the prompt template: – Quote addition: We introduce quotation marks around the prompt text, in contrast to the approach of Schick et al. [24], where quotes around input text were omitted.”
MethodParameter-efficient Modularised Bias Mitigation via AdapterFusion
Kumar, D., Lesota, O., Zerveas, G., Cohen, D., Eickhoff, C., Schedl, M., & Rekabsaz, N. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.201 · Cited by 37
Undesirable Biases in NLP: Addressing Challenges of Measurement
van der Wal, O., Bachmann, D., Leidinger, A., van Maanen, L., Zuidema, W. H., & Schulz, K. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15195 · Cited by 53
“…assess the model’s biases, for example, by providing the model with an item from a bias benchmark and asking it whether the item contains a stereotype (Schick et al., 2021) or which description of a person or social group (one being stereotypical, the other not) is more likely (Scao et al., 2022).”
BackgroundPromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation
Sahu, G., Vechtomova, O., Bahdanau, D., & Laradji, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.323 · Cited by 51
Debiasing large language models: research opportunities*
Yogarajan, V., Dobbie, G., & Keegan, T. T. (2024) · Journal of the Royal Society of New Zealand · doi:10.1080/03036758.2024.2398567 · Cited by 30
“…bias (Caliskan et al. 2017; May et al. 2019; Karimi Mahabadi et al. 2020; Liang et al. 2021), developing debiasing techniques (May et al. 2019; Schick et al. 2021; Meade et al. 2022), and benchmarking datasets for bias evaluations (Nadeem et al. 2021; Besse et al. 2022; Yogarajan et al.…”
Logic Against Bias: Textual Entailment Mitigates Stereotypical Sentence Reasoning
Luo, H., & Glass, J. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.89 · Cited by 19
Societal Biases in Language Generation: Progress and ChallengesPreprint
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04054 · Cited by 12
“…et al., 2019; Sheng et al., 2019, 2020; Shwartz et al., 2020; Vig et al., 2020; Yeo and Chen, 2020; Huang et al., 2020; Dhamala et al., 2021; Schick et al., 2021), GPT-3 (Brown et al., 2020), CTRL (Dhamala et al., 2021), TransformerXL (Shwartz et al., 2020; Vig et al., 2020; Huang et al.,…”
BackgroundEvaluating and Mitigating Discrimination in Language Model Decisions
Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., & Ganguli, D. (2023) · arXiv · doi:10.48550/arxiv.2312.03689 · Cited by 11
“In the context of LMs, Schick et al. (2021) note that language models can recognize toxicity in their own outputs, and Si et al. (2022) demonstrated that the right prompts could reduce bias in language models on the BBQ benchmark (Parrish et al., 2021 Model-generated evaluations Finally, a range of recent works explor…”
BackgroundQiu, P., Zhou, S., & Ferrara, E. (2025) · Information Sciences · doi:10.1016/j.ins.2025.122702 · Cited by 34
“In response, researchers have (i) designed benchmarks for representational fairness, (ii) introduced self-diagnostic techniques that prompt models to detect and suppress toxic or biased outputs, and (iii) proposed frameworks to identify training-data shifts that induce bias [19, 30, 21].”
Li, H., Wu, Y., Schlegel, V., Batista-Navarro, R., Nguyen, T.-T., Kashyap, A. R., Zeng, X., Beck, D., WINKLER, S. S., & Nenadić, G. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.49 · Cited by 26
Transformer based Answer-Aware Bengali Question Generation
Ruma, J. F., Mayeesha, T. T., & Rahman, R. M. (2023) · International Journal of Cognitive Computing in Engineering · doi:10.1016/j.ijcce.2023.09.003 · Cited by 16
Layered Bias: Interpreting Bias in Pretrained Large Language Models
Prakash, N., & Lee, R. K.-W. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.22 · Cited by 33
“Another intriguing method is "self-debiasing" as proposed by (Schick et al., 2021).”
MethodConFit: Improving Resume-Job Matching using Data Augmentation and Contrastive Learning
Yu, X., Zhang, J., & Yu, Z. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688108 · Cited by 35
“…subtle forms, we do not condone using ConFit for real-world applications without using debiasing techniques such as Bolukbasi et al. [3], Cheng et al. [9], Gaci et al. [14], Guo et al. [18], Schick et al. [44], and in general, we do not condone the use of ConFit for any morally unjust purposes.”
A Survey on the Impact of Pre-Trained Language Models in Sentiment Classification Task
Gautam, H., Gaur, A., & Yadav, D. K. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00805-z · Cited by 25
“Quantifica-tion and mitigation of bias are essential to ensure fairness, reliability, and trust in real-world applications [189].”
Power of Explanations: Towards automatic debiasing in hate speech detection
Cai, Y., Zimek, A., Wunder, G., & Ntoutsi, E. (2022) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa54385.2022.10032325 · Cited by 17
“(NLP) [7], [27] and can be easily learned by even the most advanced text classifiers [14].”
BackgroundA comprehensive analysis of gender, racial, and prompt-induced biases in large language models
Torres, N., Ulloa, C., Araya, I., Ayala, M., & Jara, S. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00696-6 · Cited by 20
“Schick et al. [25] showed that LLMs could be steered toward generating false or biased information through care-fully designed prompts, raising concerns about the potential misuse of these models.”
Friedl, P. (2023) · Law Innovation and Technology · doi:10.1080/17579961.2023.2184134 · Cited by 16
Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks
Hauzenberger, L., Masoudian, S., Kumar, D., Schedl, M., & Rekabsaz, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.386 · Cited by 31
“…into the space with minimum correlations to protected features (Ravfogel et al., 2020; Kaneko and Bol-legala, 2021; Bolukbasi et al., 2016), utilizing a distribution alignment loss (Guo et al., 2022), or penalizing bias by utilizing the encoded information in models (Schick et al., 2021).”
Background
Metadata
- Authors
- Schick, T., Udupa, S., & Schütze, H.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Schick, Timo
- Udupa, Sahana
- Schütze, Hinrich
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 1408-1424
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3134354193
- Cited by
- 227
- References
- 58
- Citations by year
- 2026: 29 · 2025: 40 · 2024: 36 · 2023: 70 · 2022: 36 · 2021: 16
- FWCI
- 13.96
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, cc-by)
- Universitätsbibliothek der LMU
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- self-diagnosis, bias in language models, toxicity, bias detection, decoding algorithms, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.