CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Open accessAbstract
Warning: This paper contains explicit statements of offensive stereotypes and may be upsetting.Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks.However, there is ample evidence that they use the cultural biases that are undoubtedly present in the corpora they are trained on, implicitly creating harm with biased representations.To measure some forms of social bias in language models against protected demographic groups in the US, we introduce the Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs).CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age.In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping.The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups.We find that all three of the widelyused MLMs we evaluate substantially favor sentences that express stereotypes in every category in CrowS-Pairs.As work on building less biased models advances, this dataset can be used as a benchmark to evaluate progress.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many works in the literature refer to the metric as the dataset and use these interchangeably.
“In the literature, many works refer to the metric as the dataset, and use these interchangeably.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nangia et al. 2020
The Conclusion calls CrowS-Pairs a dataset and says the authors expect it to serve as a metric, supporting some overlap in how the terms are used. The paper does not establish that many works in the literature refer to the metric as the dataset or use the terms interchangeably. (LLM, conf=0.76)
Experimental — accuracy not yet measuredEach CrowS-Pairs pair consists of one sentence demonstrating a stereotype and another replacing the disadvantaged group with an advantaged one.
“Each pair consists of one sentence demonstrating a stereotype, and the other sentence replacing the disadvantaged group with an advantaged one (”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nangia et al. 2020
The Introduction says each pair contains a sentence about a historically disadvantaged group and a minimally different sentence about a contrasting advantaged group. But the disadvantaged-group sentence can either demonstrate or violate a stereotype; the Data Collection section reports that anti-stereotype examples make up 15% of the dataset. Thus the claim is accurate about the group replacement, but not about every pair demonstrating a stereotype. (LLM, conf=0.91)
Experimental — accuracy not yet measuredInstructGPT does not significantly improve over GPT-3 on the Winogender and CrowS-Pairs datasets.
“InstructGPT does not significantly improve over GPT-3 on the Winogender (Rudinger et al., 2018) and CrowSPairs (Nangia et al., 2020) datasets.”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Nangia et al., 2020
InstructGPT does not significantly improve over GPT-3 on the Winogender and CrowSPairs datasets.
“InstructGPT does not significantly improve over GPT-3 on the Winogender (Rudinger et al., 2018) and CrowSPairs (Nangia et al., 2020) datasets.”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Nangia et al., 2020
The CrowS-Pairs dataset consists of stereotype sentence pairs proposed alongside a pseudo-log-likelihood metric over this specific set of sentence pairs.
“One example is the CrowS-Pairs (Nangia et al. 2020) dataset consisting of stereotype sentence pairs proposed alongside a pseudo-log-likelihood metric over this specific set of sentence pairs:”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nangia et al. 2020
CrowS-Pairs covers nine types of biases: race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status.
“The dataset covers nine types of biases: race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nangia et al. 2020
CrowS-Pairs provides 1,508 sentence pairs to evaluate stereotypes of historically disadvantaged social groups.
“Crowdsourced Stereotype Pairs (CrowS-Pairs) (Nangia et al. 2020) provides 1,508 sentence pairs to evaluate stereotypes of historically disadvantaged social groups.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nangia et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Self-citationSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Black sexual politics: African Americans, gender, and the new racism
(2005) · Choice Reviews Online · doi:10.5860/choice.42-3729 · Cited by 1,940 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Self-citationMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · see citing works
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1× · see citing works
“Discussing Bias Upon surveying 146 NLP papers that analyze or mitigate bias, Blodgett et al. (2020) provide recommendations to guide such research.”
BackgroundRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · see citing works
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · see citing works
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · see citing works
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · see citing works
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598 · see citing works
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6399 · Cited by 579 · see citing works
On Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461 · see citing works
Self-citationGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · see citing works
Salazar, J., Liang, D., Nguyen, T. Q., & Kirchhoff, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.240 · Cited by 414 · see citing works
Inherent Disagreements in Human Textual Inferences
Pavlick, E., & Kwiatkowski, T. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00293 · Cited by 214 · see citing works
Towards Debiasing Sentence Representations
Liang, P. P., Li, I., Zheng, E., Lim, Y. C., Salakhutdinov, R., & Morency, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.488 · Cited by 163 · see citing works
Gonen, H., & Goldberg, Y. (2019) · arXiv (Cornell University) · Cited by 156 · see citing works
Social Bias in Elicited Natural Language Inferences
Rudinger, R., May, C., & Durme, B. V. (2017) · doi:10.18653/v1/w17-1609 · Cited by 98 · see citing works
Fair Work: Crowd Work Minimum Wage with One Line of Code
Whiting, M. E., Hugh, G., & Bernstein, M. S. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5283 · Cited by 80 · see citing works
Rozado, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231189 · Cited by 54 · see citing works
Reducing Sentiment Bias in Language Models via Counterfactual EvaluationPreprint
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J. W., Maini, V., Yogatama, D., & Kohli, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03064 · Cited by 50 · see citing works
Towards Understanding Gender Bias in Relation Extraction
Gaut, A., Sun, T., Tang, S., Huang, Y., Qian, J., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., et al. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.265 · Cited by 26 · see citing works
Efficient Inference through Cascades of Weighted Tree Transducers
May, J., Knight, K. K., & Vogler, H. (2010) · Cited by 26 · see citing works
Crazy Patriotism and Angry (Post)Black Women
Madison, D. S. (2009) · Communication and Critical/Cultural Studies · doi:10.1080/14791420903063810 · Cited by 16 · see citing works
Race, Perceptions of Femininity, and the Power of the First Lady: A Comparative Analysis
Gillespie, A. (2016) · doi:10.4324/9781315661018-25 · Cited by 4 · see citing works
CVBed: Structuring CVs usingWord Embeddings
Garg, S., Singh, S. S., Mishra, A., & Dey, K. (2017) · International Joint Conference on Natural Language Processing · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 35 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020. Multi- dimensional gender bias classification. ArXiv.
- Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. 2020. Towards debiasing sen- tence representations. In Proceedings of the 2020 Association for Computational Linguistics. Associa- tion for Computational Linguistics.
- Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language un- derstanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Álché Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 32, pages 3266-3280.
- Kellie Webster, Marta Recasens, Vera Axelrod, and Ja- son Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transac- tions of the Association for Computational Linguis- tics, 6:605-617. · DOI · Paper page
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, and Jamie Brew. 2019. HuggingFace's trans- formers: State-of-the-art natural language process- ing. ArXiv.
Cited by
Sorted by citations
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
“, CrowS-Pairs [371] and Winogender [372].”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Biases in Large Language Models: Origins, Inventory, and Discussion
Navigli, R., Conia, S., & Ross, B. (2023) · Journal of Data and Information Quality · doi:10.1145/3597307 · Cited by 364 · see citing works
Five sources of bias in natural language processing
Hovy, D., & Prabhumoye, S. (2021) · Language and Linguistics Compass · doi:10.1111/lnc3.12432 · Cited by 298 · see citing works
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, V. N. (2024) · Applied Sciences · doi:10.3390/app14052074 · Cited by 315 · see citing works
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · see citing works
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227 · see citing works
AI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233 · see citing works
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
Generative AI in Higher Education: The ChatGPT Effect
Chan, C. K. Y., & Colloton, T. (2024) · Routledge · doi:10.4324/9781003459026 · Cited by 203 · see citing works
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169 · see citing works
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149 · see citing works
Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation
Zhang, J., Bao, K., Zhang, Y., Wang, W., Feng, F., & He, X. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3608860 · Cited by 162 · see citing works
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2025) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/taslpro.2025.3606231 · Cited by 136 · see citing works
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.08747 · Cited by 135 · see citing works
On the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)
Mai, G., Huang, W., Sun, J., Song, S., Mishra, D. R., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. (2024) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3653070 · Cited by 275 · see citing works
Few-shot Learning with Multilingual Generative Language Models
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.616 · Cited by 134 · see citing works
Zhang, R., Li, H., Meng, H., Zhan, J., Gan, H., & Lee, Y.-C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713429 · Cited by 123 · see citing works
“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts
Guo, Y., Yang, Y., & Abbasi, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.72 · Cited by 120 · see citing works
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Cheng, M., Durmus, E., & Jurafsky, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.84 · Cited by 152 · see citing works
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models
Meade, N., Poole-Dayan, E., & Reddy, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.132 · Cited by 116 · see citing works
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 118 · see citing works
KLUE: Korean Language Understanding EvaluationPreprint
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 153 · see citing works
Barikeri, S., Lauscher, A., Vulić, I., & Glavaš, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.151 · Cited by 121 · see citing works
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 120 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
COLD: A Benchmark for Chinese Offensive Language Detection
Deng, J., Zhou, J., Sun, H., Zheng, C., Mi, F., Meng, H. M. L., & Huang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.796 · Cited by 103 · see citing works
Having Beer after Prayer? Measuring Cultural Bias in Large Language Models
Naous, T., Ryan, M. J., Ritter, A., & Xu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.862 · Cited by 96 · see citing works
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97 · see citing works
Debiasing Pre-trained Contextualised Embeddings
Kaneko, M., & Bollegala, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.107 · Cited by 102 · see citing works
Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Wang, Q., Madaio, M., Kane, S. K., Kapania, S., Terry, M. A., & Wilcox, L. G. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581278 · Cited by 94 · see citing works
HONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
Self-citationSustainable Modular Debiasing of Language Models
Lauscher, A., Lueken, T., & Glavaš, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.411 · Cited by 89 · see citing works
Efficient Large Scale Language Modeling with Mixtures of Experts
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S. V., Pasunuru, R., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.804 · Cited by 122 · see citing works
Navigating LLM ethics: advancements, challenges, and future directions
Jiao, J., Afroogh, S., Xu, Y., & Phillips, C. (2025) · AI and Ethics · doi:10.1007/s43681-025-00814-5 · Cited by 125 · see citing works
A Survey of Race, Racism, and Anti-Racism in NLP
Field, A., Blodgett, S. L., Talat, Z., & Tsvetkov, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.149 · Cited by 94 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76 · see citing works
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76 · see citing works
WangchanBERTa: Pretraining transformer-based Thai Language ModelsPreprint
Lowphansirikul, L., Polpanumas, C., Jantrakulchai, N., & Nutanong, S. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09635 · Cited by 117 · see citing works
Czarnowska, P., Vyas, Y., & Shah, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00425 · Cited by 72 · see citing works
Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection
Nozza, D. (2021) · doi:10.18653/v1/2021.acl-short.114 · Cited by 68 · see citing works
Quantifying and alleviating political bias in language models
Liu, R., Jia, C., Wei, J., Xu, G., & Vosoughi, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2021.103654 · Cited by 68 · see citing works
Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing
Shmueli, B., Fell, J., Ray, S., & Ku, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.295 · Cited by 67 · see citing works
Perturbation augmentation for fairer NLP
Qian, R., Ross, C., Fernandes, J., Smith, E. M., Kiela, D., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.646 · Cited by 69 · see citing works
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, A. S., Masoud, M., Mitchell, M., Radev, D., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.3 · Cited by 61 · see citing works
Fairness in Large Language Models: A Taxonomic Survey
Chu, Z., Wang, Z., & Zhang, W. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3682112.3682117 · Cited by 72 · see citing works
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models
Manela, D. D. V., Errington, D., Fisher, T., van Breugel, B., & Minervini, P. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.190 · Cited by 60 · see citing works
WinoQueer: A community-in-the-loop benchmark for anti-LGBTQ+ bias in large language models
Felkner, V., Chang, H.-C. H., Jang, E., & May, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.507 · Cited by 74 · see citing works
Probing Pre-Trained Language Models for Cross-Cultural Differences in Values
Arora, A., Kaffee, L.-A., & Augenstein, I. (2023) · Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.12 · Cited by 91 · see citing works
Delobelle, P., Tokpo, E., Calders, T., & Berendt, B. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.122 · Cited by 57 · see citing works
Fairness in Deep Learning: A Survey on Vision and Language Research
Parraga, O., Móre, M. D., Mattjie, C., Gavenski, N. S., Kupssinskü, L. S., Medronha, A., de Moura, L. V., Simões, G. S., & Barros, R. C. (2023) · ACM Computing Surveys · doi:10.1145/3637549 · Cited by 67 · see citing works
The Dawn of the Human-Machine Era: A forecast of new and emerging language technologies
Sayers, D., Sousa‐Silva, R., Höhn, S., Ahmedi, L., Allkivi-Metsoja, K., Anastasiou, D., Beňuš, Š., Bowker, L., Bytyçi, E., Catalá, A., et al. (2021) · doi:10.17011/jyx/reports/20210518/1 · Cited by 59 · see citing works
Unmasking the Mask – Evaluating Social Biases in Masked Language Models
Kaneko, M., & Bollegala, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21453 · Cited by 52 · see citing works
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296 · see citing works
Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals
Nozza, D., Bianchi, F., Lauscher, A., & Hovy, D. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.4 · Cited by 47 · see citing works
Unlearning Bias in Language Models by Partitioning Gradients
Yu, C., Jeoung, S., Kasi, A., Yu, P., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.375 · Cited by 46 · see citing works
Silva, A., Tambwekar, P., & Gombolay, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.189 · Cited by 46 · see citing works
Cao, Y., Pruksachatkun, Y., Chang, K.-W., Gupta, R., Kumar, V., Dhamala, J., & Galstyan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.62 · Cited by 44 · see citing works
Survey of Cultural Awareness in Language Models: Text and Beyond
Pawar, S., Park, J., Jin, J., Arora, A., Myung, J., Yadav, S., Haznitrama, F. G., Song, I., Oh, A. H., & Augenstein, I. (2025) · Computational Linguistics · doi:10.1162/coli.a.14 · Cited by 77 · see citing works
Evaluating Debiasing Techniques for Intersectional Biases
Subramanian, S., Han, X., Baldwin, T., Cohn, T., & Frermann, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.193 · Cited by 40 · see citing works
Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model
Fraser, K., Nejadgholi, I., & Kiritchenko, S. (2021) · doi:10.18653/v1/2021.acl-long.50 · Cited by 42 · see citing works
Névéol, A., Dupont, Y., Bezançon, J., & Fort, K. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.583 · Cited by 38 · see citing works
Fairness in Language Models Beyond English: Gaps and Challenges
Ramesh, K., Sitaram, S., & Choudhury, M. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.157 · Cited by 46 · see citing works
Stereotypes in ChatGPT: an empirical study
Busker, T., Choenni, S., & Bargh, M. S. (2023) · International Conference on Theory and Practice of Electronic Governance · doi:10.1145/3614321.3614325 · Cited by 34 · see citing works
Debiasing Pre-Trained Language Models via Efficient Fine-Tuning
Gira, M., Zhang, R., & Lee, K. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.8 · Cited by 32 · see citing works
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark
Zhou, J., Deng, J., Mi, F., Li, Y., Wang, Y., Huang, M., Jiang, X., Liu, Q., & Meng, H. M. L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.262 · Cited by 31 · see citing works
Digital forgetting in large language models: a survey of unlearning methods
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11078-6 · Cited by 31 · see citing works
Pipelines for Social Bias Testing of Large Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2022) · doi:10.18653/v1/2022.bigscience-1.6 · Cited by 29 · see citing works
BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation
Sun, T., He, J., Qiu, X., & Huang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.245 · Cited by 34 · see citing works
When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization
Ladhak, F., Durmus, E., Suzgun, M., Zhang, T., Jurafsky, D., McKeown, K., & Hashimoto, T. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.234 · Cited by 30 · see citing works
Zhou, F., Mao, Y., Yu, L., Yang, Y., & Zhong, T. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.232 · Cited by 29 · see citing works
Gender Bias in Masked Language Models for Multiple Languages
Kaneko, M., Imankulova, A., Bollegala, D., & Okazaki, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.197 · Cited by 31 · see citing works
ADEPT: A DEbiasing PrompT Framework
Yang, K., Yu, C., Fung, Y. R., Li, M., & Ji, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26279 · Cited by 29 · see citing works
Gendered Mental Health Stigma in Masked Language Models
Lin, I., Njoo, L., Field, A., Sharma, A., Reinecke, K., Althoff, T., & Tsvetkov, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.139 · Cited by 33 · see citing works
Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal
Gupta, U., Dhamala, J., Kumar, V., Verma, A., Pruksachatkun, Y., Krishna, S., Gupta, R., Chang, K.-W., Steeg, G. V., & Galstyan, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.55 · Cited by 28 · see citing works
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 49 · see citing works
T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
Wang, J., Liu, X., Di, Z., Liu, Y., & Wang, X. E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.160 · Cited by 26 · see citing works
Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art
Liu, C. C., Gurevych, I., & Korhonen, A. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00760 · Cited by 51 · see citing works
Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models
Cao, Y., Sotnikova, A., Daumé, H., Rudinger, R., & Zou, L. X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.92 · Cited by 24 · see citing works
Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions
Thakur, H., Jain, A., Vaddamanu, P., Liang, P. P., & Morency, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.30 · Cited by 24 · see citing works
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models
Xie, Z., & Lukasiewicz, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.876 · Cited by 30 · see citing works
Machine Bias. How Do Generative Language Models Answer Opinion Polls?
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2025) · Sociological Methods & Research · doi:10.1177/00491241251330582 · Cited by 30 · see citing works
Measuring and mitigating language model biases in abusive language detection
Song, R., Giunchiglia, F., Li, Y., Shi, L., & Xu, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103277 · Cited by 23 · see citing works
Requirements Satisfiability with In-Context Learning
Santos, S., Breaux, T. D., Norton, T. B., Haghighi, S., & Ghanavati, S. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00025 · Cited by 22 · see citing works
Red teaming large language models: A comprehensive review and critical analysis
Jabbar, M. S., Al-Azani, S., Alotaibi, A., & Ahmed, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104239 · Cited by 25 · see citing works
Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review
Rostam, Z. R. K., & Kertész, G. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3763002 · Cited by 28 · see citing works
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22 · see citing works
Federated deep long-tailed learning: A survey
Li, K., Li, Y., Zhang, J., Liu, X., & Ma, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127906 · Cited by 21 · see citing works
- Authors
- Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R.
- Affiliations
- New York University (US)
- ORCID
- Bhalerao, Rasika
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2020
- Pages
- 1953-1967
- OpenAlex
- W3089430725
- Cited by
- 446
- References
- 35
- Citations by year
- 2026: 71 · 2025: 93 · 2024: 64 · 2023: 121 · 2022: 60 · 2021: 36 · 2020: 1
- FWCI
- 56.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Funders
- National Science Foundation (1922658)
- Nvidia
- Samsung
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- social bias, masked language models, stereotypes, benchmark dataset, race, age, crowdsourced data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
