StereoSet: Measuring stereotypical bias in pretrained language models
Open accessAbstract
A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. In order to assess the adverse effects of these models, it is important to quantify the bias captured in them. Existing literature on quantifying bias evaluates pretrained language models on a small set of artificially constructed bias-assessing sentences. We present StereoSet, a large-scale natural dataset in English to measure stereotypical biases in four domains: gender, profession, race, and religion. We evaluate popular models like BERT, GPT-2, RoBERTa, and XLNet on our dataset and show that these models exhibit strong stereotypical biases. We also present a leaderboard with a hidden test set to track the bias of future language models at https://stereoset.mit.edu
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Models trained on human data can exhibit human biases.
“It is well-understood that models trained on human data - such as being pre-trained on human-written text or fine-tuned on human feedback - can exhibit human biases.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Nadeem et al., 2020
The Introduction says that models learning from massive text corpora may reflect real-world stereotypical biases, and the Abstract reports strong stereotypical biases in pretrained language models. These passages support the human-written-text example, but our copy does not address models fine-tuned on human feedback. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 15 · Cited by 643
Research works printed but not linked: 32
- Vamsi Aribandi, Yi Tay, and Donald Metzler. 2021. How reliable are model diagnostics? In Proceed- ings of ACL Findings.
- Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604. · DOI · Paper page
- Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of FAccT.
- Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Pro- ceedings of Neural Information Processing Systems (NeurIPS), pages 4349-4357.
- Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
- Patricia Hill Collins. 2004. Black sexual politics: African Americans, gender, and the new racism. Routledge.
- Alexander M Czopp, Aaron C Kay, and Sapna Cheryan. 2015. Positive stereotypes are pervasive and powerful. Perspectives on Psychological Sci- ence, 10(4):451-463.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of North American Chap- ter of the Association for Computational Linguistics, pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics. · DOI · Paper page
- Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and dynamics of mechanical turk workers. In Proceedings of the ACM Interna- tional Conference on Web Search and Data Mining, WSDM '18, pages 135 -143, New York, NY, USA. Association for Computing Machinery. · DOI
- Anthony G. Greenwald and Mahzarin R. Banaji. 1995. Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological review, 102(1):4.
- Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the Association for Computational Linguistics, pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
- Milos Jakubicek, Adam Kilgarriff, Vojtech Kovar, Pavel Rychly, and Vit Suchomel. 2013. The tenten corpus family. In Proceedings of the International Corpus Linguistics Conference CL.
- Adam Kilgarriff. 2009. Simple maths for keywords. In Proceedings of the Corpus Linguistics Conference 2009 (CL2009), page 171.
- Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In Proceedings of Joint Conference on Lexical and Computational Se- mantics, pages 43-53.
- Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceed- ings of the First Workshop on Gender Bias in Natu- ral Language Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
- Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the Association for Computational Linguistics, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
- Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as cau- casian is to police: Detecting and removing mul- ticlass bias in word embeddings. In Proceedings of the North American Chapter of the Association for Computational Linguistics, pages 615-621, Min- neapolis, Minnesota. Association for Computational Linguistics. · DOI
- Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measur- ing social biases in sentence encoders. In Proceed- ings of the North American Chapter of the Associa- tion for Computational Linguistics, pages 622-628, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI · Paper page
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeffrey Dean. 2013. Distributed represen- tations of words and phrases and their composition- ality. In Proceedings of Neural Information Pro- cessing Systems (NeurIPS), NIPS 13, pages 3111 - 3119, Red Hook, NY, USA.
- Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of Em- pirical Methods in Natural Language Processing (EMNLP), pages 1953-1967, Online. Association for Computational Linguistics. · DOI · Paper page
- Brian Nosek, Mahzarin Banaji, and Anthony Green- wald. 2002. Math = male, me = female, therefore math != me. Journal of personality and social psy- chology, 83:44-59. · DOI
- Jeffrey Pennington, Richard Socher, and Christo- pher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of Em- pirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Rep- resentations. In Proceedings of the North American Chapter of the Association for Computational Lin- guistics), pages 2227-2237. Association for Com- putational Linguistics. · DOI · Paper page
- Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of North American Chapter of the Association for Computa- tional Linguistics (NAACL), pages 8-14.
- Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the of the Association for Computational Linguistics, Online. Association for Computational Linguistics. · DOI
- Emily Sheng, Kai-Wei Chang, Premkumar Natara- jan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language genera- tion. In Proceedings of the Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412, Hong Kong, China. Association for Computational Linguistics. · DOI · Paper page
- Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. science, 185(4157):1124-1131.
- Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: A free collaborative knowledgebase. Com- mun. ACM, 57(10):78-85. · DOI
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretrain- ing for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'e Buc, E. Fox, and R. Garnett, editors, Proceedings of Neural Infor- mation Processing Systems (NeurIPS), pages 5753- 5763.
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of North American Chap- ter of the Association for Computational Linguistics, pages 15-20.
- Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE In- ternational Conference on Computer Vision (ICCV), ICCV 15, pages 19 -27, USA. IEEE Computer So- ciety. · DOI · Paper page
- Authors
- Nadeem, M., Bethke, A., & Reddy, S.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of California, Santa Barbara (US)
- University of Cambridge (GB)
- ORCID
- Reddy, Siva
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 5356-5371
- arXiv
- 2004.09456
- OpenAlex
- W3019416653
- Cited by
- 643
- References
- 15
- Citations by year
- 2026: 71 · 2025: 131 · 2024: 98 · 2023: 191 · 2022: 85 · 2021: 48 · 2020: 19
- FWCI
- 95.21
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- stereotypical bias, pre-trained language models, gender stereotypes, language model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
