The tail wagging the dog: Dataset construction biases of social bias benchmarks
Open accessAbstract
How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given model?In this work, we study this question by contrasting social biases with non-social biases that stem from choices made during dataset construction (which might not even be discernible to the human eye).To do so, we empirically simulate various alternative constructions for a given benchmark based on seemingly innocuous modifications (such as paraphrasing or random-sampling) that maintain the essence of their social bias.On two wellknown social bias benchmarks (WINOGENDER and BIASNLI), we observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models and consequently the relative ordering of these models when ranked by measured bias.We hope these troubling observations motivate more robust measures of social biases.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Coreference resolution tasks require nuanced and near-perfect understanding of language, and resolving 'he' versus 'she' may be indicative of overall model behavior that does not clearly isolate nor demonstrate bias.
“Coreference resolution tasks require nuanced and near-perfect understanding of language, for instance, and resolving”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Selvam et al. (2023)
The WinoGender-Alternate Construction section says that a change in pronoun resolution need not isolate gender-occupation bias: positional bias or spurious correlations can also affect the result. It says near-perfect language understanding is needed for the assumption that a pronoun-driven change indicates that bias, not that coreference resolution tasks themselves require near-perfect understanding. (LLM, conf=0.82)
Experimental — accuracy not yet measuredSeveral widely-used evaluation datasets suffer from reliability and validity issues, including ambiguities about whether instances accurately reflect real-world stereotypes, inconsistent treatment of social groups, assumptions of near-perfect understanding of language, and lack of syntactic and semantic diversity.
“several widely-used evaluation datasets suffer from reliability and validity issues, including ambiguities about whether instances accurately reflect real-world stereotypes, inconsistent treatment of social groups, assumptions of near-perfect understanding of language, and lack of syntactic and semantic diversity (Blodgett et al. 2021; Gupta et al. 2023; Selvam et al. 2023).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Selvam et al. 2023
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredSmall changes to Winogender and Bias NLI's datasets that do not meaningfully change semantics, such as synonym substitutions or adjective additions, can drastically change bias scores.
“In fact, Selvam et al. (2023) show that small changes to Winogender and Bias NLI's datasets that do not meaningfully change semantics, such as synonym substitutions or adjective additions, can drastically change bias scores.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Selvam et al. (2023)
Selvam et al. (2023) has raised questions of the validity of these types of bias evaluation benchmarks.
“Selvam et al. (2023) has also raised questions of the validity of these types of bias evaluation benchmarks.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Selvam et al. (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · see citing works
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · see citing works
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · see citing works
UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514 · see citing works
Self-citationCrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446 · see citing works
The Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 431 · see citing works
Self-citationBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186 · see citing works
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · see citing works
Correcting Length Bias in Neural Machine Translation
Murray, K., & Chiang, D. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6322 · Cited by 133 · see citing works
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131 · see citing works
Self-citationCzarnowska, P., Vyas, Y., & Shah, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00425 · Cited by 72 · see citing works
Cao, Y. T., & Daumé, H. (2021) · Computational Linguistics · doi:10.1162/coli_a_00413 · Cited by 47 · see citing works
Flagged first, then by citations
Research works printed but not linked: 33
- Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 298-306.
- Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1889-1904, Online. Association for Computational Linguistics. · DOI
- Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ra- mamurthy, Moninder Singh, and Mikhail Yurochkin. 2022. Your fairness may vary: Pretrained language model fairness in toxic text classification. In An- nual Meeting of the Association for Computational Linguistics (ACL) -Findings.
- Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Re- contextualizing fairness in NLP: The case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 727-740, Online only. Association for Computational Linguistics.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to home- maker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29.
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, and et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS).
- Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186. · DOI · Paper page
- Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. 2022. On the intrinsic and extrinsic fairness evaluation metrics for contextualized lan- guage representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 561-570, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zong- wei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv, abs/2204.02311. · arXiv
- Sunipa Dev, Tao Li, Jeff M. Phillips, and Vivek Sriku- mar. 2020. On measuring and mitigating biased infer- ences of word embeddings. Conference on Artificial Intelligence (AAAI).
- Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2021a. Oscar: Orthogonal subspace correction and rectification of biases in word embeddings. In Con- ference on Empirical Methods in Natural Language Processing (EMNLP).
- Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Ar- jun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021b. Harms of gender exclusivity and challenges in non-binary representation in language technolo- gies. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ri- cardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Annual Meeting of the Asso- ciation for Computational Linguistics (ACL).
- Hannah Rose Kirk, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, Yuki Asano, et al. 2021. Bias out-of-the-box: An em- pirical analysis of intersectional occupational biases in popular generative language models. Advances in Neural Information Processing Systems (NeurIPS).
- Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. 2020. Look at the first sen- tence: Position bias in question answering. In Con- ference on Empirical Methods in Natural Language Processing (EMNLP).
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Confer- ence on Learning Representations (ICLR).
- Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to- fine inference. In Conference of the North American Chapter of the Association for Computational Lin- guistics (NAACL).
- Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021. Collecting a large-scale gender bias dataset for coref- erence resolution and machine translation. In Con- ference on Empirical Methods in Natural Language Processing (EMNLP) -Findings.
- Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabhar- wal, and Vivek Srikumar. 2020. UnQovering Stereo- typical Biases via Underspecified Questions. In Con- ference on Empirical Methods in Natural Language Processing (EMNLP) -Findings.
- Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. WANLI: Worker and AI Collabora- tion for Natural Language Inference Dataset Creation. arXiv preprint arXiv:2201.05955. · arXiv
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
- Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Conference on Empirical Methods in Natural Language Process- ing (EMNLP).
- Shrimai Prabhumoye, Rafal Kocielnik, Mohammad Shoeybi, Anima Anandkumar, and Bryan Catan- zaro. 2021. Few-shot instruction prompts for pre- trained language models to detect social biases. arXiv preprint arXiv:2112.07868. · arXiv
- Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Conference of the North American Chapter of the Association for Computa- tional Linguistics (NAACL).
- Viktor Schlegel, Goran Nenadic, and Riza Batista- Navarro. 2020. Beyond leaderboards: A survey of methods for revealing weaknesses in natural lan- guage inference data and models. arXiv preprint arXiv:2005.14709. · DOI
- Preethi Seshadri, Pouya Pezeshkpour, and Sameer Singh. 2022. Quantifying social biases us- ing templates is unreliable. arXiv preprint arXiv:2210.04337. · arXiv
- Tejas Srinivasan and Yonatan Bisk. 2021. Worst of both worlds: Biases compound in pre-trained vision-and- language models. In Workshop on Gender Bias in Natural Language Processing.
- Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. LaMDA: Language Models for Dialog Appli- cations. arXiv preprint arXiv:2201.08239. · arXiv
- Shubham Toshniwal, Patrick Xia, Sam Wiseman, Karen Livescu, and Kevin Gimpel. 2021. On generaliza- tion in coreference resolution. In Proceedings of the Workshop on Computational Models of Reference, Anaphora and Coreference.
- T. Winograd. 1972. Understanding natural language. Cognitive psychology, 3(1):1-191.
- Chong Zhang, Jieyu Zhao, Huan Zhang, Kai-Wei Chang, and Cho-Jui Hsieh. 2021. Double perturbation: On the robustness of robustness and counterfactual bias evaluation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3899-3916, Online. Association for Computational Linguistics. · DOI
- Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sab- harwal, and Kai-Wei Chang. 2021. Ethical-advice taker: Do language models understand natural lan- guage interventions? In Annual Meeting of the Asso- ciation for Computational Linguistics (ACL) -Find- ings.
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Conference of the North American Chap- ter of the Association for Computational Linguistics (NAACL).
Cited by
Sorted by citations
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
- Authors
- Selvam, N., Dev, S., Khashabi, D., Khot, T., & Chang, K.-W.
- Affiliations
- University of California, Los Angeles (US)
- Google (United States) (US)
- Johns Hopkins University (US)
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2023
- Pages
- 1373-1386
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385570014
- Cited by
- 11
- References
- 15
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 5 · 2023: 2
- FWCI
- 2.35
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Cisco Systems
- Allen Institute for Artificial Intelligence
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- social bias, benchmark design, paraphrase, random sampling, bias measurement, model ranking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
