Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language Models
Open accessAbstract
Pieter Delobelle, Ewoenam Tokpo, Toon Calders, Bettina Berendt. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Probability-based metrics are not sufficient checks for bias prior to deployment.
“and caution that these metrics are not sufficient checks for bias prior to deployment.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Delobelle et al. 2022
Embedding-based measures of bias can be highly dependent on different design choices, such as the construction of template sentences, the choice of seed words, and the type of representation.
“Delobelle et al. (2022) also point out that embedding-based measures of bias can be highly dependent on different design choices, such as the construction of template sentences, the choice of seed words, and the type of representation”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Delobelle et al. 2022
Probability-based metrics may be only weakly correlated with biases that appear in downstream tasks.
“Similar to the shortcomings of embedding-based metrics, Delobelle et al. (2022) and Kaneko, Bollegala, and Okazaki (2022) point out that probability-based metrics may be only weakly correlated with biases that appear in downstream tasks”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Delobelle et al. 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“We also did not draw much attention to many other negative impacts of language models that practitioners should consider, e.g. high energy usage or not including all stakeholders when training a language model (Bender et al., 2021).”
BackgroundDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1× · see citing works
“This finetuning is typically done with different datasets that are labeled for the task at hand and here we can observe extrinsic bias with allocational harms (Goldfarb-Tarrant et al., 2021; Blodgett et al., 2020), e.g. gender imbalances in co-reference resolution (see § 3.2).”
BackgroundInfluentialStereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · see citing works
On Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461 · see citing works
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446 · see citing works
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
Evaluating the Underlying Gender Bias in Contextualized Word Embeddings
Basta, C., Costa-jussà, M. R., & Casas, N. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3805 · Cited by 189 · see citing works
Guo, W., & Caliskan, A. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462536 · Cited by 180 · see citing works
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · cited 4× · see citing works
“However, other measures, like CrowS-pairs (Nangia et al., 2020), correlate less with two extrinsic measures, which we suspect to be related to issues found by Blodgett et al. (2021), although more experiments are needed to confirm this.”
BackgroundResultInfluentialGarrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., & Ureña-López, L. A. (2021) · Applied Sciences · doi:10.3390/app11073184 · Cited by 123 · see citing works
HONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95 · see citing works
Flagged first, then by citations
Research works printed but not linked: 37
- Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 1889-1904. Association for Computational Linguistics. · DOI
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. · arXiv
- Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT's gender bias. In Proceedings of the Second Workshop on Gender Bias in Natu- ral Language Processing, pages 1-16, Barcelona, Spain (Online). Association for Computational Lin- guistics.
- Elisa Bassignana, Valerio Basile, and Viviana Patti. 2018. Hurtlex: A multilingual lexicon of words to hurt. In 5th Italian Conference on Computational Linguistics, CLiC-it 2018, volume 2253, pages 1-6. CEUR-WS.
- Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Pro- ceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 4356-4364, Red Hook, NY, USA.
- Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186. · DOI · Paper page
- Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568-4595, Online. Association for Computational Linguistics. · DOI
- Rodrigo Alejandro Chávez Mulsa and Gerasimos Spanakis. 2020. Evaluating bias in Dutch word em- beddings. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 56-71, Barcelona, Spain (Online). Associa- tion for Computational Linguistics.
- Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa-based Lan- guage Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255-3265, Online. Association for Computational Linguistics. · DOI
- Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Sriku- mar. 2020. On measuring and mitigating biased in- ferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 7659-7666.
- Emily Dinan, Angela Fan, Ledell Wu, Jason We- ston, Douwe Kiela, and Adina Williams. 2020. Multi-dimensional gender bias classification. arXiv preprint arXiv:2005.00614. · arXiv
- Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ri- cardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1926-1940, Online. Associa- tion for Computational Linguistics. · DOI · Paper page
- Hila Gonen and Y. Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender bi- ases in word embeddings but do not remove them. In NAACL.
- Anthony G Greenwald, Debbie E McGhee, and Jor- dan LK Schwartz. 1998. Measuring individual dif- ferences in implicit cognition: the implicit associa- tion test. Journal of personality and social psychol- ogy, 74(6):1464.
- Masahiro Kaneko and Danushka Bollegala. 2021. Unmasking the mask-evaluating social biases in masked language models. arXiv preprint arXiv:2104.07496. · arXiv
- S. Kullback and R. A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statis- tics, 22(1):79 -86. · DOI · Paper page
- Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contex- tualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy. Associ- ation for Computational Linguistics. · DOI · Paper page
- Anne Lauscher, Tobias Lüken, and Goran Glavaš. 2021. Sustainable modular debiasing of language models. arXiv preprint arXiv:2109.03646. · arXiv
- Hector Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Princi- ples of Knowledge Representation and Reasoning.
- Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards under- standing and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565-6576. PMLR.
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907.11692. · arXiv
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcom- ing few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. · arXiv
- Tomas Mikolov, Kai Chen, G.s Corrado, and Jeffrey Dean. 2013. Efficient estimation of word represen- tations in vector space. Proceedings of Workshop at ICLR, 2013.
- Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532- 1543.
- Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301. · arXiv
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106. · DOI · Paper page
- Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Compu- tational Linguistics. · DOI
- João Sedoc and Lyle Ungar. 2019. The role of pro- tected class word lists in bias identification of con- textualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Lan- guage Processing, pages 55-61, Florence, Italy. As- sociation for Computational Linguistics. · DOI
- Pamela Stone and Meg Lovejoy. 2004. Fast-track women and the "choice" to stay home. The ANNALS of the American Academy of Political and Social Sci- ence, 596(1):62-83. · DOI
- Yi Chern Tan and L. Elisa Celis. 2019. Assessing so- cial and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems, volume 32.
- Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. 2021. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 2232-2242, On- line. Association for Computational Linguistics. · DOI
- Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stu- art M Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In NeurIPS.
- Kellie Webster, Marta Recasens, Vera Axelrod, and Ja- son Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transac- tions of the Association for Computational Linguis- tics, 6:605-617. · DOI · Paper page
- Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beu- tel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2020. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032. · arXiv
- Jieyu Zhao, Subhabrata Mukherjee, Saghar Hos- seini, Kai-Wei Chang, and Ahmed Hassan Awadal- lah. 2020. Gender bias in multilingual embed- dings and cross-lingual transfer. arXiv preprint arXiv:2005.00699. · arXiv
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cot- terell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629-634, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15-20, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page
Cited by
Sorted by citations
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
More human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329 · see citing works
Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580688 · Cited by 248 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76 · see citing works
- Authors
- Delobelle, P., Tokpo, E., Calders, T., & Berendt, B.
- Affiliations
- KU Leuven (BE)
- University of Antwerp (BE)
- Weizenbaum Institute (DE)
- Venue
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2022
- Pages
- 1693-1706
- OpenAlex
- W4287887133
- Cited by
- 57
- References
- 14
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 15 · 2023: 28
- FWCI
- 11.89
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Lirias (submittedVersion, other-oa)
- Funders
- Bundesministerium für Bildung und Forschung (16DII113)
- Fonds Wetenschappelijk Onderzoek (30992574)
- Vlaamse regering
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- bias metrics, fairness measurement, pre-trained language models, algorithmic fairness, bias detection, bias in language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
