CoChat
Sign Up
Cited by 249Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionTopic Modeling

BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation

Open access
Dhamala, Jwala, Sun, Tony, Kumar, Varun, Krishna, Satyapriya, Pruksachatkun, Yada, Chang, Kai-Wei, Gupta, Rahul·2021·Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3442188.3445924
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology. We also propose new automated metrics for toxicity, psycholinguistic norms, and text gender polarity to measure social biases in open-ended text generation from multiple angles. An examination of text generated from three popular language models reveals that the majority of these models exhibit a larger social bias than human-written Wikipedia text across all domains. With these results we highlight the need to benchmark biases in open-ended language generation and caution users of language generation models on downstream tasks to be cognizant of these embedded prejudices.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3442188.3445924

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Question-answering models may rely on stereotypes to answer questions in ambiguous contexts.

    “Question-answering models may rely on stereotypes to answer questions in ambiguous contexts (Dhamala et al. 2021; Parrish et al. 2022).”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021

    The Introduction (p. 1) says language-generation models can produce stereotypical text, and Related Work (p. 2) discusses NLP model biases involving stereotyping. The paper does not establish that question-answering models rely on stereotypes to answer questions in ambiguous contexts; it studies open-ended text generation instead. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Language models can produce biased outputs, leak private data, generate misinformation, and be used maliciously.

    “Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously”

    From Training language models to follow instructions with human feedback (2022) · cites this paper as Dhamala et al., 2021

  • LLMs may exhibit social biases and toxicity during generation, resulting in biased outputs.

    “LLMs may exhibit social biases and toxicity [37, 53, 153] during the generation process, resulting in the production of biased outputs.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [37]

  • BOLD is collected by scraping English Wikipedia pages that mention a group in the bias domain and truncating sentences to form prompts.

    “The dataset is collected by scraping English Wikipedia pages that mention a group in the bias domain ( i.e. , profession), and truncating sentences to form prompts.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021

  • BOLD introduces 23,679 prompts to assess bias in profession, gender, race, religion, and political ideology.

    “BOLD introduces 23,679 prompts to assess bias in profession, gender, race, religion, and political ideology.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. [2][2] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems . 4349-4357. -
  2. [3][3] Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the selfassessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49-59. -
  3. [4][4] Sven Buechel and Udo Hahn. 2016. Emotion analysis as a regression problem-Dimensional models and their implications on emotion representation and metrical evaluation. In Proceedings of the Twenty-second European Conference on Artificial Intelligence . 1114-1122. -
  4. [7][7] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186. -
  5. [8][8] Kai-Wei Chang, Vinod Prabhakaran, and V. Ordonez. 2019. Bias and Fairness in Natural Language Processing. In EMNLP/IJCNLP . -
  6. [9][9] Benjamin Collier and J. Bear. 2012. Conflict, criticism, or confidence: an empirical examination of the gender gap in wikipedia contributions. In CSCW '12 . -
  7. [10][10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1) . -
  8. [14][14] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations . -
  9. [16][16] Svetlana Kiritchenko and Saif M. Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In *SEM@NAACL-HLT . -
  10. [17][17] Philipp Koehn. 2009. Statistical machine translation . Cambridge University Press. -
  11. [20][20] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations . -
  12. [22][22] Edward Loper and Steven Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics . 63-70. -
  13. [23][23] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019). - · arXiv
  14. [25][25] Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 (2020). - · arXiv
  15. [26][26] Nikita Nangia, C. Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowSPairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In EMNLP . -
  16. [27][27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9. -
  17. [28][28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9. -
  18. [31][31] Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019. The WomanWorkedasaBabysitter: On Biases in Language Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP) . 3398-3403. -
  19. [33][33] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In International Conference on Learning Representations . -
  20. [36][36] Claudia Wagner, David Garcia, Mohsen Jadidi, and Markus Strohmaier. 2015. It's a Man's Wikipedia? Assessing Gender Inequality in an Online Encyclopedia. In International AAAI Conference on Weblogs and Social Media . USA, 454-463. -
  21. [38][38] Alex Wang and Kyunghyun Cho. 2019. BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation . 30-36. -
  22. [42][42] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. HuggingFace's Transformers: State-of-the-art Natural Language Processing. In EMNLP . -

Cited by

Sorted by citations

Related

Authors
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R.
Affiliations
  • Amazon (United States) (US)
  • University of California, Santa Barbara (US)
ORCID
Dhamala, Jwala
Venue
Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2021-03
Pages
862-872
DOI
10.1145/3442188.3445924
OpenAlex
W3123340107
Cited by
244
References
24
Citations by year
2026: 48 · 2025: 65 · 2024: 57 · 2023: 45 · 2022: 21 · 2021: 8
FWCI
29.66
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
social bias, open-ended text generation, bias metrics, toxicity, psycholinguistic norms, gender polarity, gender bias, racial bias, religious bias, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review