The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Open accessAbstract
Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology. We also propose new automated metrics for toxicity, psycholinguistic norms, and text gender polarity to measure social biases in open-ended text generation from multiple angles. An examination of text generated from three popular language models reveals that the majority of these models exhibit a larger social bias than human-written Wikipedia text across all domains. With these results we highlight the need to benchmark biases in open-ended language generation and caution users of language generation models on downstream tasks to be cognizant of these embedded prejudices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Question-answering models may rely on stereotypes to answer questions in ambiguous contexts (Dhamala et al. 2021; Parrish et al. 2022).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Dhamala et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“LLMs may exhibit social biases and toxicity [37, 53, 153] during the generation process, resulting in the production of biased outputs.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [37]
The cited source says what the claim attributes to it.
In the citing paper
“The dataset is collected by scraping English Wikipedia pages that mention a group in the bias domain ( i.e. , profession), and truncating sentences to form prompts.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“BOLD introduces 23,679 prompts to assess bias in profession, gender, race, religion, and political ideology.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“RealToxicityPrompts (Gehman et al. 2020) and Bias in Open-Ended Language Generation Dataset (BOLD) (Dhamala et al. 2021) both provide web-based sentence prefixes”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538
Loper, E., & Bird, S. (2002) · doi:10.3115/1118108.1118117 · Cited by 3,323
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
Understanding Back-Translation at Scale
Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1045 · Cited by 1,082
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 2×
“Recently, there has been growing evidence on how machine learning models without proper fairness checks risk reinforcing undesirable stereotypes, subjecting users to disparate treatment and enforcing de facto segregation [1, 23].”
BackgroundAmazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684
Self-citation
Flagged first, then by citations
Research works printed but not linked: 18
- Tolga Bolukbasi Kai-Wei Chang James Y Zou Venkatesh Saligrama and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems. 4349--4357.
- Sven Buechel and Udo Hahn. 2016. Emotion analysis as a regression problem---Dimensional models and their implications on emotion representation and metrical evaluation. In Proceedings of the Twenty-second European Conference on Artificial Intelligence. 1114--1122.
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186.
- Kai-Wei Chang Vinod Prabhakaran and V. Ordonez. 2019. Bias and Fairness in Natural Language Processing. In EMNLP/IJCNLP.
- Benjamin Collier and J. Bear. 2012. Conflict, criticism, or confidence: an empirical examination of the gender gap in wikipedia contributions. In CSCW '12.
- CHE Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text.
- Svetlana Kiritchenko and Saif M. Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In *SEM@NAACL-HLT.
- Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations.
- Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019). · arXiv
- Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 (2020). · arXiv
- Nikita Nangia, C. Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In EMNLP.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In International Conference on Learning Representations.
- Claudia Wagner, David Garcia, Mohsen Jadidi, and Markus Strohmaier. 2015. It's a Man's Wikipedia? Assessing Gender Inequality in an Online Encyclopedia. In International AAAI Conference on Weblogs and Social Media. USA, 454--463.
- 10.18653/v1/W19-2304 · DOI
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. HuggingFace's Transformers: State-of-the-art Natural Language Processing. In EMNLP.
Cited by
Sorted by citations
Items 201–249 of 249
Creating Frameworks and Datasets at the Intersection of AI Safety and Elections
Dmonte, A., Wang, Y., Nakov, P., Li, H., Han, X., Zampieri, M., Lybarger, K., Albanese, M., & Baldwin, T. (2025) · Springer eBooks · doi:10.1007/978-3-031-89853-2_9
Khari, M. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-2670-2.ch015 · Cited by 2
Evaluation of Ethical Bias in Responses Generated by Spanish LLMs
Vega-Rueda, J. C., & Rodríguez-Garavito, C. H. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08206-0_14
Working Towards Toxic datasets for LLM Safeguarding
Guo, L., Wang, Z., Wang, Z., & He, T. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227669
Quantifying and Mitigating Bias in GPT-Based Language Models
Vashisht, J., & Vashisht, M. (2025) · International Conference on Next Generation of Green Information and Emerging Technologies (GIET) · doi:10.1109/giet65294.2025.11234792
Uncovering Cultural Biases and Stereotypes in Large Language Models
Dorleon, G., & Shujaa, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4861-3_5
Acharya: A Multimodal BERT & CNN and LLM Agent Architecture in Job Matching for Ethical Candidate
Sekar, R. R., Rao, P. V., Ruthik, P., Sumanth, P., & Reddy, B. M. K. (2025) · International Conference on Innovative Data Communication Technologies and Application (ICIDCA) · doi:10.1109/icidca66325.2025.11280422
A red teaming framework for large language models: a case study on faithfulness evaluation
Alotaibi, A., Mughus, R., & Ahmed, M. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09779-y · Cited by 1
Exacerbating Differences in Polarity: Bias Adversarial Attack on Generative Large Language Models
Gu, T., Qing, G., Hao, F., & Chang, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3645995
Evaluation of AI ethics tools in language models: a developers’ perspective case study
Silva, J., Moreira, D. A. B., dos Santos, G. O., Ferreira, A., Maia, H., Avila, S., & Pedrini, H. (2026) · AI and Ethics · doi:10.1007/s43681-025-00914-2 · Cited by 2
Concept Control for LLM Safety Using Radial Basis Function Representations
Amos, M., Song, Y., & Pagnucco, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4969-6_17
EVENS: Equality versus Equity Notion Spectrum of LLMs
Chen, Q., Cheng, R., Liu, Y., Xie, Z., Zhao, K., Li, P., Rotolo, A., Liu, Y., & Shen, W. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769262
Detecting and Evaluating Bias in Large Language Models: Concepts, Methods, and Challenges
Gao, Z., Tong, L., & Zhang, Z. J. (2026) · Journal of Behavioral Data Science · doi:10.35566/jbds/gao · Cited by 1
Standardized Assessment of LLM English ProficiencyPreprint
Min, S., Wang, S., Gao, X., Wang, H., Jin, Z., Ling, C., & Ding, N. (2026) · Research Square · doi:10.21203/rs.3.rs-8820245/v1
Quantifying and Mitigating Occupational Bias in Open-Source Large Language Models
Raj, A., Haque, Z., Kumari, K. S., Kanmani, P., & Mukkamala, R. R. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402061
Zhang, F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6416179
Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study
Bouchouchi, N., Lesot, M., Laugel, T., Renard, X., Marsala, C., & Detyniecki, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37654-1_21 · Cited by 1
AutoFairer: Automatic Bias Detection and Sanitization for LLMsPreprint
Nath, S., Uddin, G., & Raza, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6510396
A survey on bias and fairness in machine unlearning
Partohaghighi, M., Marcia, R. F., West, B. J., & Chen, Y. (2026) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2026.03.003 · Cited by 2
On Subjectivity, Bias and Fairness in Language Model Learning
Gaci, Y. (2023) · theses.fr (ABES) · doi:10.70675/dca82074za0c4z46d1z8a6bz96ce30bea3b6
Bahrova, J., Augustýnek, M., Kubíček, J., Tomaszek, L., & Malčík, M. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3682324
Writing with AI Can Reduce Gender Bias in Hiring Evaluations
Liu, A. T. H., Lee, M., & Bai, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791136
Sunitha, K., & Sharma, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18346-0_23
Calibrating Stereotype Preference with Multi-Agent Graph Reasoning in Large Language Models
Amin, M. N., & Jesser, A. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491743
Bias Ahead: Sensitive Prompts as Early Warnings for Fairness in Large Language Models
Voria, G., de Lucía, M. E., Raia, A., Lucia, A. D., Catolino, G., & Palomba, F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c67878.2026.00023
Cross-Linguistic and Text-Length Effects on Emotion Word Recognition in Large Language Models
Yan, C., Luo, J., & He, Y. (2026) · International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE) · doi:10.1109/aetcse69203.2026.11504347
Evaluating Social Bias in RAG Systems: When External Context Helps and Reasoning Hurts
Parihar, S., & Cheng, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_44
How two AI models depict ageing: a cross-cultural analysis of ChatGPT and Doubao
Zhong, X. (2026) · Social Semiotics · doi:10.1080/10350330.2026.2686256
Hedging and Non-Affirmation: Quantifying LLM Alignment on Questions of Human Rights
Javed, R., Parent, C., Kay, J., Yanni, D., Zaini, A., Sheikh, A., Rauh, M., Gerych, W., Comanescu, R., Gabriel, I., et al. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806734
The Consensus Trap: Dissecting Subjectivity and the “Ground Truth” Illusion in Data Annotation
Munir, S., Mah, B., Kalsi, K., Kapania, S., Posada, J., Law, E., Wang, D., & Ahmed, S. I. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806510 · Cited by 1
Haq, I., & Saldías-Fuentes, B. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812419 · Cited by 6
Does It Lead to Prejudice or Discrimination? An Assessment of Stereotypes in LLM Decision-Making
Aoshima, T., & Akiyama, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812286
Nagaraj, P., Muneeswaran, V., Amer, A., Hafez, M., Muthiah, R., Islam, M. T., & Ijaz, M. F. (2025) · Asian Conference on Communication and Networks (ASIANComNet) · doi:10.1109/asiancomnet68615.2025.11579569
Evaluating LLM-Generated Wikipedia Content: Political Topics in a French Setting
Vermeirsche, J., SanJuan, É., & Jiménez, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29532-3_18
All too perfect: bias and aspiration in persona generation with LLMs
Corrêa, N. K., Mallmann, R., Kaczér, D., Mai, F., Ilievska, A., & Mönig, J. M. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11641-3
Xiao, H. (2026) · International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE) · doi:10.1109/icbase70763.2026.11619439
Standardized assessment of LLM English proficiency
Min, S., Wang, S., Gao, X., Wang, H., Jin, Z., Ling, C., & Ding, N. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100656
Quantifying Social Biases in Language Model Classifiers is Domain-Dependent
Quiroga, T., Bravo-Márquez, F., & Barrière, V. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3834859
FairBotBench: A Bilingual Benchmark for Evaluating Chatbot Behavior in E-commerce
Janusko, T., Bochnia, R., Hempel, G., Tomschke, S., Anke, J., & Thiele, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_8
BALM: Bias-Aware Generation for Large Language Models
Dorleon, G., & Shujaa, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-34896-8_2
Cicatiello, C., & Fusco, P. (2026) · Encyclopedia · doi:10.3390/encyclopedia6090182 · Cited by 1
Soto-Romero, M., RAMIREZ-PULIDO, K., Cervantes-Arrioja, A. C., & Cabello, A. L. R. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-37438-7_40
Epistemic Injustice in LLMs: Mind Audit and Intervention Based on Thought Attribution
Zhai, Y. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-37199-7_4
FairCareer-CHN: A Dataset for Evaluating Bias in LLM-Based Career Recommendation
Li, X. (2026) · International Conference on Computational Intelligence and Machine Learning (CIML) · doi:10.1109/ciml69155.2026.11692774
LangBiTe: model-driven bias testing of text-to-text large language models
Morales, S., Clarisó, R., & Cabot, J. (2026) · Software & Systems Modeling · doi:10.1007/s10270-026-01425-2
Regional Affective Differences in LLMs
Chetvergov, A., Evseev, A., Solovev, M., Sivoraksha, T., Ukolov, S., Kuschenko, V., Chistyakova, M., & Bolovtsov, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41546-2_16
Fair-MAD:For FGKB-Guided Adjudication of Stereotype-Prone Questions
Dong, Z., Yu, Z., Zuo, J., & Zhu, E. (2026) · International Conference on Computer Science, Machine Learning and Agents (CSMLA) · doi:10.1109/csmla70906.2026.11724473
Serçe, A., Kaçıkan, E., & Onan, A. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11727894
Dolunay, A., & Arpalıklı, F. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1980883
Metadata
- Authors
- Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R.
- Affiliations
- Amazon (United States) (US)
- University of California, Santa Barbara (US)
- ORCID
- Dhamala, Jwala
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 862-872
- OpenAlex
- W3123340107
- Cited by
- 244
- References
- 24
- Citations by year
- 2026: 48 · 2025: 65 · 2024: 57 · 2023: 45 · 2022: 21 · 2021: 8
- FWCI
- 29.66
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- social bias, open-ended text generation, bias metrics, toxicity, psycholinguistic norms, gender polarity, gender bias, racial bias, religious bias, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
