The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Open accessAbstract
Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology. We also propose new automated metrics for toxicity, psycholinguistic norms, and text gender polarity to measure social biases in open-ended text generation from multiple angles. An examination of text generated from three popular language models reveals that the majority of these models exhibit a larger social bias than human-written Wikipedia text across all domains. With these results we highlight the need to benchmark biases in open-ended language generation and caution users of language generation models on downstream tasks to be cognizant of these embedded prejudices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Question-answering models may rely on stereotypes to answer questions in ambiguous contexts (Dhamala et al. 2021; Parrish et al. 2022).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Dhamala et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“LLMs may exhibit social biases and toxicity [37, 53, 153] during the generation process, resulting in the production of biased outputs.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [37]
The cited source says what the claim attributes to it.
In the citing paper
“The dataset is collected by scraping English Wikipedia pages that mention a group in the bias domain ( i.e. , profession), and truncating sentences to form prompts.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“BOLD introduces 23,679 prompts to assess bias in profession, gender, race, religion, and political ideology.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“RealToxicityPrompts (Gehman et al. 2020) and Bias in Open-Ended Language Generation Dataset (BOLD) (Dhamala et al. 2021) both provide web-based sentence prefixes”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dhamala et al. 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538
Loper, E., & Bird, S. (2002) · doi:10.3115/1118108.1118117 · Cited by 3,323
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
Understanding Back-Translation at Scale
Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1045 · Cited by 1,082
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 2×
“Recently, there has been growing evidence on how machine learning models without proper fairness checks risk reinforcing undesirable stereotypes, subjecting users to disparate treatment and enforcing de facto segregation [1, 23].”
BackgroundAmazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684
Self-citation
Flagged first, then by citations
Research works printed but not linked: 18
- Tolga Bolukbasi Kai-Wei Chang James Y Zou Venkatesh Saligrama and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems. 4349--4357.
- Sven Buechel and Udo Hahn. 2016. Emotion analysis as a regression problem---Dimensional models and their implications on emotion representation and metrical evaluation. In Proceedings of the Twenty-second European Conference on Artificial Intelligence. 1114--1122.
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186.
- Kai-Wei Chang Vinod Prabhakaran and V. Ordonez. 2019. Bias and Fairness in Natural Language Processing. In EMNLP/IJCNLP.
- Benjamin Collier and J. Bear. 2012. Conflict, criticism, or confidence: an empirical examination of the gender gap in wikipedia contributions. In CSCW '12.
- CHE Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text.
- Svetlana Kiritchenko and Saif M. Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In *SEM@NAACL-HLT.
- Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations.
- Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019). · arXiv
- Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 (2020). · arXiv
- Nikita Nangia, C. Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In EMNLP.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In International Conference on Learning Representations.
- Claudia Wagner, David Garcia, Mohsen Jadidi, and Markus Strohmaier. 2015. It's a Man's Wikipedia? Assessing Gender Inequality in an Online Encyclopedia. In International AAAI Conference on Weblogs and Social Media. USA, 454--463.
- 10.18653/v1/W19-2304 · DOI
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. HuggingFace's Transformers: State-of-the-art Natural Language Processing. In EMNLP.
Cited by
Sorted by citations
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“(2023) / (Dhamala et al., 2021) / Ferrara (2023) / (Gehman et al.”
MethodBias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“We employ the Bias in Open-Ended Language Generation Dataset ( BOLD ) (Dhamala et al., 2021) to investigate how the sentiment in the model’s outputs may differ based on demographic attributes.”
MethodEvaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580688 · Cited by 248
Khan, W., Daud, A. Y., Khan, K., Muhammad, S., & Haq, R. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100026 · Cited by 260
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211
“Toxi-Gen (Hartvigsen et al., 2022) and BOLD (Dhamala et al., 2021) evaluate the bias in language models.”
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,457
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters
Wan, Y., Pu, G., Sun, J., Garimella, A., Chang, K.-W., & Peng, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.243 · Cited by 280
Self-citationSocietal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131
Self-citationTypology of Risks of Generative Text-to-Image Models
Bird, C., Ungless, E., & Kasirzadeh, A. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604722 · Cited by 121
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119
Algorithmic fairness datasets: the story so far
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00854-z · Cited by 161
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 152
Explicitly unbiased large language models still form biased associations
Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T. L. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2416228122 · Cited by 306
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Solaiman, I., & Dennison, C. (2021) · arXiv · doi:10.48550/arxiv.2106.10328 · Cited by 90
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
Self-citationChoi, D., Lee, S., Kim, S.-I., Lee, K., Yoo, H. J., Lee, S., & Hong, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641989 · Cited by 84
Kim, T., Shin, D., Kim, Y.-H., & Hong, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642693 · Cited by 75
StarCoder 2 and The Stack v2: The Next Generation
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.19173 · Cited by 74
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
Goodman, S. M., Buehler, E., Clary, P., Coenen, A., Donsbach, A., Horne, T. N., Lahav, M., MacDonald, R., Michaels, R. B., Narayanan, A., et al. (2022) · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3517428.3544819 · Cited by 71
Demystifying ChatGPT: An In-depth Survey of OpenAI’s Robust Large Language Models
Bhattacharya, P., Prasad, V. K., Verma, A., Gupta, D. K., Sapsomboon, A., Viriyasitavat, W., & Dhiman, G. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10115-5 · Cited by 84
Introduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63
Schramowski, P., Tauchmann, C., & Kersting, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533192 · Cited by 86
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, A. S., Masoud, M., Mitchell, M., Radev, D., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.3 · Cited by 77
Fairness in Large Language Models: A Taxonomic Survey
Chu, Z., Wang, Z., & Zhang, W. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3682112.3682117 · Cited by 106
BiasAsker: Measuring the Bias in Conversational AI System
Wan, Y., Wang, W., He, P., Gu, J., Bai, H., & Lyu, M. R. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616310 · Cited by 54
RADio – Rank-Aware Divergence Metrics to Measure Normative Diversity in News Recommendations
Vrijenhoek, S., Bénédict, G., Granada, M. G., Odijk, D., & de Rijke, M. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3546780 · Cited by 61
On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark
Sun, H., Xu, G., Deng, J., Cheng, J., Zheng, C., Zhou, H., Peng, N., Zhu, X., & Huang, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.308 · Cited by 71
A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models
Derner, E., Batistič, K., Zahálka, J., & Babuška, R. (2024) · IEEE Access · doi:10.1109/access.2024.3450388 · Cited by 74
Cao, Y., Pruksachatkun, Y., Chang, K.-W., Gupta, R., Kumar, V., Dhamala, J., & Galstyan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.62 · Cited by 70
Self-citationOvalle, A., Goyal, P., Dhamala, J., Jaggers, Z., Chang, K.-W., Galstyan, A., Zemel, R., & Gupta, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594078 · Cited by 43
Self-citationLarge language models meet NLP: a survey
Qin, L., Chen, Q., Feng, X., Wu, Y., Zhang, Y., Li, Y., Li, M., Che, W., & Yu, P. S. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50472-3 · Cited by 203
Berman, G., Goyal, N., & Madaio, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642398 · Cited by 51
Gray, M., Samala, R. K., Liu, Q., Skiles, D., Xu, J., Tong, W., & Wu, L. (2023) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.3117 · Cited by 47
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
Understanding Human-AI Workflows for Generating Personas
Shin, J., Hedderich, M. A., Rey, B. J., Lucero, A., & Oulasvirta, A. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660729 · Cited by 58
Assessing and alleviating state anxiety in large language models
Ben‐Zion, Z., Witte, K., Jagadish, A. K., Duek, O. A., Harpaz‐Rotem, I., Khorsandian, M.-C., Burrer, A., Seifritz, E., Homan, P., Schulz, E., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01512-6 · Cited by 59
Wordcraft: a Human-AI Collaborative Editor for Story Writing
Coenen, A., Davis, L., Ippolito, D., Reif, E., & Yuan, A. (2021) · arXiv · doi:10.48550/arxiv.2107.07430 · Cited by 35
Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources
Mehrabi, N., Zhou, P., Morstatter, F., Pujara, J., Ren, X., & Galstyan, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.410 · Cited by 35
Disambiguating Algorithmic Bias: From Neutrality to Justice
Edenberg, E., & Wood, A. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604695 · Cited by 38
Is Generative AI Increasing the Risk for Technology‐Mediated Trauma Among Vulnerable Populations?
Abdulai, A. (2024) · Nursing Inquiry · doi:10.1111/nin.12686 · Cited by 36
When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization
Ladhak, F., Durmus, E., Suzgun, M., Zhang, T., Jurafsky, D., McKeown, K., & Hashimoto, T. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.234 · Cited by 30
Reward modeling for mitigating toxicity in transformer-based language models
Faal, F., Schmitt, K., & Yu, J. Y. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03944-z · Cited by 35
Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal
Gupta, U., Dhamala, J., Kumar, V., Verma, A., Pruksachatkun, Y., Krishna, S., Gupta, R., Chang, K.-W., Steeg, G. V., & Galstyan, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.55 · Cited by 38
Self-citationBounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints
Lu, A., Zhang, H., Zhang, Y., Wang, X., & Yang, D. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.148 · Cited by 52
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 73
Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
Neumann, A., Kirsten, E., Zafar, M. B., & Singh, J. P. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732038 · Cited by 47
LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions
Hundt, A., Azeem, R., Mansouri, M. I., & Brandão, M. (2025) · International Journal of Social Robotics · doi:10.1007/s12369-025-01301-x · Cited by 56
Evaluation of African American Language Bias in Natural Language Generation
Deas, N., Grieser, J. A., Kleiner, S., Patton, D. U., Turcan, E., & McKeown, K. R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.421 · Cited by 41
Red teaming large language models: A comprehensive review and critical analysis
Jabbar, M. S., Al-Azani, S., Alotaibi, A., & Ahmed, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104239 · Cited by 31
Do-Not-Answer: Evaluating Safeguards in LLMs
Wang, Y., Li, H., Han, X., Nakov, P., & Baldwin, T. (2024) · Findings of the Association for Computational Linguistics: EACL 2024 · doi:10.18653/v1/2024.findings-eacl.61 · Cited by 21
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 22
Fairness-Aware Structured Pruning in Transformers
Zayed, A., Mordido, G., Shabanian, S., Baldini, I., & Chandar, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30256 · Cited by 25
Nguyen, H., Nguyen, V., Ludovise, S., & Santagata, R. (2024) · Learning Media and Technology · doi:10.1080/17439884.2024.2435834 · Cited by 26
Choose Your Lenses: Flaws in Gender Bias Evaluation
Orgad, H., & Belinkov, Y. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.17 · Cited by 24
Integration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 20
KoBBQ: Korean Bias Benchmark for Question Answering
Jin, J., Kim, J., Lee, N., Yoo, H., Oh, A. H., & Lee, H. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00661 · Cited by 27
Ali, M., Rao, P. V., Mai, Y., & Xie, B. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671097 · Cited by 19
A DSL for Testing LLMs for Fairness and Bias
Morales, S., Clarisó, R., & Cabot, J. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3640310.3674093 · Cited by 22
Gender Bias in Natural Language Processing and Computer Vision: A Comparative Survey
Bartl, M., Mandal, A., Leavy, S., & Little, S. (2024) · ACM Computing Surveys · doi:10.1145/3700438 · Cited by 23
Winder, P. D., Hildebrand, C., & Hartmann, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0325459 · Cited by 25
Using Natural Sentence Prompts for Understanding Biases in Language Models
Alnegheimish, S., Guo, A., & Sun, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.203 · Cited by 16
How Gender Debiasing Affects Internal Model Representations, and Why It Matters
Orgad, H., Goldfarb-Tarrant, S., & Belinkov, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.188 · Cited by 27
A hybrid human–AI tool for scientometric analysis
Correia, A., Grover, A., Jameel, S., Schneider, D., Antunes, P. A., & Fonseca, B. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10548-7 · Cited by 18
Social-Group-Agnostic Bias Mitigation via the Stereotype Content Model
Omrani, A., Ziabari, A. S., Yu, C., Golazizian, P., Kennedy, B., Atari, M., Ji, H., & Dehghani, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.227 · Cited by 16
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Fleisig, E., Amstutz, A., Atalla, C., Blodgett, S. L., Daumé III, H., Olteanu, A., Sheng, E., Vann, D., & Wallach, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.343 · Cited by 14
“BOLD (Dhamala et al., 2021) contains examples of AI-generated text labeled automatically for toxicity, sentiment, “regard” toward targeted demographic groups, psycholinguistic norms, and gender polarity.”
BackgroundLiu, H., Cao, Y., Wu, X., Qiu, C., Gu, J., Liu, M., & Hershcovich, D. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104099 · Cited by 19
Tang, K., Zhou, W., Zhang, J., Liu, A., Deng, G., Li, S., Qi, P., Zhang, W., Zhang, T., & Yu, N. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670284 · Cited by 26
On Measuring Social Biases in Prompt-Based Multi-Task Learning
Akyürek, A. F., Paik, S., Kocyigit, M. Y., Akbıyık, S., Runyun, Ş. L., & Wijaya, D. T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.42 · Cited by 21
Ferguson, S. A., Aoyagui, P. A., Rizvi, R., Kim, Y., & Kuzminykh, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687056 · Cited by 17
Lazy Data Practices Harm Fairness Research
Simson, J., Fabris, A., & Kern, C. A. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658931 · Cited by 13
Debiasing large language models: research opportunities*
Yogarajan, V., Dobbie, G., & Keegan, T. T. (2024) · Journal of the Royal Society of New Zealand · doi:10.1080/03036758.2024.2398567 · Cited by 18
Societal Biases in Language Generation: Progress and ChallengesPreprint
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04054 · Cited by 12
Self-citationWhich Stereotypes Are Moderated and Under-Moderated in Search Engine Autocompletion?
Leidinger, A., & Rogers, R. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594062 · Cited by 12
Layered Bias: Interpreting Bias in Pretrained Large Language Models
Prakash, N., & Lee, R. K.-W. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.22 · Cited by 15
Abrar, A., Oeshy, N. T., Kabir, M., & Ananiadou, S. (2025) · AI & Society · doi:10.1007/s00146-025-02721-z · Cited by 24
LaMPost: AI Writing Assistance for Adults with Dyslexia Using Large Language Models
Goodman, S. M., Buehler, E., Clary, P., Coenen, A., Donsbach, A., Horne, T. N., Lahav, M., MacDonald, R., Michaels, R. B., Narayanan, A., et al. (2024) · Communications of the ACM · doi:10.1145/3626952 · Cited by 16
A Causal Explainable Guardrails for Large Language Models
Chu, Z., Wang, Y., Li, L., Wang, Z., Qin, Z., & Ren, K. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690217 · Cited by 27
AI metrics and policymaking: assumptions and challenges in the shaping of AI
Sioumalas-Christodoulou, K., & Tympas, A. (2025) · AI & Society · doi:10.1007/s00146-025-02181-5 · Cited by 12
Artificial Intelligence in Scientific Writing: A Deuteragonistic Role?
Wohlfarth, B., Streit, S. R., & Guttormsen, S. (2023) · Cureus · doi:10.7759/cureus.45513 · Cited by 13
Toxicity in Multilingual Machine Translation at Scale
Costa‐jussà, M. R., Smith, E., Ropers, C., Licht, D. J., Maillard, J., Ferrando, J., & Escolano, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.642 · Cited by 24
Measuring Gender Bias in Contextualized Embeddings
Katsarou, S., Rodríguez-Gálvez, B., & Shanahan, J. (2022) · doi:10.3390/cmsf2022003003 · Cited by 11
Discovering and Mitigating Biases in CLIP-based Image Editing
Tanjim, M. M., Singh, K. K., Kafle, K., Sinha, R., & Cottrell, G. W. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00296 · Cited by 11
Intersectional biases in narratives produced by open-ended prompting of generative language models
Shieh, E., Vassel, F.-M., Sugimoto, C. R., & Monroe‐White, T. (2026) · Nature Communications · doi:10.1038/s41467-025-68004-9 · Cited by 21
On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz Experiments
Fang, J., Aréchiga, N., Namikoshi, K., Bravo, N. S., Hogan, C., & Shamma, D. A. (2024) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3652988.3673967 · Cited by 14
Ethical AI Development: Mitigating Bias in Generative Models
Jadon, A. (2025) · Springer eBooks · doi:10.1007/978-3-031-87931-9_10 · Cited by 9
Cantini, R., Orsino, A., Ruggiero, M., & Talia, D. (2025) · Machine Learning · doi:10.1007/s10994-025-06862-6 · Cited by 34
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9
Faltings, F., Galley, M., Brantley, K., Peng, B., Cai, W., Zhang, Y., Gao, J., & Dolan, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.270 · Cited by 11
Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems
Wan, Y., Zhao, J., Chadha, A. R., Peng, N., & Chang, K.-W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.648 · Cited by 8
Self-citationA Brief Survey on Safety of Large Language Models
Gao, Z., Liu, X., Lan, Y.-S., & Yang, Z. (2024) · Journal of Computing and Information Technology · doi:10.20532/cit.2024.1005778 · Cited by 11
Raza, S., Bamgbose, O., Ghuge, S., Tavakoli, F., Reji, D. J., & Bashir, S. R. (2025) · Machine Learning · doi:10.1007/s10994-025-06767-4 · Cited by 34
Metadata
- Authors
- Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R.
- Affiliations
- Amazon (United States) (US)
- University of California, Santa Barbara (US)
- ORCID
- Dhamala, Jwala
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 862-872
- OpenAlex
- W3123340107
- Cited by
- 244
- References
- 24
- Citations by year
- 2026: 48 · 2025: 65 · 2024: 57 · 2023: 45 · 2022: 21 · 2021: 8
- FWCI
- 29.66
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- social bias, open-ended text generation, bias metrics, toxicity, psycholinguistic norms, gender polarity, gender bias, racial bias, religious bias, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
