CoChat
Sign Up
Cited by 97Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionTopic Modeling

“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset

Open access
Smith, Eric Michael, Hall, Melissa, Kambadur, Melanie, Presani, Eleonora, Williams, Adina·2022·Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2022.emnlp-main.625
Open this paper in CoChat — free accountOpen PDF

Abstract

As language models grow in popularity, it becomes increasingly important to clearly measure all possible markers of demographic identity in order to avoid perpetuating existing societal harms.Many datasets for measuring bias currently exist, but they are restricted in their coverage of demographic axes and are commonly used with preset bias tests that presuppose which types of biases models can exhibit.In this work, we present a new, more inclusive bias measurement dataset, HOLIS-TICBIAS, which includes nearly 600 descriptor terms across 13 different demographic axes.HOLISTICBIAS was assembled in a participatory process including experts and community members with lived experience of these terms.These descriptors combine with a set of bias measurement templates to produce over 450,000 unique sentence prompts, which we use to explore, identify, and reduce novel forms of bias in several generative models.We demonstrate that HOLISTICBIAS is effective at measuring previously undetectable biases in token likelihoods from language models, as well as in an offensiveness classifier.We will invite additions and amendments to the dataset, which we hope will serve as a basis for more easy-to-use and standardized methods for evaluating bias in NLP models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.emnlp-main.625

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • HolisticBias contains 460,000 sentence prompts corresponding to 13 demographic axes with nearly 600 associated descriptor terms, generated with a participatory process with members of the social groups.

    “HolisticBias (Smith et al. 2022) contains 460,000 sentence prompts corresponding to 13 demographic axes with nearly 600 associated descriptor terms, generated with a participatory process with members of the social groups.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Smith et al. 2022

  • Each HolisticBias sentence contains a demographic descriptor term in a conversational context, formed from sentence templates with inserted identity words.

    “Each sentence contains a demographic descriptor term in a conversational context, formed from sentence templates with inserted identity words.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Smith et al. 2022

  • HolisticBias can also be used as a prompting dataset, with several instances framed as questions.

    “HolisticBias (Smith et al. 2022), described in Section 4.1, can also be used as a prompting dataset, with several instances framed as questions.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Smith et al. 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1889-1904, Online. Association for Computational Linguistics. · DOI
  2. Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021. Just say no: Analyzing the stance of neu- ral dialogue generation in offensive contexts. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 4846- 4862, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  3. Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. Redditbias: A real-world resource for bias evaluation and debiasing of conversational lan- guage models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941-1955.
  4. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the inter- national AAAI conference on web and social media, volume 14, pages 830-839.
  5. Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 610-623.
  6. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5454- 5476, Online. Association for Computational Lin- guistics. · DOI · Paper page
  7. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fair- ness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1004-1015, Online. Association for Computational Linguistics. · DOI · Paper page
  8. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Ad- vances in Neural Information Processing Systems 29: Annual Conference on Neural Information Process- ing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4349-4357.
  9. Beth Haller, Bruce Dorries, and Jessica Rahn. 2006. Media labeling versus the us disability community identity: a study of shifting cultural language. Dis- ability & Society, 21(1):61-75.
  10. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169-182, Dublin, Ireland. Association for Computational Lin- guistics.
  11. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stan- forth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing sen- timent bias in language models via counterfactual evaluation. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 65-83, Online. Association for Computational Linguistics. · DOI · Paper page
  12. Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021. On transferability of bias mitigation ef- fects in language model fine-tuning. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3770-3783, Online. Association for Computational Linguistics. · DOI
  13. Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the- box: An empirical analysis of intersectional occupa- tional biases in popular generative language models. Advances in Neural Information Processing Systems, 34.
  14. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in con- textualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Lan- guage Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  15. Jungseob Lee, Midan Shim, Suhyune Son, Yujin Kim, Chanjun Park, and Heuiseok Lim. 2022. Empiri- cal study on blenderbot 2.0 errors analysis in terms of model, data and user-centric approach. arXiv preprint arXiv:2201.03239. · arXiv
  16. Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with opti- mized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087. · arXiv
  17. Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. 2020. Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5502-5515.
  18. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020. Does gender matter? to- wards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4403-4416.
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  20. Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Ad- ina Williams, and Douwe Kiela. 2021. Dynaboard: An evaluation-as-a-service platform for holistic next- generation benchmarking. In Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 10351-10367.
  21. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochasti- cally larger than the other. The annals of mathemati- cal statistics, pages 50-60.
  22. A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bor- des, D. Parikh, and J. Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476. · arXiv
  23. Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, and Samuel R. Bowman. 2021. What ingredients make for an effective crowdsourc- ing protocol for difficult NLU data collection tasks? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1221-1235, Online. Association for Computational Linguistics. · DOI
  24. Alexandra Olteanu, Kartik Talamadupula, and Kush R Varshney. 2017. The limits of abstract evaluation metrics: The case of hate speech detection. In Pro- ceedings of the 2017 ACM on web science conference, pages 405-406.
  25. Zoe Papakipos and Joanna Bitton. 2022. Augly: Data augmentations for robustness. arXiv preprint arXiv:2201.06494. · arXiv
  26. Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. 2022. Per- turbation augmentation for fairer nlp. arXiv preprint arXiv:2205.12586. · arXiv
  27. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5370-5381.
  28. Adithya Renduchintala, Denise Diaz, Kenneth Heafield, Xian Li, and Mona Diab. 2021. Gender bias ampli- fication during speed-quality optimization in neural machine translation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 99-109, Online. Association for Computational Linguistics. 9192 · DOI
  29. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
  30. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics. · DOI
  31. Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, and Nanyun Peng. 2021a. Revealing per- sona biases in dialogue systems. arXiv preprint arXiv:2104.08728. · arXiv
  32. Kurt Shuster, Eric Michael Smith, Da Ju, and Jason Weston. 2021. Multi-modal open-domain dialogue. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 4863-4883.
  33. Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020a. Control- ling style in generated dialogue. arXiv preprint arXiv:2009.10855. · arXiv
  34. Eric Michael Smith and Adina Williams. 2021. Hi, my name is martha: Using names to measure and mitigate bias in generative dialogue models. arXiv preprint arXiv:2109.03300. · arXiv
  35. Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020b. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021-2030.
  36. Tom W Smith. 1992. Changing racial labels: From "colored" to "negro" to "black" to "african american". Public Opinion Quarterly, 56(4):496-514.
  37. Nancy N Soja, Susan Carey, and Elizabeth S Spelke. 1991. Ontological categories guide young children's inductions of word meaning: Object terms and sub- stance terms. Cognition, 38(2):179-211.
  38. Anna Sotnikova, Yang Trista Cao, Hal Daumé III, and Rachel Rudinger. 2021. Analyzing stereotypes in generative text inference tasks. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4052-4065, Online. Association for Computational Linguistics. · DOI
  39. Yi Chern Tan and L. Elisa Celis. 2019. Assessing so- cial and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13209-13220.
  40. Emiel Van Miltenburg, Desmond Elliott, and Piek Vossen. 2018. Talking about other people: an end- less range of possibilities. In Proceedings of the 11th International Conference on Natural Language Generation, pages 415-420.
  41. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stu- art M. Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. 9193
  42. Alex Wang and Kyunghyun Cho. 2019. Bert has a mouth, and it must speak: Bert as a markov ran- dom field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36.
  43. Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, and Xuanjing Huang. 2021. TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 347-355, Online. Association for Computational Linguistics. · DOI
  44. Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2020. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032. · arXiv
  45. Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, and Furu Wei. 2021a. Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 10653-10659, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  46. Jing Xu, Arthur Szlam, and Jason Weston. 2022. Be- yond goldfish memory: Long-term open-domain con- versation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180-5197, Dublin, Ireland. Association for Computational Linguistics.
  47. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213.
  48. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020. Dialogpt: Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278.
  49. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629-634, Min- neapolis, Minnesota. Association for Computational Linguistics. · DOI
  50. Lal Zimman and Will Hayworth. 2020. How we got here: Short-scale change in identity labels for trans, cis, and non-binary people in the 2000s. Proceedings of the Linguistic Society of America, 5(1):499-513.

Cited by

Sorted by citations

Authors
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A.
ORCID
Venue
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2022
Pages
9180-9211
DOI
10.18653/v1/2022.emnlp-main.625
OpenAlex
W4385574250
Cited by
97
References
22
Citations by year
2026: 28 · 2025: 25 · 2024: 21 · 2023: 22 · 2022: 1
FWCI
15.77
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
bias in language models, bias mitigation, generative language models, social harm

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review