CoChat
Sign Up
Cited by 56Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionSentiment Analysis and Opinion Mining

Bias against 93 stigmatized groups in masked language models and downstream sentiment classification tasks

Open access
Mei, Katelyn, Fereidooni, Sonia, Caliskan, Aylin·2023·2023 ACM Conference on Fairness Accountability and Transparency
DOI 10.1145/3593013.3594109
Open this paper in CoChat — free accountOpen PDF

Abstract

Warning: The content of this paper may be upsetting or triggering.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3593013.3594109

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The presence of biases in large language models has been well-documented, including negative sentiment and toxicity directed towards some social groups, stereotypical linguistic associations, and lack of recognition of certain language dialects.

    “From negative sentiment and toxicity directed towards some social groups, to stereotypical linguistic associations, to lack of recognition of certain language dialects, the presence of biases of LLMs have been well-documented ( e.g. , Blodgett and O'Connor 2017; Hutchinson et al. 2020; Mei, Fereidooni, and Caliskan 2023; Mˇ echura 2022; Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019; Shen”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Mei, Fereidooni, and Caliskan 2023

    The Discussion (page unavailable) reports that sentiment classifiers are more likely to assign negative sentiment to stigmatized groups, and §2.2 describes stereotypical associations in language models. The cited paper does not substantiate the sentence’s examples of toxicity directed at groups or failure to recognize certain dialects. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Sentiment classifiers may incorrectly classify statements about stigmatized groups as negative.

    “and sentiment classifiers may incorrectly classify statements about stigmatized groups ( e.g. , people with disabilities, mental illness, or low socioeconomic status) as negative (Mei, Fereidooni, and Caliskan 2023).”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Mei, Fereidooni, and Caliskan 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. In Proceedings of the 2021 AAAI/ACM Confer- ence on AI, Ethics, and Society (Virtual Event, USA) (AIES '21). Association for Computing Machinery, New York, NY, USA, 298-306. https://doi.org/10.1145/ 3461702.3462624 · DOI · Paper page
  2. Gary L. Albrecht, Verónica García Walker, and Judith A. Levy. 1982. Social distance from the stigmatized. A test of two theories. Social science & medicine 16 14 (1982), 1319-27.
  3. Sarah Alnegheimish, Alicia Guo, and Yi Sun. 2022. Using Natural Sentence Prompts for Understanding Biases in Language Models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Association for Computational Linguis- tics, Seattle, United States, 2824-2830. https://doi.org/10.18653/v1/2022.naacl- main.203 · DOI
  4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Lan- guage (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050 (2020). · arXiv
  5. Shikha Bordia and Samuel R. Bowman. 2019. Identifying and Reducing Gender Bias in Word-Level Language Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. Association for Computational Linguistics, Minneapolis, Minnesota, 7-15. https://doi.org/10.18653/v1/N19-3002 · DOI · Paper page
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
  7. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Seman- tics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186. https://doi.org/10.1126/science.aal4230 arXiv:https://www.science.org/doi/pdf/10.1126/science.aal4230 · DOI · Paper page
  8. Patrick Corrigan, Fred E Markowitz, Amy Watson, David Rowan, and Mary Ann Kubiak. 2003. An attribution model of public discrimination towards persons with mental illness. Journal of health and Social Behavior (2003), 162-179.
  9. Christian S Crandall and Dallie Moriarty. 1995. Physical illness stigma and social rejection. British Journal of Social Psychology 34, 1 (1995), 67-83.
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171-4186.
  11. Federica Durante, Courtney Bearns Tablante, and Susan T. Fiske. 2017. Poor but Warm, Rich but Cold (and Competent): Social Classes in the Stereotype Content Model. Journal of Social Issues 73, 1 (2017), 138-157. https://doi.org/10.1111/josi. 12208 arXiv:https://spssi.onlinelibrary.wiley.com/doi/pdf/10.1111/josi.12208 · DOI
  12. Iona H Ginsburg and BRUCE G LINK. 1993. Psychosocial consequences of rejec- tion and stigma feelings in psoriasis patients. International Journal of Dermatology 32, 8 (1993), 587-591.
  13. Erving Goffman. 1986 -1963. Stigma : notes on the management of spoiled identity (first touchstone edition. ed.). Simon & Schuster, Inc., New York.
  14. Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp. 2022. More than a feeling: Accuracy and Application of Sentiment Analysis. International Journal of Research in Marketing (2022).
  15. Gregory M Herek. 2009. Hate crimes and stigma-related experiences among sexual minority adults in the United States: Prevalence estimates from a national probability sample. Journal of interpersonal violence 24, 1 (2009), 54-74.
  16. HF Canonical Model Maintainers. 2022. distilbert-base-uncased-finetuned-sst-2- english (Revision bfdd146). https://doi.org/10.57967/hf/0181 · DOI
  17. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2019. Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. https://doi.org/10.48550/ ARXIV.1911.03064 · DOI
  18. Sophie Jentzsch and Cigdem Turan. 2022. Gender Bias in BERT -Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP). Association for Computational Linguistics, Seattle, Washington, 184-199. https://doi.org/10.18653/v1/2022.gebnlp-1.20 · DOI
  19. Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. arXiv preprint arXiv:1805.04508 (2018). · arXiv
  20. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring Bias in Contextualized Word Representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics, Florence, Italy, 166-172. https://doi.org/10.18653/v1/ W19-3823 · DOI · Paper page
  21. Inna Wanyin Lin, Lucille Njoo, Anjalie Field, Ashish Sharma, Katharina Reinecke, Tim Althoff, and Yulia Tsvetkov. 2022. Gendered Mental Health Stigma in Masked Language Models. arXiv preprint arXiv:2210.15144 (2022). · arXiv
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). · arXiv
  23. Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings. In Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Lin- guistics, Minneapolis, Minnesota, 615-621. https://doi.org/10.18653/v1/N19-1062 · DOI
  24. Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. arXiv preprint arXiv:1903.10561 (2019). · arXiv
  25. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276-282.
  26. Mary E McLaughlin, Myrtle P Bell, and Donna Y Stringer. 2004. Stigma and accep- tance of persons with disabilities: Understudied aspects of workforce diversity. Group & Organization Management 29, 3 (2004), 302-333.
  27. Matthew K Meisel, Michelle Haikalis, Suzanne M Colby, and Nancy P Barnett. 2022. Education-based stigma and discrimination among young adults not in 4-year college. BMC psychology 10, 1 (2022), 26.
  28. Cilia Mejia-Lancheros, James Lachaud, Julia Woodhall-Melnik, Patricia O'Campo, Stephen W Hwang, and Vicky Stergiopoulos. 2021. Longitudinal interrelation- ships of mental health discrimination and stigma with housing and well-being outcomes in adults with mental illness and recent experience of homelessness. Social Science & Medicine 268 (2021), 113463.
  29. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereo- typical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers). Association for Computational Linguistics, Online, 5356-5371. https: //doi.org/10.18653/v1/2021.acl-long.416 · DOI · Paper page
  30. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre- trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 9-14. https://doi.org/10. 18653/v1/2020.emnlp-demos.2 · DOI
  31. Richard Parker and Peter Aggleton. 2007. HIV-and AIDS-related stigma and discrimination: A conceptual framework and implications for action. In Culture, society and sexuality. Routledge, 459-474.
  32. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learn- ing Library. In Advances in Neural Information Processing Systems 32.
  33. Curran Associates, Inc., 8024-8035. http://papers.neurips.cc/paper/9015-pytorch-an- imperative-style-high-performance-deep-learning-library.pdf
  34. Minh Hieu Phan and Philip O. Ogunbona. 2020. Modelling Context and Syntacti- cal Features for Aspect-based Sentiment Analysis. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics. Association for Com- putational Linguistics, Online, 3211-3220. https://doi.org/10.18653/v1/2020.acl- main.293 · DOI
  35. Juan Manuel Pérez, Juan Carlos Giudici, and Franco Luque. 2021. pysen- timiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks. arXiv:2106.09462 [cs.CL] · arXiv
  36. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 Task 4: Sentiment Analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, 502-518. https://doi.org/10.18653/v1/S17-2088 · DOI
  37. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR abs/1910.01108 (2019). arXiv:1910.01108 http://arxiv.org/abs/1910.01108 · arXiv
  38. Timo Schick and Hinrich Schütze. 2021. It's Not Just Size That Matters: Small Lan- guage Models Are Also Few-Shot Learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Association for Computational Linguistics, Online, 2339-2352. https://doi.org/10.18653/v1/2021.naacl-main.185 · DOI · Paper page
  39. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 4222-4235. https://doi.org/10.18653/v1/2020.emnlp-main.346 · DOI · Paper page
  40. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. "I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 9180-9211. https://aclanthology. org/2022.emnlp-main.625
  41. Rachel A. Smith. 2007. Language of the Lost: An Explication of Stigma Com- munication. Communication Theory 17, 4 (10 2007), 462-485. https://doi. org/10.1111/j.1468-2885.2007.00307.x arXiv:https://academic.oup.com/ct/article- pdf/17/4/462/21953142/jcomthe0462.pdf · DOI
  42. Claude M Steele. 1997. A threat in the air: How stereotypes shape intellectual identity and performance. American psychologist 52, 6 (1997), 613.
  43. Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation. https://doi.org/10.48550/ARXIV.2107.02137 · DOI · Paper page
  44. Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. Advances in neural information processing systems 32 (2019).
  45. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems 33 (2020), 12388-12401.
  46. Robert Wolfe and Aylin Caliskan. 2021. Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 518-532. https://doi.org/10.18653/v1/2021.emnlp-main.41 · DOI
  47. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019).
  48. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A Robustly Optimized BERT Pre-training Approach with Post-training. In Proceedings of the 20th Chinese Na- tional Conference on Computational Linguistics. Chinese Information Processing Society of China, Huhhot, China, 1218-1227. https://aclanthology.org/2021.ccl- 1.108

Cited by

Sorted by citations

Authors
Mei, K., Fereidooni, S., & Caliskan, A.
Affiliations
University of Washington (US)
ORCID
Venue
2023 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2023-06-12
Pages
1699-1710
DOI
10.1145/3593013.3594109
OpenAlex
W4380319657
Cited by
56
References
6
Citations by year
2026: 17 · 2025: 23 · 2024: 11 · 2023: 5
FWCI
11.96
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
National Institute of Standards and Technology (NIST 60NANB20D212T)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
masked language models, sentiment classification, bias in language models, stigmatized groups

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review