CoChat
Sign Up
Cited by 43Ethics and Social Impacts of AILGBTQ Health, Identity, and PolicyGender Studies in Language

“I’m fully who I am”: Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation

Open access
Ovalle, Anaelia, Goyal, Palash, Dhamala, Jwala, Jaggers, Zachary, Chang, Kai-Wei, Galstyan, Aram, Zemel, Richard, Gupta, Rahul·2023·2023 ACM Conference on Fairness Accountability and Transparency
DOI 10.1145/3593013.3594078
Open this paper in CoChat — free account

Abstract

Warning: This paper contains examples of gender non-affirmative language which could be offensive, upsetting, and/or triggering. Transgender and non-binary (TGNB) individuals disproportionately experience discrimination and exclusion from daily life. Given the recent popularity and adoption of language generation technologies, the potential to further marginalize this population only grows. Although a multitude of NLP fairness literature focuses on illuminating and addressing gender biases, assessing gender harms for TGNB identities requires understanding how such identities uniquely interact with societal gender norms and how they differ from gender binary-centric perspectives. Such measurement frameworks inherently require centering TGNB voices to help guide the alignment between gender-inclusive NLP and whom they are intended to serve. Towards this goal, we ground our work in the TGNB community and existing interdisciplinary literature to assess how the social reality surrounding experienced marginalization of TGNB persons contributes to and persists within Open Language Generation (OLG). This social knowledge serves as a guide for evaluating popular large language models (LLMs) on two key aspects: (1) misgendering and (2) harmful responses to gender disclosure. To do this, we introduce TANGO, a dataset of template-based real-world text curated from a TGNB-oriented community. We discover a dominance of binary gender norms reflected by the models; LLMs least misgendered subjects in generated text when triggered by prompts whose subjects used binary pronouns. Meanwhile, misgendering was most prevalent when triggering generation with singular they and neopronouns. When prompted with gender disclosures, TGNB disclosure generated the most stigmatizing language and scored most toxic, on average. Our findings warrant further research on how TGNB harms manifest in LLMs and serve as a broader case study toward concretely grounding the design of gender-inclusive AI in community voices and interdisciplinary literature.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3593013.3594078

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Commercial LLMs can be influenced by the political priorities of their creators or in the data that they are trained on, potentially allowing non-academics to influence decisions about funding research into sensitive topics.

    “A more fundamental limitation is that commercial LLMs can be influenced by the political priorities of their creators or in the data that they are trained on, potentially allowing non-academics to influence decisions about funding research into sensitive topics, such as gender identity (e.g., Ovalle et al. 2023;Voutyrakou and Voutyrakos 2026).”

    From Scoring Grant Applications with Large Language Models (2026) · cites this paper as Ovalle et al. 2023

    The Introduction (p. 1) says LLMs may propagate societal biases, and the Conclusion (p. 10) reports gender-related biases in generated text. The paper does not connect those biases to creators’ political priorities or say that non-academics can influence research-funding decisions. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 58 · Cited by 43

Research works printed but not linked: 46

  1. [3][3] Y Gavriel Ansara and Peter Hegarty. 2013. Misgendering in English language contexts: Applying non-cisgenderist methods to feminist research. International Journal of Multiple Research Approaches 7, 2 (2013), 160-177. -
  2. [4][4] Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. arXiv preprint arXiv:2106.03521 (2021). - · arXiv
  3. [6][6] Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media , Vol. 14. 830-839. -
  4. [7][7] Bronwyn M Bjorkman. 2017. Singular they and the syntactic representation of gender in English. Glossa: a journal of general linguistics 2, 1 (2017). -
  5. [9][9] Walter O Bockting, Michael H Miner, Rebecca E Swinburne Romine, Autumn Hamilton, and Eli Coleman. 2013. Stigma, mental health, and resilience in an online sample of the US transgender population. American journal of public health 103, 5 (2013), 943-951. -
  6. [10][10] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016). -
  7. [11][11] Sabrina Burtscher and Katta Spiel. 2020. " But where would I even start?" developing (gender) sensitivity in HCI research and practice. In Proceedings of the Conference on Mensch und Computer . 431-441. -
  8. [12][12] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186. -
  9. [13][13] [Cambridge. (n.d.). Determiners used as pronouns. https://dictionary.cambridge. org/us/grammar/british-grammar/determiners-used-as-pronouns.](https://dictionary.cambridge.org/us/grammar/british-grammar/determiners-used-as-pronouns) -
  10. [16][16] Kristen Clements-Nolle, Rani Marx, and Mitchell Katz. 2006. Attempted suicide among transgender persons: The influence of gender-based discrimination and victimization. Journal of homosexuality 51, 3 (2006), 53-69. -
  11. [18][18] Kirby Conrod. 2019. Pronouns raising and emerging . Ph. D. Dissertation. -
  12. [19][19] Jamell Dacon, Harry Shomer, Shaylynn Crum-Dacon, and Jiliang Tang. 2022. Detecting Harmful Online Conversational Content towards LGBTQIA+ Individuals. arXiv preprint arXiv:2207.10032 (2022). - · arXiv
  13. [20][20] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta Optimizers. arXiv preprint arXiv:2212.10559 (2022). - · arXiv
  14. [21][21] Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff M Phillips, and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges in non-binary representation in language technologies. arXiv preprint arXiv:2108.12084 (2021). - · arXiv
  15. [22][22] Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, et al. 2022. On Measures of Biases and Harms in NLP. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022 . 246-267. -
  16. [25][25] Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2019. Queens are powerful too: Mitigating gender bias in dialogue generation. arXiv preprint arXiv:1911.03842 (2019). - · arXiv
  17. [26][26] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A Survey for In-context Learning. arXiv preprint arXiv:2301.00234 (2022). - · arXiv
  18. [27][27] Virginia K Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2022. Towards WinoQueer: Developing a Benchmark for Anti-Queer Bias in Large Language Models. arXiv preprint arXiv:2206.11484 (2022). - · arXiv
  19. [29][29] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020). - · arXiv
  20. [31][31] Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google's perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138 (2017). - · arXiv
  21. [33][33] MSazzad Hussain, Juchen Li, Louise A Ellis, Laura Ospina-Pinillos, Tracey A Davenport, Rafael A Calvo, and Ian B Hickie. 2015. Moderator assistant: A natural language generation-based intervention to support mental health via social media. Journal of Technology in Human Services 33, 4 (2015), 304-329. -
  22. [35][35] Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015. Likert scale: Explored and explained. British journal of applied science & technology 7, 4 (2015), 396. -
  23. [36][36] Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022. Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender. arXiv preprint arXiv:2202.11923 (2022). - · arXiv
  24. [37][37] Haochen Liu, Wentao Wang, Yiqi Wang, Hui Liu, Zitao Liu, and Jiliang Tang. 2020. Mitigating gender bias for neural dialogue generation with adversarial learning. arXiv preprint arXiv:2009.13028 (2020). - · arXiv
  25. [39][39] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). - · arXiv
  26. [41][41] Kevin A McLemore. 2018. A minority stress perspective on transgender individuals' experiences with misgendering. Stigma and Health 3, 1 (2018), 53. -
  27. [43][43] Debora Nozza, Federico Bianchi, Anne Lauscher, and Dirk Hovy. 2022. Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion . 26-34. -
  28. [45][45] Pearson. (n.d.). Gender Policing and Gender Accountability. https://revelpreview. pearson.com/epubs/pearson\_kimmel\_soc/OPS/xhtml/ch09\_pg0013.xhtml. [Accessed 25-Jan-2023]. -
  29. [46][46] Adam Poulsen, Eduard Fosch-Villaronga, and Roger Andre Søraa. 2020. Queering machines. Nature Machine Intelligence 2, 3 (2020), 152-152. -
  30. [47][47] Jae A Puckett, Alix B Aboussouan, Allura L Ralston, Brian Mustanski, and Michael E Newcomb. 2021. Systems of cissexism and the daily production of stress for transgender and gender diverse people. International Journal of Transgender Health (2021), 1-14. -
  31. [48][48] Jae A Puckett, Meredith R Maroney, Lauren P Wadsworth, Brian Mustanski, and Michael E Newcomb. 2020. Coping with discrimination: The insidious effects of gender minority stigma on depression and anxiety in transgender individuals. Journal of Clinical Psychology 76, 1 (2020), 176-194. -
  32. [50][50] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9. -
  33. [51][51] Inioluwa Deborah Raji, Morgan Klaus Scheuerman, and Razvan Amironesei. 2021. You can't sit with us: Exclusionary pedagogy in ai ethics education. In Proceedings - of the 2021 ACM Conference on Fairness, Accountability, and Transparency . 515525. -
  34. [52][52] Alejandro Ramos-Soto, Alberto Bugarín, and Senén Barro. 2016. On the role of linguistic descriptions of data in the building of natural language generation systems. Fuzzy Sets and Systems 285 (2016), 31-51. -
  35. [54][54] Brian A Rood, Sari L Reisner, Francisco I Surace, Jae A Puckett, Meredith R Maroney, and David W Pantalone. 2016. Expecting rejection: Understanding the minority stress experiences of transgender and gender-nonconforming individuals. Transgender Health 1, 1 (2016), 151-164. -
  36. [55][55] Koustuv Saha, Sang Chan Kim, Manikanta D Reddy, Albert J Carter, Eva Sharma, Oliver L Haimson, and Munmun De Choudhury. 2019. The language of LGBTQ+ minority stress experiences on social media. Proceedings of the ACM on humancomputer interaction 3, CSCW (2019), 1-22. -
  37. [56][56] Tulika Saha, Saraansh Chopra, Sriparna Saha, Pushpak Bhattacharyya, and Pankaj Kumar. 2021. A large-scale dataset for motivational dialogue system: An application of natural language generation to mental health. In 2021 International Joint Conference on Neural Networks (IJCNN) . IEEE, 1-8. -
  38. [57][57] Anthony J Sanford and Ruth Filik. 2007. 'They' as a gender-unspecified singular pronoun: Eye tracking reveals a processing cost. Quarterly Journal of Experimental Psychology 60, 2 (2007), 171-178. -
  39. [58][58] Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118 (2020). - · arXiv
  40. [60][60] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2020. Towards controllable biases in language generation. arXiv preprint arXiv:2005.00268 (2020). - · arXiv
  41. [62][62] Jeanette Silveira. 1980. Generic masculine words and thinking. Women's Studies International Quarterly 3, 2-3 (1980), 165-178. -
  42. [63][63] Yolande Strengers, Lizhen Qu, Qiongkai Xu, and Jarrod Knibbe. 2020. Adhering, steering, and queering: Treatment of gender in natural language generation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-14. -
  43. [65][65] Rylan J Testa, Janice Habarth, Jayme Peta, Kimberly Balsam, and Walter Bockting. 2015. Development of the gender minority stress and resilience measure. Psychology of Sexual Orientation and Gender Diversity 2, 1 (2015), 65. -
  44. [67][67] Alayo Tripp and Benjamin Munson. 2022. Perceiving gender while perceiving language: Integrating psycholinguistics and gender theory. Wiley Interdisciplinary Reviews: Cognitive Science 13, 2 (2022), e1583. -
  45. [68][68] Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and PoSen Huang. 2021. Challenges in detoxifying language models. arXiv preprint arXiv:2109.07445 (2021). - · arXiv
  46. [69][69] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022). · arXiv

Authors
Ovalle, A., Goyal, P., Dhamala, J., Jaggers, Z., Chang, K.-W., Galstyan, A., Zemel, R., & Gupta, R.
Affiliations
  • University of California, Los Angeles (US)
  • Amazon (United States) (US)
ORCID
Venue
2023 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2023-06-12
Pages
1246-1266
DOI
10.1145/3593013.3594078
OpenAlex
W4377157114
Cited by
43
References
58
Citations by year
2026: 9 · 2025: 18 · 2024: 10 · 2023: 6
FWCI
10.77
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Social Psychology › LGBTQ Health, Identity, and Policy
  • Social Sciences › Gender Studies › Gender Studies in Language
Keywords
transgender and nonbinary people, large language models, misgendering, gender bias, gender disclosure, singular they, binary gender norms, bias measurement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review