Gender bias and stereotypes in Large Language Models
Open accessAbstract
Large Language Models (LLMs) have made substantial progress in the past several months, shattering state-of-the-art benchmarks in many domains. This paper investigates LLMs’ behavior with respect to gender stereotypes, a known issue for prior models. We use a simple paradigm to test the presence of gender bias, building on but differing from WinoBias, a commonly used gender bias dataset, which is likely to be included in the training data of current LLMs. We test four recently published LLMs and demonstrate that they express biased assumptions about men and women’s occupations. Our contributions in this paper are as follows: (a) LLMs are 3-6 times more likely to choose an occupation that stereotypically aligns with a person’s gender; (b) these choices align with people’s perceptions better than with the ground truth as reflected in official job statistics; (c) LLMs in fact amplify the bias beyond what is reflected in perceptions or the ground truth; (d) LLMs ignore crucial ambiguities in sentence structure 95% of the time in our study items, but when explicitly prompted, they recognize the ambiguity; (e) LLMs provide explanations for their choices that are factually inaccurate and likely obscure the true reason behind their predictions. That is, they provide rationalizations of their biased behavior. This highlights a key property of these models: LLMs are trained on imbalanced datasets; as such, even with the recent successes of reinforcement learning with human feedback, they tend to reflect those imbalances back at us. As with other types of societal biases, we suggest that LLMs must be carefully tested to ensure that they treat minoritized individuals and communities equitably.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If a medical LLM has been trained on data that contains misinformation or biased content, LLMs deployed for clinical decision-making are at risk of reproducing these problematic contents.
“If a medical LLM has been trained on data that contains misinformation or biased content, LLMs deployed for clinical decision-making are at risk of reproducing these problematic contents [120,124–126].”
From Bias in medical AI: Implications for clinical decision-making (2024) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
Sallam, M. (2023) · doi:10.1101/2023.02.19.23286155 · Cited by 3,123 · cited 2× · see citing works
“…in which LLM-based applications have been either proposed or developed for real-world applications in the domains of medicine [12, 38, 68, 79], law [3, 23, 73, 74], finance [15, 26, 99], education [17, 62, 79], and many others, the potential repercussions of reproducing and amplifying…”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · see citing works
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · see citing works
Expectations of brilliance underlie gender distributions across academic disciplines
Leslie, S.-J., Cimpian, A., Meyer, M., & Freeland, E. (2015) · Science · doi:10.1126/science.1261375 · Cited by 1,516 · see citing works
Lectures on Government and Binding: The Pisa Lectures
Chomsky, N. (1993) · Cited by 1,472 · see citing works
Gender stereotypes about intellectual ability emerge early and influence children's interests
Bian, L., Leslie, S.-J., & Cimpian, A. (2017) · Science · doi:10.1126/science.aah6524 · Cited by 1,206 · see citing works
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · see citing works
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1× · see citing works
“However, as is well known, language models perpetuate and occasionally amplify biases, stereotypes, and negative perceptions of minoritized groups in society [10, 13, 14, 66, 69, 84, 85, 90].”
BackgroundMath = male, me = female, therefore math ≠ me.
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.83.1.44 · Cited by 971 · see citing works
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879 · see citing works
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855 · see citing works
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 3× · see citing works
“…of the broader society that the models are embedded in. Gender bias has been shown to exist in word embeddings [9, 16, 19, 31, 51, 61, 89, 100, 101, 103], as well as in a broad array of models developed specifically for various NLP tasks, such as auto-captioning, sentiment analysis, toxicity…”
BackgroundThe role of AI in drug discovery: challenges, opportunities, and strategies
Blanco-González, A., Cabezón, A., Seco-González, A., Conde-Torres, D., Antelo-Riveiro, P., Piñeiro, Á., & Garcia-Fandino, R. (2023) · Pharmaceuticals · doi:10.3390/ph16060891 · Cited by 690 · see citing works
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · see citing works
ChatGPT for (Finance) research: The Bananarama Conjecture
Dowling, M., & Lucey, B. M. (2023) · Finance research letters · doi:10.1016/j.frl.2023.103662 · Cited by 611 · see citing works
Burgess, D., & Borgida, E. (1999) · Psychology Public Policy and Law · doi:10.1037/1076-8971.5.3.665 · Cited by 609 · see citing works
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549 · see citing works
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 522 · see citing works
Evaluating Gender Bias in Machine Translation
Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1164 · Cited by 492 · see citing works
Mitigating Gender Bias in Natural Language Processing: Literature Review
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1159 · Cited by 492 · see citing works
Persistent Anti-Muslim Bias in Large Language Models
Abid, A., Farooqi, M., & Zou, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462624 · Cited by 476 · see citing works
On Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461 · see citing works
Providing a unified account of definite noun phrases in discourse
Grosz, B. J., Joshi, A. K., & Weinstein, S. (1983) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/981311.981320 · Cited by 451 · see citing works
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · see citing works
The Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 431 · cited 2× · see citing works
“…has been shown to exist in word embeddings [9, 16, 19, 31, 51, 61, 89, 100, 101, 103], as well as in a broad array of models developed specifically for various NLP tasks, such as auto-captioning, sentiment analysis, toxicity detection, machine translation, and more [46, 58, 71, 80, 83, 87, 91, 95].”
BackgroundCan Gender-Fair Language Reduce Gender Stereotyping and Discrimination?
Sczesny, S., Formanowicz, M., & Moser, F. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00025 · Cited by 404 · see citing works
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384 · see citing works
Learning Gender-Neutral Word Embeddings
Zhao, J., Zhou, Y., Li, Z., Wang, W., & Chang, K.-W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1521 · Cited by 376 · see citing works
Social Bias Frames: Reasoning about Social and Power Implications of Language
Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.486 · Cited by 375 · see citing works
Arnold, J. (2000) · Cognition · doi:10.1016/s0010-0277(00)00073-1 · Cited by 367 · see citing works
Bozkurt, A., Xiao, J., Lambert, S., Pazurek, A., Crompton, H., Koseoglu, S., Farrow, R., Bond, M., Nerantzi, C., Honeychurch, S., et al. (2023) · Zenodo · doi:10.5281/zenodo.7636568 · Cited by 350 · see citing works
Measuring Bias in Contextualized Word Representations
Kurita, K., Vyas, N., Pareek, A., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3823 · Cited by 348 · see citing works
Gender Bias in Contextualized Word Embeddings
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordóñez, V., & Chang, K.-W. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1064 · Cited by 341 · see citing works
Gender and Dialect Bias in YouTube’s Automatic Captions
Tatman, R. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1606 · Cited by 338 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
A Computer Method for Calculating Kendall's Tau with Ungrouped Data
Knight, W. R. (1966) · Journal of the American Statistical Association · doi:10.1080/01621459.1966.10480879 · Cited by 324 · see citing works
Twenty Years of Stereotype Threat Research: A Review of Psychological Mediators
Pennington, C. R., Heim, D., Levy, A. R., & Larkin, D. T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0146487 · Cited by 324 · see citing works
Reducing Gender Bias in Abusive Language Detection
Park, J. H., Shin, J., & Fung, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1302 · Cited by 323 · see citing works
Women are underrepresented in fields where success is believed to require brilliance
Meyer, M. A., Cimpian, A., & Leslie, S. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00235 · Cited by 298 · see citing works
Release Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292 · see citing works
Large pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291 · see citing works
Gender Bias in Neural Natural Language Processing
Lu, K., Mardziel, P., Wu, F., Amancharla, P., & Datta, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62077-6_14 · Cited by 284 · see citing works
The processing of Italian subject pronouns
Carminati, M. N. (2002) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 275 · see citing works
Fairness Without Demographics in Repeated Loss MinimizationPreprint
Hashimoto, T., Srivastava, M., Namkoong, H., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.08010 · Cited by 263 · see citing works
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262 · cited 1× · see citing works
“Following Turpin et al. [93] we therefore suggest that models’ explanations often misrepresent the true reason for their predictions.”
BackgroundKirk, H. R., Jun, Y., Volpin, F., Iqbal, H., Benussi, E., Dreyer, F., Shtedritski, A., & Asano, Y. M. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2102.04130 · Cited by 251 · see citing works
Carreiras, M., Garnham, A., Oakhill, J. V., & Cain, K. (1996) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755647 · Cited by 216 · see citing works
Megahed, F. M., Chen, Y., Ferris, J. A., Knoth, S., & Jones‐Farmer, L. A. (2023) · Quality Engineering · doi:10.1080/08982112.2023.2206479 · Cited by 207 · see citing works
Lam, S. K., Uduwage, A., Dong, Z., Sen, S., Musicant, D. R., Terveen, L., & Riedl, J. (2011) · International Symposium on Wikis and Open Collaboration (WikiSym) · doi:10.1145/2038558.2038560 · Cited by 198 · see citing works
Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study
Nov, O., Singh, N., & Mann, D. (2023) · JMIR Medical Education · doi:10.2196/46939 · Cited by 198 · see citing works
Evaluating the Underlying Gender Bias in Contextualized Word Embeddings
Basta, C., Costa-jussà, M. R., & Casas, N. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3805 · Cited by 189 · see citing works
On So‐Called “Pronouns” in English, 1966
Zanuttini, R., Leu, T., & Kayne, R. (2013) · doi:10.1002/9781394260140.ch1 · Cited by 168 · see citing works
Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
Zhuo, T. Y., Huang, Y., Chen, C., & Xing, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.12867 · Cited by 164 · see citing works
Getting Gender Right in Neural Machine Translation
Vanmassenhove, E., Hardmeier, C., & Way, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1334 · Cited by 158 · see citing works
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · cited 1× · see citing works
“However, as is well known, language models perpetuate and occasionally amplify biases, stereotypes, and negative perceptions of minoritized groups in society [10, 13, 14, 66, 69, 84, 85, 90].”
BackgroundComprehending Pronouns: A Role for Word-Specific Gender Stereotype Information
Kennison, S. M., & Trofe, J. L. (2003) · Journal of Psycholinguistic Research · doi:10.1023/a:1023599719948 · Cited by 140 · see citing works
Microaggressions and traumatic stress: Theory, research, and clinical treatment.
Nadal, K. L. Y. (2018) · American Psychological Association eBooks · doi:10.1037/0000073-000 · Cited by 136 · see citing works
Pronominalization and discourse coherence, discourse structure and pronoun interpretation
Gordon, P. C., & Scearce, K. A. (1995) · Memory & Cognition · doi:10.3758/bf03197233 · Cited by 128 · see citing works
Gabriel, U., Gygax, P. M., Sarrasin, O., et al. (2008) · Behavior Research Methods · doi:10.3758/brm.40.1.206 · Cited by 122 · see citing works
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level ConstraintsPreprint
Zhao, J., Wang, T., Yatskar, M., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09457 · Cited by 122 · see citing works
Kreiner, H., Sturt, P., & Garrod, S. (2007) · Journal of Memory and Language · doi:10.1016/j.jml.2007.09.003 · Cited by 106 · see citing works
Expressions of masculinity and associations with suicidal ideation among young males
King, T. L., Shields, M., Sojo, V. E., Daraganova, G., Currier, D., O'Neil, A. E., King, K., & Milner, A. J. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-2475-y · Cited by 104 · see citing works
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97 · see citing works
ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B., Ricke, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.14882 · Cited by 97 · see citing works
Ms. Categorized: Gender, notability, and inequality on Wikipedia
Tripodi, F. B. (2021) · New Media & Society · doi:10.1177/14614448211023772 · Cited by 91 · see citing works
Nationality Bias in Text GenerationPreprint
Venkit, P. N., Gautam, S., Panchanadikar, R., et al. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2302.02463 · Cited by 88 · see citing works
Gender stereotypes and biases in early childhood: A systematic review
King, T. L., Scovelle, A. J., Meehl, A., Milner, A. J., & Priest, N. (2021) · Australasian Journal of Early Childhood · doi:10.1177/1836939121999849 · Cited by 87 · see citing works
Psychology Cannot Afford to Ignore the Many Harms Caused by Microaggressions
Williams, M. T. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691619893362 · Cited by 84 · see citing works
Evidence of immediate activation of gender information from a social role name
Reynolds, D., Garnham, A., & Oakhill, J. V. (2005) · Quarterly Journal of Experimental Psychology · doi:10.1080/02724980543000088 · Cited by 82 · see citing works
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 74 · see citing works
Nationality Bias in Text Generation
Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T.-H., & Wilson, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.9 · Cited by 74 · see citing works
ChatGPT for (Finance) Research: The Bananarama ConjecturePreprint
Dowling, M., & Lucey, B. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4322651 · Cited by 73 · see citing works
Gender Stereotyping and Prejudice in Young Children A Developmental Intergroup Perspective
Arthur, A. E., Bigler, R. S., Liben, L. S., Gelman, S. A., & Ruble, D. N. (2008) · doi:10.1093/oso/9780195189742.003.0005 · Cited by 71 · see citing works
Gender Bias in Neural Natural Language ProcessingPreprint
Lu, K., Mardziel, P., Wu, F., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11714 · Cited by 70 · see citing works
Esaulova, Y., Reali, C., & von Stockhausen, L. (2013) · Language Cognition and Neuroscience · doi:10.1080/01690965.2013.794295 · Cited by 66 · see citing works
The Implications of ChatGPT for Legal Services and Society
(2022) · doi:10.2139/ssrn.4294197 · Cited by 63 · see citing works
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, A. S., Masoud, M., Mitchell, M., Radev, D., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.3 · Cited by 61 · see citing works
Don't Touch my Projectile: Gender Bias and Stereotyping in Syntactic Examples
Macaulay, M., & Brice, C. (1997) · Language · doi:10.1353/lan.1997.0031 · Cited by 60 · see citing works
Wikipedia, sociology, and the promise and pitfalls of Big Data
Adams, J., & Brückner, H. (2015) · Big Data & Society · doi:10.1177/2053951715614332 · Cited by 58 · see citing works
The Ghost in the Machine has an American accent: value conflict in GPT-3
Johnson, R. L., Pistilli, G., Menédez-González, N., Duran, L. D. D., Panai, E., Kalpokiene, J., & Bertulfo, D. J. (2022) · arXiv · doi:10.48550/arxiv.2203.07785 · Cited by 50 · see citing works
Towards Measuring the Representation of Subjective Global Opinions in Language Models
Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16388 · Cited by 48 · see citing works
King, T. L., Singh, A., & Milner, A. J. (2019) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2019.01.011 · Cited by 40 · see citing works
Math = Male, Me = Female, therefore Math = MePreprint
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2016) · doi:10.31234/osf.io/y2g6s · Cited by 40 · see citing works
Examining Gender and Race Bias in Two Hundred Sentiment Analysis SystemsPreprint
Kiritchenko, S., & Mohammad, S. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04508 · Cited by 37 · see citing works
Learning Gender-Neutral Word EmbeddingsPreprint
Zhao, J., Zhou, Y., Li, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01496 · Cited by 33 · see citing works
A Computer Method for Calculating Kendall's Tau with Ungrouped Data
Knight, W. R. (1966) · Journal of the American Statistical Association · doi:10.2307/2282833 · Cited by 32 · see citing works
Feature-Wise Bias AmplificationPreprint
Leino, K. R. M., Black, E., Fredrikson, M., Sen, S., & Datta, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08999 · Cited by 32 · see citing works
Mitigating Gender Bias Amplification in Distribution by Posterior Regularization
Jia, S., Meng, T., Zhao, J., & Chang, K.-W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.264 · Cited by 30 · see citing works
Pipelines for Social Bias Testing of Large Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2022) · doi:10.18653/v1/2022.bigscience-1.6 · Cited by 29 · see citing works
Gender Bias and Stereotypes in Linguistic Example Sentences
Kotek, H., Dockum, R., Babinski, S., et al. (2021) · Language · doi:10.1353/lan.2021.0060 · Cited by 27 · see citing works
Self-citationEvaluating the Underlying Gender Bias in Contextualized Word EmbeddingsPreprint
Basta, C., Costa‐jussà, M. R., & Casas, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08783 · Cited by 27 · see citing works
Xie, Q., Han, W., Lai, Y., et al. (2023) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2304.05351 · Cited by 24 · see citing works
Twenty years of stereotype threat research: A review of psychological mediators.Preprint
Pennington, C. R., Heim, D., Levy, A., & Larkin, D. (2018) · doi:10.31234/osf.io/j9ae5 · Cited by 21 · see citing works
Don't Touch my Projectile: Gender Bias and Stereotyping in Syntactic Examples
Macaulay, M., & Brice, C. (1997) · Language · doi:10.2307/417327 · Cited by 20 · see citing works
Who’s Afraid of ChatGPT? An Examination of ChatGPT’s Implications for Legal WritingPreprint
Armstrong, A. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4336929 · Cited by 20 · see citing works
What are the biases in my word embedding?Preprint
Swinger, N., De‐Arteaga, M., Heffernan, N. T., Leiserson, M. D. M., & Kalai, A. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08769 · Cited by 17 · see citing works
Automation and Stock Prices: The Case of ChatGPTPreprint
Blomkvist, M., Qiu, Y., & Zhao, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4395339 · Cited by 13 · see citing works
ChatGPT, Professor of LawPreprint
Oltz, T. P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4347630 · Cited by 12 · see citing works
Azar, Z., Backus, A., Özyürek, A., et al. (2016) · Research portal (Tilburg University) · Cited by 6 · see citing works
Pronominalization and Discourse Coherence, Discourse Structure, and Pronoun Interpretation
Gordon, P. C. (1992) · PsycEXTRA Dataset · doi:10.1037/e665412011-538 · Cited by 5 · see citing works
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852
Flagged first, then by citations
Research works printed but not linked: 10
- Paola Cepeda, Hadas Kotek, Katharina Pabst, and Kristen Syrett. 2021. Gender bias in linguistics textbooks: Has anything changed since Macaulay & Brice (1997)? Language 97, 4 (2021), 678-702.
- Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2023. Deep reinforcement learning from human preferences. arXiv:1706.03741 [stat.ML] · arXiv
- Bethany Gardner. 2020. Gender bias through production about and memory for names. Ph. D. Dissertation. Vanderbilt University.
- Margaret Grant, Hadas Kotek, Jayun Bae, and Jeffrey Lamontagne. 2016. Stereo- typical Gender Effects in 2016. Presentation at CUNY Conference on Human Sentence Processing 30.
- Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A Framework for Modeling the Local Coherence of Discourse. Computational Linguistics 21, 2 (1995), 203-225. https://aclanthology.org/J95-2003
- Melissa Hart. 2005. Big Data's Disparate Impact. , 741-791 pages. https: //papers.ssrn.com/sol3/papers.cfm?abstract_id=788066
- Howard Lasnik. 1976. Remarks on Coreference. Linguistic Analysis 2 (1976), 1-22.
- Livia Polanyi and Diana Strassmann. 1996. Storytellers and gatekeepers in economics. In Rethinking language and gender research: Theory and practice, Victoria J. Bergvall, Janet M. Bing, and Alice F. Freed (Eds.). Routledge, London, 126-152.
- Steven J Spencer, Christine Logel, and Paul G Davies. 2016. Stereotype threat. Annual review of psychology 67 (2016), 415-437.
- Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. 2022. A Study of Implicit Bias in Pretrained Language Models against People with Disabilities. In Proceedings of the 29th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 1324-1332. https://aclanthology.org/2022.coling-1.113
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · see citing works
Bias in medical AI: Implications for clinical decision-making
Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000651 · Cited by 607 · see citing works
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404 · see citing works
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119 · see citing works
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 118 · see citing works
Cheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91 · see citing works
Natural language processing in the era of large language models
Zubiaga, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1350306 · Cited by 81 · see citing works
Lehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2404328121 · Cited by 66 · see citing works
Currie, G. M., Hawk, K. E., & Rohren, E. M. (2025) · Seminars in Nuclear Medicine · doi:10.1053/j.semnuclmed.2024.05.005 · Cited by 52 · see citing works
Chandra, M., Naik, S., Ford, D., Okoli, E., De Choudhury, M., Ershadi, M., Ramos, G., Hernandez, J., Bhattacharjee, A., Warreth, S., et al. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732063 · Cited by 44 · see citing works
Alvero, A. J., Lee, J., Regla-Vargas, A., Kizilcec, R. F., Joachims, T., & Antonio, A. L. (2024) · Journal of Big Data · doi:10.1186/s40537-024-00986-7 · Cited by 48 · see citing works
Klar, M. (2025) · Smart Learning Environments · doi:10.1186/s40561-025-00385-2 · Cited by 39 · see citing works
AI–AI bias: Large language models favor communications generated by large language models
Laurito, W., Davis, B., Grietzer, P., Gavenčiak, T., Böhm, A., & Kulveit, J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415697122 · Cited by 32 · see citing works
Modifying AI, Enhancing Essays: How Active Engagement with Generative AI Boosts Writing Quality
Yang, K., Raković, M., Liang, Z., Yan, L., Zeng, Z., Fan, Y., Gašević, D., & Chen, G. (2025) · Proceedings of the 15th International Learning Analytics and Knowledge Conference · doi:10.1145/3706468.3706544 · Cited by 26 · see citing works
The Impact of Generative AI on Creativity in Software Development: A Research Agenda
Jackson, V., Vasilescu, B., Russo, D., Ralph, P., Prikladnicki, R., Izadi, M., D’Angelo, S., Inman, S., Andrade, A., & van der Hoek, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708523 · Cited by 23 · see citing works
Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation
Chien, J., & Danks, D. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658946 · Cited by 22 · see citing works
“These definitions primarily focus on observing how one person acts toward another person or group, neglecting changes in internal cognitive states or reasoning, considerations of alternative possibilities, experiential fluctuations, self-identification, and similar psychological and social impacts.”
BackgroundPlaygrounds and Prejudices: Exploring Biases in Generative AI For Children.
Baines, A., Gruia, L., Collyer-Hoar, G., & Rubegni, E. (2024) · Proceedings of the 23rd Annual ACM Interaction Design and Children Conference · doi:10.1145/3628516.3659404 · Cited by 21 · see citing works
Ink and algorithm: Exploring temporal dynamics in generative <scp>AI</scp> ‐assisted writing
Yang, K., Cheng, Y., Zhao, L., Raković, M., Swiecki, Z., Gašević, D., & Chen, G. (2026) · British Journal of Educational Technology · doi:10.1111/bjet.70031 · Cited by 4 · see citing works
Understanding Gender Bias in AI-Generated Product Descriptions
Kelly, M., Tahaei, M., Smyth, P., & Wilcox, L. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732169 · Cited by 4 · see citing works
- Authors
- Kotek, H., Dockum, R., & Sun, D.
- Affiliations
- Apple (United States) (US)
- Swarthmore College (US)
- Venue
- Proceedings of The ACM Collective Intelligence Conference
- Publisher
- ACM
- Published
- 2023-11-05
- Pages
- 12-24
- OpenAlex
- W4386302153
- Cited by
- 381
- References
- 107
- Citations by year
- 2026: 115 · 2025: 162 · 2024: 99 · 2023: 4 · 2019: 1
- FWCI
- 95.12
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Collective Intelligence Conference (CI) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- gender bias, gender stereotypes, large language models, occupational stereotypes, bias amplification, explanation generation, imbalanced training data, reinforcement learning from human feedback, algorithmic fairness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
