What’s in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Background: Artificial intelligence chatbots such as ChatGPT (OpenAI) have garnered excitement about their potential for delegating writing tasks ordinarily performed by humans. Many of these tasks (eg, writing recommendation letters) have social and professional ramifications, making the potential social biases in ChatGPT’s underlying language model a serious concern. Objective: Three preregistered studies used the text analysis program Linguistic Inquiry and Word Count to investigate gender bias in recommendation letters written by ChatGPT in human-use sessions (N=1400 total letters). Methods: We conducted analyses using 22 existing Linguistic Inquiry and Word Count dictionaries, as well as 6 newly created dictionaries based on systematic reviews of gender bias in recommendation letters, to compare recommendation letters generated for the 200 most historically popular “male” and “female” names in the United States. Study 1 used 3 different letter-writing prompts intended to accentuate professional accomplishments associated with male stereotypes, female stereotypes, or neither. Study 2 examined whether lengthening each of the 3 prompts while holding the between-prompt word count constant modified the extent of bias. Study 3 examined the variability within letters generated for the same name and prompts. We hypothesized that when prompted with gender-stereotyped professional accomplishments, ChatGPT would evidence gender-based language differences replicating those found in systematic reviews of human-written recommendation letters (eg, more affiliative, social, and communal language for female names; more agentic and skill-based language for male names). Results: Significant differences in language between letters generated for female versus male names were observed across all prompts, including the prompt hypothesized to be neutral, and across nearly all language categories tested. Historically female names received significantly more social referents (5/6, 83% of prompts), communal or doubt-raising language (4/6, 67% of prompts), personal pronouns (4/6, 67% of prompts), and clout language (5/6, 83% of prompts). Contradicting the study hypotheses, some gender differences (eg, achievement language and agentic language) were significant in both the hypothesized and nonhypothesized directions, depending on the prompt. Heteroscedasticity between male and female names was observed in multiple linguistic categories, with greater variance for historically female names than for historically male names. Conclusions: ChatGPT reproduces many gender-based language biases that have been reliably identified in investigations of human-written reference letters, although these differences vary across prompts and language categories. Caution should be taken when using ChatGPT for tasks that have social consequences, such as reference letter writing. The methods developed in this study may be useful for ongoing bias testing among progressive generations of chatbots across a range of real-world scenarios. Trial Registration: OSF Registries osf.io/ztv96; https://osf.io/ztv96
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 1×
“This correction was selected because it limits the false discovery rate but is resilient to losses in power that accompany corrections such as the Bonferroni method [30].”
The Ambivalent Sexism Inventory: Differentiating hostile and benevolent sexism.
Glick, P., & Fiske, S. T. (1996) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.70.3.491 · Cited by 5,202 · cited 1×
“Benevolent sexism endorses the idea that women should be cherished and protected by more powerful men and rewards women for enacting traditional gender roles [31,32].”
Science faculty’s subtle gender biases favor male students
Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J., & Handelsman, J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1211286109 · Cited by 3,150 · cited 1×
“The factors contributing to gender disparities are multifaceted, but research indicates that bias in the hiring and selection process plays a contributing role [17-19].”
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
The Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · cited 1×
“…for unidentified social biases in ChatGPT’s algorithm is a particular concern, given that prior generations of natural language processing tools have frequently been found to contain gender and racial biases [4-6] because of the widespread presence of biased language in training data sets.”
A review of topic modeling methods
Vayansky, I., & Kumar, S. A. P. (2020) · Information Systems · doi:10.1016/j.is.2020.101582 · Cited by 716 · cited 1×
“Future research can use alternate approaches to language analyses (eg, topic modeling) [43] to add increased context to the effects observed here.”
Koch, A. J., D'Mello, S. D., & Sackett, P. R. (2015) · Journal of Applied Psychology · doi:10.1037/a0036734 · Cited by 583
Exploring the Color of Glass: Letters of Recommendation for Female and Male Medical Faculty
TRIX, F., & PSENKA, C. (2003) · Discourse & Society · doi:10.1177/0957926503014002277 · Cited by 553 · cited 4×
“…applicants often include more agentic and achievement-oriented language (eg, “accomplished” and “exceptional skills”) than those written for women, whereas female applicants receive more communal and affiliative language (eg, “compassionate,” “hardworking,” and “strong interpersonal skills”) [7,8].”
InfluentialPronoun Use Reflects Standings in Social Hierarchies
Kacewicz, E., Pennebaker, J. W., Davis, M., Jeon, M., & Graesser, A. C. (2014) · Journal of Language and Social Psychology · doi:10.1177/0261927x13502654 · Cited by 541 · cited 2×
“Standard LIWC variables selected for this study included 2 summary variables: analytic , which we anticipated to have correspondence with male skill-based stereotypes, and clout , a variable reflecting status and power [28].”
Progress toward gender equality in the United States has slowed or stalled
England, P., Levine, A., & Mishel, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1918891117 · Cited by 475 · cited 2×
“As noted earlier, despite a steady increase in the proportion of women in historically male-dominated professions [14], women remain underrepresented in top executive and leadership roles, particularly in medicine and other STEM fields.”
Eaton, A. A., Saunders, J. F., Jacobson, R. K., & West, K. (2020) · Sex Roles · doi:10.1007/s11199-019-01052-w · Cited by 391 · cited 1×
“The factors contributing to gender disparities are multifaceted, but research indicates that bias in the hiring and selection process plays a contributing role [17-19].”
Schmader, T., Whitehead, J., & Wysocki, V. H. (2007) · Sex Roles · doi:10.1007/s11199-007-9291-4 · Cited by 320 · cited 1×
“LIWC has been used to identify patterns of language bias in several prior investigations of recommendation letters written by humans, including recommendation letters written for medical residency applications [10,24], chemistry and biochemistry job applicants [25], and academic positions [12].”
Gender differences in recommendation letters for postdoctoral fellowships in geoscience
Dutt, K., Pfaff, D. L., Bernstein, A. F., Dillard, J. S., & Block, C. J. (2016) · Nature Geoscience · doi:10.1038/ngeo2819 · Cited by 232 · cited 1×
“…likely to contain references to drive [8,10], whereas letters written for women are more likely to contain references to personal life [7,11] and use “doubt-raisers” [12] or minimal assurances such as “She can do the job” rather than unequivocal endorsements (“She is the best for the job”) [9,13].”
Jordan, K. N., Sterling, J., Pennebaker, J. W., & Boyd, R. L. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1811987116 · Cited by 199 · cited 3×
“In addition to relative frequency variables such as this, LIWC also generates 4 summary variables [26] for each text file, two of which were selected for the reported work: analytic language , computed based on the formula [articles + prepositions – pronouns − auxiliary verbs – adverb –…”
Influence of Gender on Surgical Residency Applicants’ Recommendation Letters
Turrentine, F. E., Dreisbach, C. N., St Ivany, A. R., Hanks, J. B., & Schroen, A. T. (2019) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2018.12.020 · Cited by 198 · cited 2×
“Other studies have found that letters written for male applicants tend to be more likely to name specific accomplishments [8,9] and more likely to contain references to drive [8,10], whereas letters written for women are more likely to contain references to personal life [7,11] and use…”
Understanding Undesirable Word Embedding Associations
Ethayarajh, K., Duvenaud, D., & Hirst, G. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1166 · Cited by 160 · cited 1×
“…these tools in mitigating bias is commonly assessed through word vector associations tests that measure how closely associated specific words or phrases are with respect to sets of attribute words such as “male” versus “female” [35,36] (although other measures of association exist as well [37-39]).”
Raising Doubt in Letters of Recommendation for Academia: Gender Differences and Their Impact
Madera, J. M., Hebl, M. R., Dial, H. R., Martin, R. C., & Valian, V. V. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9541-1 · Cited by 149 · cited 3×
“We also included linguistic indicators of doubt-raising to supplement the “words to avoid” specially created category [12]: negations , tentative , and risk (the presence of which signals doubt), and emotion and positive emotion , the absence of which in a letter of recommendation may signal doubt.”
Gender and Racial Bias in Radiology Residency Letters of Recommendation
Grimm, L. J., Redmond, R. A., Campbell, J. C., & Rosette, A. S. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.08.008 · Cited by 132 · cited 1×
“…reviews of biased language in letter writing have predominantly focused on gender differences, it is important to note that studies have also identified biased language in letters written for Black, Indigenous, and people of color when compared with those written for White applicants [40,41].”
The Presence of Gender Bias in Letters of Recommendations Written for Urology Residency Applicants
Filippou, P. L., Mahajan, S., Deal, A. M., Wallen, E., Tan, H., Pruthi, R. S., & Smith, A. B. (2019) · Urology · doi:10.1016/j.urology.2019.05.065 · Cited by 123 · cited 2×
“…letters written for male applicants tend to be more likely to name specific accomplishments [8,9] and more likely to contain references to drive [8,10], whereas letters written for women are more likely to contain references to personal life [7,11] and use “doubt-raisers” [12] or minimal…”
The Cambridge Handbook of the Psychology of Prejudice
Guha, M. (2017) · Reference Reviews · doi:10.1108/rr-01-2017-0018 · Cited by 113
Gender bias in reference letters for residency and academic medicine: a systematic review
Khan, S., Kirubarajan, A., Shamsheri, T., Clayton, A., & Mehta, G. (2021) · Postgraduate Medical Journal · doi:10.1136/postgradmedj-2021-140045 · Cited by 92 · cited 4×
“…applicants often include more agentic and achievement-oriented language (eg, “accomplished” and “exceptional skills”) than those written for women, whereas female applicants receive more communal and affiliative language (eg, “compassionate,” “hardworking,” and “strong interpersonal skills”) [7,8].”
InfluentialWe look like our names: the manifestation of name stereotypes in facial appearance
Zwebner, Y., Sellier, A.-L., Rosenfeld, N., Goldenberg, J., & Mayo, R. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000076 · Cited by 60 · cited 1×
“For example, Zwebner et al [42] recently offered experimental evidence that appearance stereotypes associated with names are powerful enough to lead people to make choices about their facial appearances that correspond to social stereotypes of their names.”
Chaloner, K., & Maldonado, A. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3804 · Cited by 50 · cited 1×
“…these tools in mitigating bias is commonly assessed through word vector associations tests that measure how closely associated specific words or phrases are with respect to sets of attribute words such as “male” versus “female” [35,36] (although other measures of association exist as well [37-39]).”
Exploring the gender gap: Letters of recommendation to pediatric surgery fellowship.
Hoffman, A. L., Ghoubrial, R., McCormick, M., Matemavi, P., & Cusick, R. (2020) · American Journal of Surgery · doi:10.1016/j.amjsurg.2019.08.005 · Cited by 30 · cited 1×
“…and more likely to contain references to drive [8,10], whereas letters written for women are more likely to contain references to personal life [7,11] and use “doubt-raisers” [12] or minimal assurances such as “She can do the job” rather than unequivocal endorsements (“She is the best for the…”
Gender and Ethnic bias in Letters of Recommendation: Considerations for School Counselors
Akos, P. T., & Kretchmar, J. (2016) · Professional School Counseling · doi:10.5330/1096-2409-20.1.102 · Cited by 29 · cited 1×
“…reviews of biased language in letter writing have predominantly focused on gender differences, it is important to note that studies have also identified biased language in letters written for Black, Indigenous, and people of color when compared with those written for White applicants [40,41].”
Marwan, Y., Waly, F. J., Algarni, N., Addar, A., Saran, N., & Snell, L. S. (2017) · Journal of surgical education · doi:10.1016/j.jsurg.2017.01.006 · Cited by 25 · cited 1×
“Given the influential role that recommendation letters play in the selection and hiring process [44], gender bias in letters may negatively impact women’s advancement in medicine and other fields that emphasize agentic and leadership qualities, even if they possess these qualities to a similar…”
Gender and Racial Bias in Letters of Recommendation for Orthopedic Surgery Residency Positions
Girgis, M. Y., Qazi, S., Patel, A., Yu, D., Lu, X., & Sewards, J. M. (2022) · Journal of surgical education · doi:10.1016/j.jsurg.2022.08.021 · Cited by 24 · cited 1×
“LIWC has been used to identify patterns of language bias in several prior investigations of recommendation letters written by humans, including recommendation letters written for medical residency applications [10,24], chemistry and biochemistry job applicants [25], and academic positions [12].”
VAST: The Valence-Assessing Semantics Test for Contextualizing Language Models
Wolfe, R., & Caliskan, A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21400 · Cited by 12
Fine-Tuning Language Models to Mitigate Gender Bias in Sentence Encoders
Dolci, T. (2022) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice55688.2022.00036 · Cited by 11 · cited 1×
“There are numerous ongoing attempts to mitigate bias in LLMs, such as through fine-tuning sentence encoders on semantic similarity tasks [33] and the development of bias-sensitive tokens [34].”
Research works printed but not linked: 7
- Bolukbasi, T, arXiv
- BrownTBMannBRyderNSubbiahMKaplanJDhariwalPNeelakantanAShyamPSastryGAskellAAgarwalSHerbert-VossAKruegerGHenighanTChildRRameshAZieglerDMWuJWinterCHesseCChenMSiglerELitwinMGraySChessBClarkJBernerCMcCandlishSRadfordASutskeverIAmodeiDLanguage models are few-shot learnersProceedings of the 34th International Conference on Neural Information Processing Systems2020NIPS '20December 6-12, 2020Vancouver, BC1877901
- 10.1037/0022-3514.70.3.491 · doi:10.1037/0022-3514.70.3.491 · Paper page
- Liang, PP, arXiv
- Cer, D, arXiv
- Brunet, ME, arXiv
- Avoiding unintended gender bias in letters of recommendationNational Center for Women & Information Technology20102024-01-11https://tinyurl.com/2c8eh8v7
Cited by
Raman, R., Mandal, S., Das, P., Kaur, T., Sanjanasri, J. P., & Nedungadi, P. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/3085910 · Cited by 214
“Moreover, the identi fi ed gender bias in sentiment analysis underscores the importance of considering demographic factors [13, 89, 92] when designing and evaluating AI-based tools, thus informing more equitable technology implementation strategies.”
BackgroundSociodemographic biases in medical decision making by large language models
Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Apakama, D. U., Horowitz, C. R., Charney, A. W., Freeman, R., Kummer, B., Glicksberg, B. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03626-6 · Cited by 173
“Our findings align with earlier reports identifying gender biases in LLM-based clinical recommendations 20,35,36 .”
ChatGPT for Textual Analysis? How to Use Generative LLMs in Accounting Research
de Kok, T. (2025) · Management Science · doi:10.1287/mnsc.2023.03253 · Cited by 160
ChatGPT Promises and Challenges in Education: Computational and Ethical Perspectives
Adel, A., Ahsan, A., & Davison, C. (2024) · Education Sciences · doi:10.3390/educsci14080814 · Cited by 108
Omar, M., Sorin, V., Agbareia, R., Apakama, D. U., Soroush, A., Sakhuja, A., Freeman, R., Horowitz, C. R., Richardson, L. D., Nadkarni, G. N., et al. (2025) · International Journal for Equity in Health · doi:10.1186/s12939-025-02419-0 · Cited by 110
“Many studies found gender bias in tasks such as recommendation letter generation [36], medical case generation [35], and diagnostic reasoning [37].”
Metadata
- Authors
- Kaplan, D. M., Palitsky, R., Arconada Alvarez, S. J., Pozzo, N. S., Greenleaf, M. N., Atkinson, C. A., & Lam, W. A.
- Affiliations
- Emory University (US)
- University of Arizona (US)
- Georgia Institute of Technology (US)
- Corresponding author
- Kaplan, Deanna M
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-03-05
- Volume
- 26
- Pages
- e51837
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC10951834
- OpenAlex
- W4392456350
- Cited by
- 55
- References
- 29
- Citations by year
- 2026: 12 · 2025: 36 · 2024: 7
- FWCI
- 22.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institutes of Health (UL1TR002378)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Gender Studies › Gender Studies in Language
- Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
- Keywords
- gender bias, letters of recommendation, ChatGPT, AI-generated text, gender stereotypes, gendered language, LIWC, prompt effects, social language, personal pronouns, heteroscedasticity
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Female, Humans, Language, Linguistics, Male, Sexism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.