CoChat

What’s in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT

Open access
Kaplan, Deanna M, Palitsky, Roman, Arconada Alvarez, Santiago J, Pozzo, Nicole S, Greenleaf, Morgan N, Atkinson, Ciara A, Lam, Wilbur A · 2024 · Journal of Medical Internet Research

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.

Abstract

Background: Artificial intelligence chatbots such as ChatGPT (OpenAI) have garnered excitement about their potential for delegating writing tasks ordinarily performed by humans. Many of these tasks (eg, writing recommendation letters) have social and professional ramifications, making the potential social biases in ChatGPT’s underlying language model a serious concern. Objective: Three preregistered studies used the text analysis program Linguistic Inquiry and Word Count to investigate gender bias in recommendation letters written by ChatGPT in human-use sessions (N=1400 total letters). Methods: We conducted analyses using 22 existing Linguistic Inquiry and Word Count dictionaries, as well as 6 newly created dictionaries based on systematic reviews of gender bias in recommendation letters, to compare recommendation letters generated for the 200 most historically popular “male” and “female” names in the United States. Study 1 used 3 different letter-writing prompts intended to accentuate professional accomplishments associated with male stereotypes, female stereotypes, or neither. Study 2 examined whether lengthening each of the 3 prompts while holding the between-prompt word count constant modified the extent of bias. Study 3 examined the variability within letters generated for the same name and prompts. We hypothesized that when prompted with gender-stereotyped professional accomplishments, ChatGPT would evidence gender-based language differences replicating those found in systematic reviews of human-written recommendation letters (eg, more affiliative, social, and communal language for female names; more agentic and skill-based language for male names). Results: Significant differences in language between letters generated for female versus male names were observed across all prompts, including the prompt hypothesized to be neutral, and across nearly all language categories tested. Historically female names received significantly more social referents (5/6, 83% of prompts), communal or doubt-raising language (4/6, 67% of prompts), personal pronouns (4/6, 67% of prompts), and clout language (5/6, 83% of prompts). Contradicting the study hypotheses, some gender differences (eg, achievement language and agentic language) were significant in both the hypothesized and nonhypothesized directions, depending on the prompt. Heteroscedasticity between male and female names was observed in multiple linguistic categories, with greater variance for historically female names than for historically male names. Conclusions: ChatGPT reproduces many gender-based language biases that have been reliably identified in investigations of human-written reference letters, although these differences vary across prompts and language categories. Caution should be taken when using ChatGPT for tasks that have social consequences, such as reference letter writing. The methods developed in this study may be useful for ongoing bias testing among progressive generations of chatbots across a range of real-world scenarios. Trial Registration: OSF Registries osf.io/ztv96; https://osf.io/ztv96

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/51837

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Results of 2-tailed t tests comparing language categories arranged by hypothesized differences. H2 and H3: female>male refers to language categories that were expected to appear more in letters for female names (left side of the figure). H2 and H3: male>female refers to language categories that were expected to appear more in letters for male names (right side of the figure). *P<.05; whisker bars designate SE. ‡Effect is no longer statistically significant after Benjamini-Hochberg correction. Each bar chart in the figure represents a separate 2-tailed t test. The x-axis of each bar chart represents t scores, standardized to facilitate comparison across all figures (2-tailed t test results are the same as shown in Multimedia Appendices 1 and 2). Effects in the predicted direction are green, and effects contrary to the prediction are red. Color saturation indicates effect size using Cohen d standards for strong (>0.6), medium (0.3-0.6), and weak (<0.3) effects.
Figure 1 Results of 2-tailed t tests comparing language categories arranged by hypothesized differences. H2 and H3: female>male refers to language categories that were expected to appear more in letters for female names (left side of the figure). H2 and H3: male>female refers to language categories that were expected to appear more in letters for male names (right side of the figure). *P<.05; whisker bars designate SE. ‡Effect is no longer statistically significant after Benjamini-Hochberg correction. Each bar chart in the figure represents a separate 2-tailed t test. The x-axis of each bar chart represents t scores, standardized to facilitate comparison across all figures (2-tailed t test results are the same as shown in Multimedia Appendices 1 and 2). Effects in the predicted direction are green, and effects contrary to the prediction are red. Color saturation indicates effect size using Cohen d standards for strong (>0.6), medium (0.3-0.6), and weak (<0.3) effects.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 7

  1. Bolukbasi, T, arXiv
  2. BrownTBMannBRyderNSubbiahMKaplanJDhariwalPNeelakantanAShyamPSastryGAskellAAgarwalSHerbert-VossAKruegerGHenighanTChildRRameshAZieglerDMWuJWinterCHesseCChenMSiglerELitwinMGraySChessBClarkJBernerCMcCandlishSRadfordASutskeverIAmodeiDLanguage models are few-shot learnersProceedings of the 34th International Conference on Neural Information Processing Systems2020NIPS '20December 6-12, 2020Vancouver, BC1877901
  3. 10.1037/0022-3514.70.3.491 · doi:10.1037/0022-3514.70.3.491 · Paper page
  4. Liang, PP, arXiv
  5. Cer, D, arXiv
  6. Brunet, ME, arXiv
  7. Avoiding unintended gender bias in letters of recommendationNational Center for Women & Information Technology20102024-01-11https://tinyurl.com/2c8eh8v7

Cited by

See all 55

Metadata

Authors
Kaplan, D. M., Palitsky, R., Arconada Alvarez, S. J., Pozzo, N. S., Greenleaf, M. N., Atkinson, C. A., & Lam, W. A.
Affiliations
  • Emory University (US)
  • University of Arizona (US)
  • Georgia Institute of Technology (US)
Corresponding author
Kaplan, Deanna M
Venue
Journal of Medical Internet Research
Published
2024-03-05
Volume
26
Pages
e51837
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4392456350
Cited by
55
References
29
Citations by year
2026: 12 · 2025: 36 · 2024: 7
FWCI
22.71
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • National Institutes of Health (UL1TR002378)
  • National Center for Advancing Translational Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Gender Studies › Gender Studies in Language
  • Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
Keywords
gender bias, letters of recommendation, ChatGPT, AI-generated text, gender stereotypes, gendered language, LIWC, prompt effects, social language, personal pronouns, heteroscedasticity
MeSH terms
Systematic Reviews as Topic, Artificial Intelligence, Female, Humans, Language, Linguistics, Male, Sexism

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.