CoChat
Sign Up
Cited by 59Ethics and Social Impacts of AIAI and HR TechnologiesNames, Identity, and Discrimination Research

Measuring gender and racial biases in large language models: Intersectional evidence from automated resume evaluation

Open access
An, Jiafu, Huang, Difang, Lin, Chen, Tai, Mingzhu·2025·PNAS Nexus
DOI 10.1093/pnasnexus/pgaf089PMC PMC11937954
Open this paper in CoChat — free accountOpen PDF

Abstract

In traditional decision-making processes, social biases of human decision makers can lead to unequal economic outcomes for underrepresented social groups, such as women and racial/ethnic minorities (1–4). Recently, the growing popularity of large language model (LLM)-based AI signals a potential shift from human to AI-based decision-making. How would this transition affect the distributional outcomes across social groups? Here, we investigate the gender and racial biases of a number of commonly used LLMs, including OpenAI's GPT-3.5 Turbo and GPT-4o, Google's Gemini 1.5 Flash, Anthropic AI's Claude 3.5 Sonnet, and Meta's Llama 3-70b, in a high-stakes decision-making setting of assessing entry-level job candidates from diverse social groups. Instructing the models to score ∼361,000 resumes with randomized social identities, we find that the LLMs award higher assessment scores for female candidates with similar work experience, education, and skills, but lower scores for black male candidates with comparable qualifications. These biases may result in ∼1–3 percentage-point differences in hiring probabilities for otherwise similar candidates at a certain threshold and are consistent across various job positions and subsamples. Meanwhile, many models are biased against black male candidates. Our results indicate that LLM-based AI systems demonstrate significant biases, varying in terms of the directions and magnitudes across different social groups. Further research is needed to comprehend the root causes of these outcomes and develop strategies to minimize the remaining biases in AI systems. As AI-based decision-making tools are increasingly employed across diverse domains, our findings underscore the necessity of understanding and addressing the potential unequal outcomes to ensure equitable outcomes across social groups.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/pnasnexus/pgaf089

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In LLM research, bias direction varies by model and task, with some studies finding pro-Black and others anti-Black patterns depending on the context.

    “In LLM research, bias direction varies by model and task, with some studies finding pro-Black and others anti-Black patterns depending on the context [3, 29, 46].”

    From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [3]

  • LLM research supports intersectionality: resume screening found simultaneous pro-female and anti-Black male bias, resume retrieval disadvantaged Black men specifically, and LLM bias is intertwined across race, gender, and SES.

    “LLM research supports this: resume screening found simultaneous pro-female and anti-Black male bias [3], resume retrieval disadvantaged Black men specifically [94], and LLM bias is intertwined across race, gender, and SES [80].”

    From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [3]

  • Findings on gender bias direction are mixed.

    “Findings on direction are mixed: An et al. [3] found consistent pro-female bias, while others find anti-female patterns depending on task [29].”

    From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [3]

  • The consistent gender null contrasts with prior work detecting gender effects in resume screening, clinical contexts, and deliberate attempts to elicit biased outputs.

    “This consistent null contrasts with prior work detecting gender effects in resume screening[3], clinical contexts [70], and deliberate attempts to elicit biased outputs [54].”

    From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [3]

  • Neither race nor gender showed consistent effects, a result that diverges from the existing literature's primary focus on these two dimensions.

    “Neither race nor gender showed consistent effects, a result that diverges from the existing literature's primary focus on these two dimensions [3, 29, 94].”

    From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [3]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

3 of the 58 references couldn’t be matched to a known paper.

Research works printed but not linked: 24

  1. [5]Becker GS. 1957. The economics of discrimination. Chicago: University of Chicago Press.
  2. [11]Bender EM, Gebru T, McMillan-Major A, Shmitchell S. 2021. On the dangers of stochastic parrots: can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. New York (NY): Association for Computing Machinery. p. 610–623.
  3. [12]Crawford K. 2021. The atlas of AI: power, politics, and the planetary costs of artificial intelligence. New Haven (CT): Yale University Press.
  4. [21]Deshpande KV, Pan S, Foulds JR. 2020. Mitigating demographic bias in AI-based resume filtering. In Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization. New York: Association for Computing Machinery. p. 268–275.
  5. [23]Hanna A, Denton E, Smart A, Smith-Loud J. 2020. Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. New York: Association of Computing Machinery. p. 501–512.
  6. [24]Lee MH, Montgomery JM, Lai CK. 2024. Large language models portray socially subordinate groups as more homogeneous, consistent with a bias observed in humans. In The 2024 ACM Conference on Fairness, Accountability, and Transparency. New York: Association of Computing Machinery. p. 1321–1340.
  7. [25]Kotek H, Dockum R, Sun D. 2023. Gender bias and stereotypes in large language models. In Proceedings of the ACM Collective Intelligence Conference. New York: Association of Computing Machinery. p. 12–24.
  8. [28]Glaese A, et al 2022. Improving alignment of dialog agents via targeted human judgments, arXiv, arXiv:2209.14375, preprint: not peer reviewed. · arXiv
  9. [29]Bai Y, et al 2022. Constitutional AI: harmlessness from AI feedback, arXiv, arXiv:2212.08073, preprint: not peer reviewed. · arXiv
  10. [38]Crenshaw K. 2013. Demarginalizing the intersection of race and sex: a black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. In: Feminist Legal Theories. London: Routledge. p. 23– 51.
  11. [43]Sidanius J, Pratto F. 1999. Social dominance theory. In: Handbook of theories of social psychology. New York: SAGE Publications Ltd. p. 418– 438.
  12. [49]Glazko K, Mohammed Y, Kosa B, Potluri V, Mankoff J. 2024. Identifying and improving disability bias in GPT-based resume screening. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. New York: Association of Computing Machinery. p. 687–700.
  13. [51]Gerszberg NR. 2024. Quantifying gender bias in large language models: when ChatGPT becomes a hiring manager [PhD thesis]. Massachusetts Institute of Technology.
  14. [53]Morehouse K, Pan W, Contreras JM, Banaji MR. 2024. Bias transmission in large language models: evidence from gender-occupation bias in GPT-4. ICML 2024 Next Generation of AI Safety Workshop.
  15. [54]Veldanda AK, et al 2023. Investigating hiring bias in Large Language Models. R0–FoMo: Workshop on Robustness of Few-shot and Zero-shot Learning in Large Foundation Models at NeurIPS 2023.
  16. [56]Guo W, Caliskan A. 2021. Detecting emergent intersectional biases: contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. New York: Association of Computing Machinery. p. 122–133.
  17. [59]Ma W, Chiang B, Wu T, Wang L, Vosoughi S. 2023. Intersectional stereotypes in large language models: dataset and analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023. USA: Association for Computational Linguistics. p. 8589–8597.
  18. [60]Omrani Sabbaghi S, Wolfe R, Caliskan A. 2023. Evaluating biased attitude associations of language models in an intersectional context. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. New York: Association of Computing Machinery. p. 542–553.
  19. [61]Wilson K, Caliskan A. 2024. Gender, race, and intersectional bias in resume screening via language model retrieval. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. New York: Association of Computing Machinery. p. 1578–1590.
  20. [62]Armstrong L, Liu A, MacNeil S, Metaxa D. 2024. The silicon ceiling: auditing GPT’s Race and gender biases in hiring. In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. New York: Association of Computing Machinery. p. 1–18.
  21. [71]Friedman BM, Woodford M, editors. 2010. Handbook of monetary economics. Elsevier.
  22. [73]Atari M, Xue MJ, Park PS, Blasi D, Henrich J. 2024. Which humans? PsyarXiv https://osf.io/preprints/psyarxiv/5b26t, preprint: not peer reviewed.
  23. [74]Zewail A, Figueroa A, Graham J, Atari M. 2024. Moral stereotyping in large language. Psyarxiv. https://osf.io/preprints/psyarxiv/t9×8r, preprint: not peer reviewed. PMC12974530 41779787
  24. [77]Santurkar S, Durmus E, Ladhak F, Lee C, Liang P. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning. New York: Association of Computing Machinery. p. 29971–30004.

Cited by

Sorted by citations

Related

Authors
An, J., Huang, D., Lin, C., & Tai, M.
Affiliations
  • University of Hong Kong (HK)
  • Chinese Academy of Sciences (CN)
Corresponding author
  • Huang, Difang
  • Lin, Chen
  • Tai, Mingzhu
ORCID
Venue
PNAS Nexus
Publisher
Oxford University Press (OUP)
Published
2025-02-27
Volume
4
Issue
3
Pages
pgaf089-pgaf089
ISSN
2752-6542 (ISSN-L)
DOI
10.1093/pnasnexus/pgaf089
PMC
PMC11937954
OpenAlex
W4408357561
Cited by
58
References
58
Citations by year
2026: 41 · 2025: 17
FWCI
60.02
Type
journal article
Language
en
Open access
Licence
Locations
  • PNAS Nexus (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Funders
  • National Natural Science Foundation of China (72193284021, T2293771)
  • University of Hong Kong
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
  • Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
Keywords
large language models, algorithmic bias, gender bias, racial bias, intersectionality, hiring decisions, AI decision-making, algorithmic fairness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review