Word embeddings quantify 100 years of gender and ethnic stereotypes
Open accessThe DOI matches its Crossref record; 64 of the 69 references it cites were found in a registry or an index.
Abstract
Significance: Word embeddings are a popular machine-learning method that represents each English word by a vector, such that the geometry between these vectors captures semantic relations between the corresponding words. We demonstrate that word embeddings can be used as a powerful tool to quantify historical trends and social change. As specific applications, we develop metrics based on word embeddings to characterize how gender stereotypes and attitudes toward ethnic minorities in the United States evolved during the 20th and 21st centuries starting from 1910. Our framework opens up a fruitful intersection between machine learning and quantitative social science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs trained to predict the next word in a sentence surprised their creators not only by their inclination to be racist and sexist (34) but also by their emergent reasoning and arithmetic skills (35), ability to translate between languages (22), and propensity to semantic priming (36).”
From Evaluating large language models in theory of mind tasks (2024) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, as LLMs are trained from curated datasets and fine-tuned using human feedback, quantifying the biases that are introduced and/or mitigated by these models is crucial 179,180 .”
From Machine culture (2023) · cites this paper as [180]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI outputs tend to reproduce normative views and mirror societal biases for marginalized groups [15, 21, 36, 41, 42, 45, 53, 66, 72, 91], including PwD [10, 12, 39, 48, 54, 62, 77, 113].”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [41]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“dense vector representations called word embeddings86 are able to capture semantic relationships between words, such as sex, gender and ethnic relationships87, thus absorbing biases existing in the training corpus88.”
From Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare (2020) · cites this paper as [87]
The cited source says what the claim attributes to it.
In the citing paper
“Our findings also align with previous research on biases in word embeddings trained on internet text20,64,65.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [65]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“As additional validation, we train, using the GLoVe algorithm (27), embeddings from the New York Times Annotated Corpus (28) for every year between 1988 and 2005.”
MethodEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“, 2014), and train embeddings over three year windows14. 6.2 Related Works Word embedding was developed as a framework to represent words as a part of the AI and natural language processing pipeline (Mikolov et al., 2013b). (Bolukbasi et al., 2016) demonstrated that word embeddings capture gender stereotypes, and (Cal…”
BackgroundMethodInfluentialAdvances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Fiske, S. T., Cuddy, A. J. C., Glick, P., & Xu, J. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.82.6.878 · Cited by 7,364
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 3×
“lso capture common stereotypes, as these stereotypes are likely to be present, even if subtly, in the large corpora of training texts (Bolukbasi et al., 2016; 1 arXiv:1711.08412v1 [cs.CL] 22 Nov 2017 Caliskan, Bryson, and Narayanan, 2017; Zhao et al., 2017; van Miltenburg, 2016). For example, the vector for adjective…”
BackgroundResultInfluentialMapping the Language of Racism: Discourse and the Legitimation of Exploitation
Wetherell, M. S., & Potter, J. (1993) · Open Research Online (The Open University) · Cited by 1,747
Integrated Public Use Microdata Series
Ruggles, S., Genadek, K., Goeken, R., Grover, J., & Sobek, M. (2015) · Institute for Social Research and Data Innovation · doi:10.18128/d010.v6.0 · Cited by 1,485
Racial stereotypes of one hundred college students.
Katz, D., & Braly, K. (1933) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0074049 · Cited by 1,476
Research works printed but not linked: 1
- A Kim, CJ Yeh Stereotypes of Asian American students (ERIC Digest New York, NY, 2002).
Cited by
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463
D'Ignazio, C., & Klein, L. F. (2020) · doi:10.7551/mitpress/11805.001.0001 · Cited by 1,738
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
“Several works have presented word lists associated with social groups for gender (Bolukbasi et al., 2016; Garg et al., 2018; Gupta et al., 2022; Hall Maudslay et al., 2019; Lu et al., 2020; Zhao et al., 2017; 2018), race (Caliskan et al.”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
“We also discarded all papers using “bias” in NLP systems to measure social “bias” in text or the real world (e.g., Garg et al., 2018).”
MethodCan Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
“, 2018), with applications to public opinion (Bhatia, 2017; Garg et al., 2018; Kozlowski et al., 2019; Abul-Fottouh and Fetner, 2018), voting behavior (Black et al.”
Background
Metadata
- Authors
- Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J.
- Affiliations
- Stanford University (US)
- Chan Zuckerberg Biohub San Francisco (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2018-04-03
- Volume
- 115
- Issue
- 16
- Pages
- E3635-E3644
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5910851
- OpenAlex
- W2769358515
- Cited by
- 1,065
- References
- 69
- Citations by year
- 2026: 98 · 2025: 106 · 2024: 148 · 2023: 139 · 2022: 130 · 2021: 190 · 2020: 134 · 2019: 101 · 2018: 19
- FWCI
- 222.54
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-114747, CRII-1657155)
- Chan-Zuckerberg Biohub (-)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- word embeddings, gender stereotypes, ethnic stereotypes, attitudes toward women, ethnic minorities, women's movement, Asian immigration, career change, demographic change, US Census data
- MeSH terms
- Machine Learning, Culture, Ethnicity, Female, Humans, Language, Male, Minority Groups, Newspapers as Topic, Occupations, Religion, Social Change, Stereotyping, United States, Internet, History, 20th Century, History, 21st Century, Racism, Sexism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Word embeddings quantify 100 years of gender and ethnic stereotypes. https://app.cochat.ai/papers/10.1073/pnas.1720347115
- The paper
- Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1720347115
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.