Semantics derived automatically from language corpora contain human-like biases
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Machines learn what people know implicitly: AlphaGo has demonstrated that a machine can learn how to do things that people spend many years of concentrated study learning, and it can rapidly learn how to do them better than any human can. Caliskan et al. now show that machines can learn word associations from written texts and that these associations mirror those learned by humans, as measured by the Implicit Association Test (IAT) (see the Perspective by Greenwald). Why does this matter? Because the IAT has predictive value in uncovering the association between concepts, such as pleasantness and flowers or unpleasantness and insects. It can also tease out attitudes and beliefs—for example, associations between female names and family or male names and career. Such biases may not be expressed explicitly, yet they can prove influential in behavior. Science , this issue p. 183 ; see also p. 133
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Training deep learning models on biased data can amplify human biases, replicate toxic language, or perpetuate stereotypes of gender, sexual orientation, political leaning, or religion (e.g., Caliskan et al. 2017; Hartmann et al. 2023b).”
From Generative AI (2024) · cites this paper as Caliskan et al. 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some studies (Caliskan et al. 2017;Greenwald et al. 1998) have found that names associated with European Americans are more likely to be linked with pleasantness, while non-European names tend to be associated with unpleasantness.”
From Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models (2024) · cites this paper as Caliskan et al. 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning models reproduce various content biases inherent to their training data, including gender bias, racial or ethnic bias, bias for negative and threat-related information, and socio-economic bias 69,[86][87][88][89][90] .”
From Machine culture (2023) · cites this paper as [87]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“If people type “He is a doctor. She is a nurse” into Google Translate and translate the phrases into Turkish, Google Translate provides: “O bir hemşire. O bir doktor”. Those Turkish sentences are gender-neutral; Turkish does not differentiate between the words “he” and “she”. When translating the Turkish text into English again, Google Translate provides: “She is a nurse. He is a doctor”.”
From Discrimination, artificial intelligence, and algorithmic decision-making (2025) · cites this paper as Caliskan, Bryson and Narayanan 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“That most models produced overgeneralizations reliably across time suggests that they may have incorporated an ‘algorithmic bias’ from the training data similar to the way models trained on linguistic data can acquire gender, racial, or political biases, by picking up human tendencies implicit in training text corpora [51–53].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [51]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 2×
“For example, we repeated all the WEAT and WEFAT experiments presented above using a different pre-trained embedding: word2vec on a Google News corpus (Mikolov and Dean, 2013).”
BackgroundMethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“For example, we repeated all the WEAT and WEFAT experiments presented above using a different pre-trained embedding: word2vec on a Google News corpus (Mikolov and Dean, 2013).”
BackgroundMethodPattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Measuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 2×
“Besides, bias is known to creep in indirectly, by proxy (Barocas and Selbst, 2014).”
BackgroundFrom Frequency to Meaning: Vector Space Models of Semantics
Turney, P. D., & Pantel, P. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2934 · Cited by 2,866
Research works printed but not linked: 1
- M. Stubbs Text and Corpus Analysis: Computer-Assisted Studies of Language and Culture (Blackwell Oxford 1996).
Cited by
Items 101–200 of 291
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
Wang, Z., Qinami, K., Karakozis, I. C., Genova, K., Nair, P., Hata, K., & Russakovsky, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00894 · Cited by 274
To trust or not to trust? An assessment of trust in AI-based systems: Concerns, ethics and contexts
Omrani, N., Rivieccio, G., Fiore, U., Schiavone, F., & Agreda, S. G. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.121763 · Cited by 269
Ferrara, E. (2023) · doi:10.2139/ssrn.4614223 · Cited by 266
Governing Artificial Intelligence to benefit the UN Sustainable Development Goals
Truby, J. (2020) · Sustainable Development · doi:10.1002/sd.2048 · Cited by 266
I'd blush if I could: closing gender divides in digital skills through education
UNESCO, & Coalition, E. S. (2019) · doi:10.54675/rapc9356 · Cited by 262
Transparent to whom? No algorithmic accountability without a critical audience
Kemper, J., & Kolkman, D. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2018.1477967 · Cited by 260
“…can learn to adopt similar discriminatory attitudes based on words associated with a particular gender or social group (Barocas & Selbst, 2016; Caliskan, Bryson, & Narayanan, 2017) ‒ similarly, algorithmic models have been demonstrated to potentially reproduce racialized and sex-ualized…”
BackgroundDetecting racial bias in algorithms and machine learning
Turner Lee, N. (2018) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-06-2018-0056 · Cited by 260
Ferrara, E. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00250-1 · Cited by 259
Onan, A. (2019) · IEEE Access · doi:10.1109/access.2019.2945911 · Cited by 257
Deep learning-based sentiment classification of evaluative text based on Multi-feature fusion
Abdi, A., Shamsuddin, S. M., Hasan, S., & Piran, J. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.02.018 · Cited by 256
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges
Ras, G., van Gerven, M., & Haselager, P. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_2 · Cited by 255
Manzini, T., Yao Chong, L., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1062 · Cited by 255
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265
“Prior work on the societal risks of large language models has warned that models learn stereotypes and biases from their training data [14, 28, 44] that may be amplified through widespread deployments [19].”
BackgroundSmart Earth: A meta-review and implications for environmental governance
Bakker, K., & Ritts, M. (2018) · Global Environmental Change · doi:10.1016/j.gloenvcha.2018.07.011 · Cited by 244
“As many scholars have pointed out, AI and machine-learning driven approaches often contain implicit bias; incomplete datasets and fl awed algorithms can prove counterproductive and ecologically damaging (Caliskan et al., 2017; Galaz and Mouazen, 2017).”
Algorithmic bias in machine learning-based marketing models
Akter, S., Dwivedi, Y. K., Sajib, S., Biswas, K., Bandara, R. J., & Michael, K. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.083 · Cited by 242
Predictive biases in natural language processing models: A conceptual framework and overview
Shah, D. S., Schwartz, H. A., & Hovy, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.468 · Cited by 241
“However, many times these representations contain unintended or undesirable associations and societal stereotypes (e.g., connecting medical doctors more frequently to male pronouns than female pronouns) (Boluk-basi et al., 2016; Caliskan et al., 2017).”
BackgroundMethodAI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227
“…2018; Devlin et al., 2019) pretrained in a self-supervised fashion exhibit all kinds of unfair and discriminative biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Zhao et al., 2017; Rudinger et al., 2018; Gonen and Goldberg, 2019; Bordia and Bowman, 2019; Sheng et al., 2019; Basta et al.,…”
BackgroundMind the GAP: A balanced corpus of gendered ambiguous pronouns
Webster, K., Recasens, M., Axelrod, V., & Baldridge, J. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00240 · Cited by 223
Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis
Jia, X., Lynch, A., Huang, Y., Danielson, M., Lang’at, I., Milder, A., Ruby, A. E., Wang, H., Friedler, S. A., Norquist, A. J., et al. (2019) · Nature · doi:10.1038/s41586-019-1540-5 · Cited by 222
Speeding up to keep up: exploring the use of AI in the research process
Chubb, J., Cowling, P., & Reed, D. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01259-0 · Cited by 218
Identifying and Reducing Gender Bias in Word-Level Language Models
Bordia, S., & Bowman, S. R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop · doi:10.18653/v1/n19-3002 · Cited by 219
“Caliskan et al. (2017) conduct Word Embedding Association Test (WEAT).”
Cultural Schemas: What They Are, How to Find Them, and What to Do Once You’ve Caught One
Boutyline, A., & Soter, L. K. (2021) · American Sociological Review · doi:10.1177/00031224211024525 · Cited by 216
“It can also be measured by the Implicit Association Test (IAT), which captures the relative ease of keeping A and B simultaneously activated in mind.”
BackgroundInfluentialAuditing Partisan Audience Bias within Google Search
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274417 · Cited by 209
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
Uncovering and Mitigating Algorithmic Bias through Learned Latent Structure
Amini, A., Soleimany, A. P., Schwarting, W., Bhatia, S. N., & Rus, D. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314243 · Cited by 206
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219
Measuring and Reducing Gendered Correlations in Pre-trained Models
Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E., & Petrov, S. (2020) · arXiv · doi:10.48550/arxiv.2010.06032 · Cited by 195
Ash, E., & Hansen, S. (2023) · Annual Review of Economics · doi:10.1146/annurev-economics-082222-074352 · Cited by 197
Ethical and legal challenges of AI in marketing: an exploration of solutions
Kumar, D., & Suthar, N. (2024) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-05-2023-0068 · Cited by 180
Ethics and diversity in artificial intelligence policies, strategies and initiatives
Roche, C., Wall, P. J., & Lewis, D. (2023) · AI and Ethics · doi:10.1007/s43681-022-00218-9 · Cited by 179
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2304.05335 · Cited by 178
Ellahham, S., Ellahham, N., & Simsekler, M. C. E. (2020) · American Journal of Medical Quality · doi:10.1177/1062860619878515 · Cited by 177
“Human-like biases are a known and recognized component of automated systems.(51) These unintentional biases can compromise the safety of AI systems.”
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176
AI chatbots not yet ready for clinical use
Yeung, J. A., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · doi:10.1101/2023.03.02.23286705 · Cited by 176
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186
Lessons learned from AI ethics principles for future actions
Hickok, M. (2021) · AI and Ethics · doi:10.1007/s43681-020-00008-1 · Cited by 168
Addressing algorithmic bias and the perpetuation of health inequities: An AI bias aware framework
Agarwal, R., Bjarnadottir, M., Rhue, L., Dugas, M., Crowley, K., Clark, J., & Gao, G. (2023) · Health Policy and Technology · doi:10.1016/j.hlpt.2022.100702 · Cited by 170
Gmail Smart Compose: Real-Time Assisted Writing
Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330723 · Cited by 169
Artificial Intelligence in mental health and the biases of language based models
Straw, I., & Callison-Burch, C. (2020) · PLOS ONE · doi:10.1371/journal.pone.0240376 · Cited by 164
Key Considerations for Incorporating Conversational AI in Psychotherapy
Miner, A. S., Shah, N., Bullock, K. D., Arnow, B. A., Bailenson, J., & Hancock, J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00746 · Cited by 166
The emergence of economic rationality of GPT
Chen, Y., Liu, T. X., Shan, Y., & Zhong, S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2316205120 · Cited by 164
“This may be due to the reflection of biases presented in the existing data (Caliskan et al., 2017; Schramowski et al., 2022),the insufficient training of texts of the alternative contexts and frames (Chen et al.”
BackgroundInvestigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155
"I wouldn't say offensive but...": Disability-Centered Perspectives on Large Language Models
Gadiraju, V., Kane, S., Dev, S., Taylor, A., Wang, D., Denton, R., & Brewer, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3593989 · Cited by 155
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155
Toxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178
Discrimination, artificial intelligence, and algorithmic decision-making
Borgesius, F. Z. (2025) · arXiv · doi:10.48550/arxiv.2510.13465 · Cited by 152
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149
Argyle, L. P., Bail, C. A., Busby, E. C., Gubler, J. R., Howe, T., Rytting, C., Sorensen, T., & Wingate, D. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311627120 · Cited by 144
The cyclical ethical effects of using artificial intelligence in education
Dieterle, E., Dede, C., & Walker, M. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01497-w · Cited by 142
“…that education is not alone in its susceptibility to bias; examples of algorithmic bias also associated with life outcomes abound in the criminal justice system (Angwin et al., 2016; Završnik 2019), social services (Eubanks 2017), and job recruitment (Caliskan et al. 2017; Raghavan et al. 2020).”
(2020) · neural information processing systems · Cited by 142
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
Using natural language processing to analyse text data in behavioural science
Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, C. E., Rathje, S., Hartmann, J., Mohammad, S. M., et al. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-024-00392-z · Cited by 140
Addressing age-related bias in sentiment analysis,
Diaz, M., Johnson, I., Lazar, A., Piper, A. M., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173986 · Cited by 138
Large language models show human-like content biases in transmission chain experiments
Acerbi, A., & Stubbersfield, J. M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2313790120 · Cited by 137
Tenney, I., Wexler, J., Bastings, J., Bolukbasi, T., Coenen, A., Gehrmann, S., Jiang, E., Pushkarna, M., Radebaugh, C., Reif, E., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.15 · Cited by 138
“, 2019), but many questions remain about the behavior of these models under domain shift (Blitzer and Pereira, 2007) and adversarial settings (Jia and Liang, 2017), and for their tendencies to behave according to social biases (Bolukbasi et al., 2016; Caliskan et al., 2017) or shallow heuristics (e.”
BackgroundMethodBuilding and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundBrinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J., Maini, V., Yogatama, D., & Kohli, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.7 · Cited by 124
The growing ubiquity of algorithms in society: implications, impacts and innovations
Olhede, S. C., & Wolfe, P. J. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0364 · Cited by 120
Barikeri, S., Lauscher, A., Vulić, I., & Glavaš, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.151 · Cited by 121
“Caliskan et al. (2017) presented the Word Embedding Association Test (WEAT), quantifying the bias between two sets of target terms towards two sets of attribute terms.”
BackgroundMethodInfluentialThe life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119
“The Word Embedding Association Test (WEAT) Caliskan et al. (2017) is a commonly used embedding-based metric that quantifies biases in word embedding by examining how closely words related to certain concepts are associated with words related to social groups or attributes.”
MethodGeneralization bias in large language model summarization of scientific research
Peters, U., & Chin-Yee, B. (2025) · Royal Society Open Science · doi:10.1098/rsos.241776 · Cited by 126
“…reliably across time suggests that they may have incorporated an ‘algorithmic bias’ from the training data similar to the way models trained on linguistic data can acquire gender, racial, or political biases, by picking up human tendencies implicit in training text corpora [51–53].”
Perturbation Sensitivity Analysis to Detect Unintended Model Biases
Prabhakaran, V., Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1578 · Cited by 115
“this work study bias by swapping names in sentence templates (Caliskan et al., 2017; Kiritchenko and Mohammad, 2018; May et al., 2019; Gonen and Goldberg, 2019); however they use synthetic sentence templates, while we extract naturally oc-”
BackgroundMethodGarrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., & Ureña-López, L. A. (2021) · Applied Sciences · doi:10.3390/app11073184 · Cited by 123
(2018) · arXiv (Cornell University) · Cited by 113
Investigating Gender Bias in BERT
Bhardwaj, R., Majumder, N., & Poria, S. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09881-2 · Cited by 112
A Timely Intervention: Tracking the Changing Meanings of Political Concepts with Word Vectors
Rodman, E. (2020) · Political Analysis · doi:10.1017/pan.2019.23 · Cited by 112
“Other word vectorization analyses have revealed that corpora reproduce gendered or racial biases and stereotypes (Bolukbasi et al. 2016; Caliskan, Bryson, and Narayanan 2017; Garg et al. 2018).”
BackgroundSchultebraucks, K., Yadav, V., Shalev, A. Y., Bonanno, G. A., & Galatzer-Levy, I. R. (2022) · Psychological Medicine · doi:10.1017/s0033291720002718 · Cited by 108
“While we used state-of-the-art methods, there are known limitations and risk of bias that need to be pointed out with regard to facial expression recognition (Buolamwini & Gebru, 2018; Klare, Burge, Klontz, Bruegge, & Jain, 2012) and NLP (Caliskan, Bryson, & Narayanan, 2017).”
MethodInfluentialThe Biometric Assemblage: Surveillance, Experimentation, Profit, and the Measuring of Refugee Bodies
Madianou, M. (2019) · Television & New Media · doi:10.1177/1527476419857682 · Cited by 107
Algorithmic inclusion: Shaping the predictive algorithms of artificial intelligence in hiring
Kelan, E. K. (2024) · Human Resource Management Journal · doi:10.1111/1748-8583.12511 · Cited by 106
Equity in essence: a call for operationalising fairness in machine learning for healthcare
Wawira Gichoya, J., McCoy, L. G., Celi, L. A., & Ghassemi, M. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100289 · Cited by 104
Educational data augmentation in physics education research using ChatGPT
Kieser, F., Wulff, P., Kuhn, J., & Küchemann, S. (2023) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.19.020150 · Cited by 108
“It was furthermore established that LLM such as GPT 3 and others incorporate human-like biases, e.g., with regards to gender [24, 52].”
BackgroundShah, H. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0362 · Cited by 101
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97
Cognitive imperialism in artificial intelligence: counteracting bias with indigenous epistemologies
Ofosu-Asare, Y. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02065-0 · Cited by 97
How Society Can Maintain Human-Centric Artificial Intelligence
Bryson, J. J., & Theodorou, A. (2019) · Translational Systems Sciences · doi:10.1007/978-981-13-7725-9_16 · Cited by 94
Self-citationFairness in Precision Medicine
Ferryman, K. (2018) · doi:10.69985/tbql1779 · Cited by 94
HONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95
Generative language models exhibit social identity biases
Hu, T., Kyrychenko, Y., Rathje, S., Collier, N., van der Linden, S., & Roozenbeek, J. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00741-1 · Cited by 100
“Our findings are thus similar to previous research on biases in older language models [16, 57, 58], which found that word embeddings trained on”
Improving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87
“Despite the growing capability of NLP models, many state-of-the-art systems still encode serious biases and harms [13] and fail basic logical tests [52] that developers have been aware of for years, but have not been able to fix.”
BackgroundUnmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias
Bartl, M., Nissim, M., & Gatt, A. (2020) · arXiv · doi:10.48550/arxiv.2010.14534 · Cited by 84
“(2019) adapt the Word Embedding Association Test (WEAT; Caliskan et al., 2017) to pooled sentence representations, resulting in the SEAT (Sentence Encoder Association Test).”
BackgroundMethodInfluentialDesigning AI for Social Good: Seven Essential Factors
Cowls, J., King, T., Taddeo, M., & Floridi, L. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3388669 · Cited by 83
Zheng, M., Pei, J., Logeswaran, L., Lee, M., & Jurgens, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.888 · Cited by 82
Abramski, K., Citraro, S., Lombardi, L., Rossetti, G., & Stella, M. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030124 · Cited by 81
“One approach involves using Association Tests [13,32–34], which may be performed at different levels of discourse.”
BackgroundMethodAbundant intelligences: placing AI within Indigenous knowledge frameworks
Lewis, J. E., Whaanga, H., & Yolgörmez, C. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02099-4 · Cited by 80
Performance and biases of large language models in public opinion simulation.
Qu, Y., & Wang, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03609-x · Cited by 79
The Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
Benefits and Harms of Large Language Models in Digital Mental Health
De Choudhury, M., Pendse, S. R., & Kumar, N. (2023) · doi:10.31234/osf.io/y8ax9 · Cited by 78
Artificial intelligence and the public arena
Jungherr, A., & Schroeder, R. (2023) · Communication Theory · doi:10.1093/ct/qtad006 · Cited by 78
Benefits and Harms of Large Language Models in Digital Mental Health
De Choudhury, M., Pendse, S. R., & Kumar, N. (2023) · arXiv · doi:10.48550/arxiv.2311.14693 · Cited by 78
Galderisi, S., Appelbaum, P. S., Gill, N., Gooding, P., Herrman, H., Melillo, A., Myrick, K., Pathare, S., Savage, M., Szmukler, G., et al. (2024) · World Psychiatry · doi:10.1002/wps.21230 · Cited by 76
“We must understand how AI makes decisions in various critical settings 70 .”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76
“Stereotypes, Biases, & Toxicity NLP models exhibit a wide range of social and cultural biases (Caliskan et al., 2017; Bolukbasi et al., 2016; Pen-nington et al., 2014).”
BackgroundNew Information Technology and Implicit Bias
Elsbach, K. D., & Stigliani, I. (2019) · Academy of Management Perspectives · doi:10.5465/amp.2017.0079 · Cited by 74
Czarnowska, P., Vyas, Y., & Shah, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00425 · Cited by 72
“A related line of research measures bias present in sentence or word representations (Bolukbasi et al., 2016; Caliskan et al., 2017; Kurita et al., 2019; Sedoc and Ungar, 2019; Chaloner and Maldonado, 2019; Dev and Phillips, 2019; Gonen and Goldberg, 2019; Hall Maudslay et al., 2019; Liang et al., 2020; Shin et al., 2…”
BackgroundKizhner, I., Terras, M., Rumyantsev, M., Khokhlova, V., Demeshkova, E., Rudov, I., & Afanasieva, J. (2021) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaa055 · Cited by 83
Embedding Regression: Models for Context-Specific Description and Inference
RODRIGUEZ, P. L., SPIRLING, A., & STEWART, B. M. (2023) · American Political Science Review · doi:10.1017/s0003055422001228 · Cited by 106
“For example, Caliskan, Bryson, and Narayanan (2017) and Garg et al. (2018) have explored relationships between words captured by embeddings to describe problematic gender and ethnic stereotypes in society at large.”
BackgroundWhat’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A., & Viviano, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.24 · Cited by 69
“Consequently, much work has also been dedicated to creating standardized diagnostic tests to detect these biases (Caliskan et al., 2017; May et al., 2019; Nadeem et al., 2020; Sweeney and Najafian, 2019) and to remove them (Bolukbasi et al.”
Background
Metadata
- Authors
- Caliskan, A., Bryson, J. J., & Narayanan, A.
- Affiliations
- Princeton University (US)
- Corresponding author
- Caliskan, Aylin
- Bryson, Joanna J.
- Narayanan, Arvind
- Venue
- Science
- Published
- 2017-04-14
- Volume
- 356
- Issue
- 6334
- Pages
- 183-186
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2893425640
- Cited by
- 2,887
- References
- 51
- Citations by year
- 2026: 299 · 2025: 368 · 2024: 343 · 2023: 411 · 2022: 326 · 2021: 399 · 2020: 326 · 2019: 224 · 2018: 134 · 2017: 47
- FWCI
- 459.41
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- The University of Bath Online Publications Store (The University of Bath) (submittedVersion)
- Pure (University of Bath) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Social and Intergroup Psychology
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- semantic bias, machine learning, text corpora, Implicit Association Test, algorithmic bias, natural language processing, gender bias, racial bias, cultural bias, bias detection
- MeSH terms
- Machine Learning, Association, Female, Humans, Language, Male, Names, Semantics, Sex Factors, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.