Persistent Anti-Muslim Bias in Large Language Models
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
It has been observed that large-scale language models capture undesirable societal biases, e.g. relating to race and gender; yet religious bias has been relatively unexplored. We demonstrate that GPT-3, a state-of-the-art contextual language model, captures persistent Muslim-violence bias. We probe GPT-3 in various ways, including prompt completion, analogical reasoning, and story generation, to understand this anti-Muslim bias, demonstrating that it appears consistently and creatively in different uses of the model and that it is severe even compared to biases about other religious groups. For instance, Muslim is analogized to terrorist in 23% of test cases, while Jewish is mapped to its most common stereotype, money, in 5% of test cases. We quantify the positive distraction needed to overcome this bias with adversarial text prompts, and find that use of the most positive 6 adjectives reduces violent completions for Muslims from 66% to 20%, but which is still higher than for other religious groups.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Pre-trained on massive amounts of text on the Internet, LMs are known to produce linguistically fawed, factually incorrect, or even ethically problematic text at times [1,24].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs and other AI tools may have a number of biases that could arise from various factors, such as unbalanced training datasets74–77.”
From On opportunities and challenges of large multimodal foundation models in education (2025) · cites this paper as [75]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“it is also a liability; all AI systems might inherit the same problematic biases of a few foundation models [Bolukbasi et al. 2016; Caliskan et al. 2017; Abid et al. 2021, inter alia]”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Abid et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Previous work has shown that language models tend to exhibit human-like biases with respect to specific protected groups such as gender, ethnicity or religious orientation3–6.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [4]
In the citing paper
“If a medical LLM has been trained on data that contains misinformation or biased content, LLMs deployed for clinical decision-making are at risk of reproducing these problematic contents [120,124–126].”
From Bias in medical AI: Implications for clinical decision-making (2024) · cites this paper as [126]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundMethodInfluentialSelf-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.131 · Cited by 2,112 · cited 1×
“These descriptions are generated by passing images through a pretrained image captioning model [11], and fine-tuning GPT-3 to transform the short captions into long descriptions, as shown in Fig.”
MethodReformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · cited 1×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundSemi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2020) · arXiv · doi:10.48550/arxiv.2004.09456 · Cited by 643 · cited 2×
“As a result, even though the various language models have different architectures, since they are trained on similar corpora of text, they often learn similar biases [9].”
BackgroundUniversal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642
Identifying and Reducing Gender Bias in Word-Level Language Models
Bordia, S., & Bowman, S. R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop · doi:10.18653/v1/n19-3002 · Cited by 219 · cited 1×
“Previous work has explored the biases that are ultimately learned by language models, with the primary focus being racial and gender stereotypes [2, 3, 7, 12].”
BackgroundThe Woman Worked as a Babysitter: On Biases in Language GenerationPreprint
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01326 · Cited by 205 · cited 2×
“In [12], the authors made similar observations about language generation models.”
Background
Research works printed but not linked: 6
- 7. T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, A. T. Kalai, Advances in neural information processing systems 29 , 4349 (2016).
- 9. S. Bordia, S. R. Bowman, arXiv preprint arXiv:1904.03035 (2019). · arXiv
- 2. A. M. Dai, Q. V. Le, Advances in neural information processing systems 28 , 3079 (2015).
- 10. K. Lu, P. Mardziel, F. Wu, P. Amancharla, A. Datta, Logic, Language, and Security (Springer, 2020), pp. 189-202.
- 3. A. Vaswani, et al. , Advances in neural information processing systems (2017), pp. 59986008.
- 14. E. Wallace, S. Feng, N. Kandpal, M. Gardner, S. Singh, arXiv preprint arXiv:1908.07125 (2019). · arXiv
Cited by
Items 101–200 of 986
Garrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., & Ureña-López, L. A. (2021) · Applied Sciences · doi:10.3390/app11073184 · Cited by 123
Generative AI and the politics of visibility
Gillespie, T. (2024) · Big Data & Society · doi:10.1177/20539517241252131 · Cited by 122
“…2018; Keyes 2018; Scheuerman et al. 2020) and image classi fi cation (Katzman et al. 2021), and are now asking similar questions about generative AI (Abid et al. 2021; Bender et al. 2021; Gautam et al. 2024; Ghosh & Caliskan 2021; Luccioni et al. 2023; Qadri et al. 2023; Solaiman et al., 2023;…”
BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M., & Kochenderfer, M. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-0685 · Cited by 121
The limitations and ethical considerations of ChatGPT
Hua, S., Jin, S., & Jiang, S. (2024) · Data Intelligence · doi:10.1162/dint_a_00243 · Cited by 120
“Abid et al. [58] used a complementation experiment to find that Muslims are more likely to be described as “terrorists” by GPT-3 than Christians and Buddhists.”
BackgroundLarge Language Model Safety: A Holistic SurveyPreprint
Shi, D., Shen, T., Huang, Y., Li, Z., Leng, Y., Jin, R., Liu, C., Wu, X., Guo, Z., Yu, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.17686 · Cited by 120
Salinas, A., Shah, P., Huang, Y., McCormack, R., & Morstatter, F. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623257 · Cited by 118
“Studies have also explored biases related to religion [1], finding that GPT-3 generations with the word “Muslim” led to more violence-related responses compared to other religious identities.”
BackgroundDemystifying ChatGPT: An In-depth Survey of OpenAI’s Robust Large Language Models
Bhattacharya, P., Prasad, V. K., Verma, A., Gupta, D. K., Sapsomboon, A., Viriyasitavat, W., & Dhiman, G. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10115-5 · Cited by 118
Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues
Chang, Z., Li, M., Liu, Y., Wang, J., Wang, Q., & Liu, Y. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.304 · Cited by 117
The Silicon Ceiling: Auditing GPT’s Race and Gender Biases in Hiring
Armstrong, L., Liu, A., MacNeil, S., & Metaxa, D. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694699 · Cited by 124
“…task, and with prior work on bias in LLMs which has shown that they learn biases that mirror humans’ implicit biases [19], exhibit racial and gender bias [1, 32, 50], have occupational biases [49], and make up rationalizations for bias [50], the need to reconsider their use in hiring is clear.”
InfluentialEnvisioning Information Access Systems: What Makes for Good Tools and a Healthy Web?
Shah, C., & Bender, E. M. (2024) · ACM Transactions on the Web · doi:10.1145/3649468 · Cited by 66
“And LLM-based generative systems could make things even worse: it is well established that LLMs absorb and amplify biases around gender, race, and other sensitive attributes [1, 16–18, 53, 78].”
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2023) · arXiv · doi:10.48550/arxiv.2304.00228 · Cited by 115
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2024) · 2024 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw63631.2024.00018 · Cited by 114
“Our work furthers the study of harms and risks of LLMs (Bender et al., 2021; Bommasani et al., 2021; Liang et al., 2022; Abid et al., 2021; Gehman et al., 2020; Ganguli et al., 2022; Weidinger et al., 2021; 2022), focusing on demonstrating their potential of being actively misused.”
BackgroundExploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2302.05733 · Cited by 114
“Our work furthers the study of harms and risks of LLMs (Bender et al., 2021; Bommasani et al., 2021; Liang et al., 2022; Abid et al., 2021; Gehman et al., 2020; Ganguli et al., 2022; Weidinger et al., 2021; 2022), focusing on demonstrating their potential of being actively misused.”
BackgroundROBBIE: Robust Bias Evaluation of Large Generative Language Models
Esiobu, D., Tan, X., Hosseini, S., Ung, M., Zhang, Y., Fernandes, J., Dwivedi-Yu, J., Presani, E., Williams, A., & Smith, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.230 · Cited by 115
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110
“Base prompts (Table C1) are comprised of an initial instruction , replicating common conditions for most language model applications.”
Islamic Ethics and AI: An Evaluation of Existing Approaches to AI using Trusteeship Ethics
Ali, F., Bouzoubaa, K., Gelli, F., Hamzi, B., & Khan, S. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00922-4 · Cited by 108
Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on GraphsPreprint
Jin, B., Xie, C., Zhang, J., Roy, K. K., Zhang, Y., Li, Z., Li, R., Tang, X., Wang, S., Meng, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.07103 · Cited by 109
DecodingTrust: A comprehensive assessment of trustworthiness in GPT models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1361 · Cited by 107
Bias and Fairness in Chatbots: An Overview
Xue, J., Wang, Y.-C., Wei, C., Liu, X., Woo, J., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000064 · Cited by 112
Emergence of Social Norms in Generative Agent Societies: Principles and Architecture
Ren, S., Cui, Z., Song, R., Wang, Z., & Hu, S. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/874 · Cited by 109
“…least, we would like to remark that although the study of normative, generative MASs offers exciting prospects, it is crucial to remain aware of its potential negative aspects, especially since recent studies have shown that LLMs may exhibit biases and generate toxic content [Abid et al. , 2021].”
BackgroundRe-contextualizing Fairness in NLP: The Case of India
Bhatt, S., Dev, S., Talukdar, P. P., Dave, S., & Prabhakaran, V. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.55 · Cited by 107
Vision Language Models are Biased
Vo, A., Nguyen, K.-N., Taesiri, M. R., Dang, V. T., Nguyen, A. T., & Kim, D. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2505.23941 · Cited by 107
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Lee, M. H. J., Montgomery, J., & Lai, C. K. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658975 · Cited by 106
“…Bolukbasi et al. [6] and Caliskan et al. [11] documenting human-like biases within word embedding models, a wide array of studies have found biases within models for coreference resolution [49], text classification [15], machine translation [38, 46], and text generation [1, 34], among many others.”
BackgroundEconomies of Virtue: The Circulation of ‘Ethics’ in Big Tech
Phan, T., Goldenfein, J., Mann, M., & Kuch, D. (2021) · Science as Culture · doi:10.1080/09505431.2021.1990875 · Cited by 103
“From algorithmic systems that deliver biased and inequitable outcomes (Sweeney, 2013; Buolamwini and Gebru, 2018; Noble, 2018; Abid et al. , 2021; Phan and Wark, 2021) to toxic workplace cultures that exploit workers and discriminate along racist, sexist, and ableist lines (Ho ff man, 2020).”
BackgroundLarge Language Models for Text Classification: From Zero-Shot Learning to Instruction-Tuning
Chae, Y., & Davidson, T. (2025) · Sociological Methods & Research · doi:10.1177/00491241251325243 · Cited by 103
Generative language models exhibit social identity biases
Hu, T., Kyrychenko, Y., Rathje, S., Collier, N., van der Linden, S., & Roozenbeek, J. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00741-1 · Cited by 100
“Prior work has shown that language models tend to exhibit human-like biases with respect to protected groups such as gender, ethnicity, or religious orientation [2, 3, 4].”
Foundation Model Transparency Reports
Bommasani, R., Klyman, K., Longpre, S., Xiong, B., Kapoor, S., Maslej, N., Narayanan, A., & Liang, P. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31628 · Cited by 101
GPT3Mix: Leveraging Large-scale Language Models for Text AugmentationPreprint
Yoo, K. M., Park, D., Kang, J., Lee, S.-W., & Park, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08826 · Cited by 105
“Pre-trained language models that are trained on untreated corpora are known to exhibit social biases (Bordia and Bowman, 2019; Hutchinson et al., 2020; Abid et al., 2021; Bender et al., 2021) and toxicity (Gehman et al., 2020).”
BackgroundDifferentially Private Language Models for Secure Data Sharing
Mattern, J., Jin, Z., Weggenmann, B., Schölkopf, B., & Sachan, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.323 · Cited by 100
Extending the WILDS Benchmark for Unsupervised AdaptationPreprint
Sagawa, S., Koh, P. W., Lee, T., Gao, I., Xie, S. M., Shen, K., Kumar, A., Hu, W., Yasunaga, M., Marklund, H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.05090 · Cited by 99
“Previous work suggests that training on large amounts of data alone is not sufficient to avoid unwanted biases, since many papers have pointed out biases in large language models (Abid et al., 2021; Nadeem et al., 2020; Gehman et al., 2020).”
BackgroundHaving Beer after Prayer? Measuring Cultural Bias in Large Language Models
Naous, T., Ryan, M. J., Ritter, A., & Xu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.862 · Cited by 96
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10903-2 · Cited by 94
“LLMs show the same biases that occur in the datasets, ranging from religious (Abid et al. 2021) to gender discrimination (Lucy and Bamman 2021).”
BackgroundAI, data governance and privacy: Synergies and areas of international co-operation
OECD (2024) · OECD artificial intelligence papers · doi:10.1787/2476b1a4-en · Cited by 95
What Should Data Science Education Do With Large Language Models?
Tu, X., Zou, J., Su, W., & Zhang, L. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.bff007ab · Cited by 96
Self-citationComputer says ‘no’: Exploring systemic bias in ChatGPT using an audit approach
Lippens, L. (2024) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2024.100054 · Cited by 51
“The GPT-3 model instance has demonstrated anti-Muslim bias in word association tasks before, consistently linking Muslims with violence and terrorism (Abid et al., 2021).”
BackgroundMethodHow Far Are We From AGI: Are LLMs All We Need?
Feng, T., Jin, C., Liu, J., Zhu, K., Tu, H., Cheng, Z., Lin, G., & You, J. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2405.10313 · Cited by 95
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Solaiman, I., & Dennison, C. (2021) · arXiv · doi:10.48550/arxiv.2106.10328 · Cited by 90
“[4] Earlier analyses of harmful outputs in GPT-3 show negative race, gender[8], and religious[3] associations in generated text.”
BackgroundKirk, H. R., Bean, A., Vidgen, B., Röttger, P., & Hale, S. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.148 · Cited by 91
Revisiting Automated Topic Model Evaluation with Large Language Models
Stammbach, D., Zouhar, V., Hoyle, A., Sachan, M., & Ash, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.581 · Cited by 91
Mayer, C. W. F., Ludwig, S., & Brandt, S. (2023) · Journal of Research on Technology in Education · doi:10.1080/15391523.2022.2142872 · Cited by 57
“Several publications show that, for example, GPT-3 tends to discriminate against religious groups (Abid et al., 2021) or gender (Lucy & Bamman, 2021).”
BackgroundPatton: Language Model Pretraining on Text-Rich Networks
Jin, B., Zhang, W., Zhang, Y., Yu, M., Zhang, X., Zhu, Q., & Han, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.387 · Cited by 88
Larooij, M., & Törnberg, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11412-6 · Cited by 90
“This issue is closely tied to the widely debated problem of bias in LLMs (Ferrara 2023; Abid et al. 2021; Kotek et al. 2023; Ghosh and Caliskan 2023).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models
Rauh, M., Mellor, J., Uesato, J., Huang, P.-S., Welbl, J., Weidinger, L., Dathathri, S., Glaese, A., Irving, G., Gabriel, I., et al. (2022) · neural information processing systems · doi:10.52202/068431-1793 · Cited by 14
Prompt architecture induces methodological artifacts in large language models
Brucks, M. S., & Toubia, O. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319159 · Cited by 91
“For example, due to co-occurrences in their corpora, LLMs may reveal gender stereotypes, such as perceiving a “nurse” more as a “woman” and a “doctor” more as a “man” [12–14].”
Evaluating Gender Bias in Large Language Models via Chain-of-Thought PromptingPreprint
Kaneko, M., Bollegala, D., Okazaki, N., & Baldwin, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.15585 · Cited by 86
“Prior work have identified different types of social biases such as racial, religious etc. in addition to gender bias in pre-trained language models (Abid et al., 2021; Viswanath and Zhang, 2023).”
BackgroundCognitive bias in generative AI influences religious education
Zhang, J., Song, W., & Liu, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-99121-6 · Cited by 85
“Several studies have confirmed that its output exhibits systemic cultural skew, such as the 1.7 times higher semantic association between Islam and “violence” compared to other religions 46 .”
InfluentialSimulacrum of Stories: Examining Large Language Models as Qualitative Research Participants
Kapania, S., Agnew, W. S., Eslami, M., Heidari, H., & Fox, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713220 · Cited by 86
“Extensive prior research has shown how LLMs reflect biases towards gender [20], race [80], religion [2], nationality [108], and in social settings [72].”
Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks
Xing, W., Li, M., Li, M., & Han, M. (2026) · ACM Computing Surveys · doi:10.1145/3806048 · Cited by 86
“In LLMs , this may manifest as unfair task prioritization during human-robot interaction, favoring requests based on perceived demographic attributes [1, 106].”
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 85
“In NLP, this includes representational harms (like misrepresentation [266], stereotyping [4], disparate system performance [33, 369], derogatory language [29], and exclusionary norms [21 ]) and allocational harms (such as direct and indirect dis - crimination [78 ]).”
An Overview of Language Models: Recent Developments and Outlook
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000010 · Cited by 86
“It was demonstrated in [123] that GPT-3 [87] consistently exhibits a significant anti-Muslim bias in various tasks.”
BackgroundIn-Contextual Gender Bias Suppression for Large Language Models
Oba, D., Kaneko, M., & Bollegala, D. (2024) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2024.findings-eacl.121 · Cited by 82
“Prior work have identified different types of social biases such as racial, religious etc. in addition to gender bias in pre-trained language models (Abid et al., 2021; Kaneko and Bollegala, 2022; Viswanath and Zhang, 2023).”
BackgroundAbramski, K., Citraro, S., Lombardi, L., Rossetti, G., & Stella, M. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030124 · Cited by 81
“Persistent anti-Muslim bias has been detected by probing GPT-3 in various ways, including prompt completion, analogical reasoning, and story generation [35].”
BackgroundAccuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2025) · Proceedings of the 17th ACM Web Science Conference 2025 · doi:10.1145/3717867.3717903 · Cited by 81
“Recent studies show that LLMs can perpetuate discrimination due to their inherent biases (Bai et al. 2024; Salinas et al. 2023) and stereotypes (Abid, Farooqi, and Zou 2021; Cheng, Durmus, and Jurafsky 2023).”
Assessing and alleviating state anxiety in large language models
Ben‐Zion, Z., Witte, K., Jagadish, A. K., Duek, O. A., Harpaz‐Rotem, I., Khorsandian, M.-C., Burrer, A., Seifritz, E., Homan, P., Schulz, E., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01512-6 · Cited by 83
“Indeed, prior studies have extensively documented biases in LLMs related to gender 26 – 29 , race 30,31 , religion 30,32 , nationality 33 , disability 34,35 , occupation 36 and sexual orientation 37 .”
Mitigating Propensity Bias of Large Language Models for Recommender Systems
Zhang, G., Yuan, G., Cheng, D., Liu, L., Li, J., & Zhang, S. (2025) · ACM Transactions on Information Systems · doi:10.1145/3736404 · Cited by 82
“For example, Abid et al. [1] found that LLMs commonly associated Muslims with violence.”
Knowledge Editing for Large Language Models: A Survey
Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., & Li, J. (2025) · ACM Computing Surveys · doi:10.1145/3698590 · Cited by 79
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., & Li, H. (2023) · arXiv · doi:10.48550/arxiv.2308.05374 · Cited by 79
Rismani, S., Shelby, R., Smart, A., Jatho, E., Kroll, J., Moon, A., & Rostamzadeh, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 33
“The development and use of ML systems can adversely impact people, communities, and society at large [14, 32, 82, 89, 118, 125], including inequitable resource allocation [6, 22, 106], perpetuating normative narratives about people and social groups [56, 119], and the entrenchment of social inequalities [4, 70, 76].”
BackgroundWang, H., & Shu, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.09433 · Cited by 80
Cultural Bias in Large Language Models: A Comprehensive Analysis and Mitigation Strategies
Liu, Z. (2025) · Journal of Transcultural Communication · doi:10.1515/jtc-2023-0019 · Cited by 77
Improving Activation Steering in Language Models with Mean-CentringPreprint
Jorgensen, O., Cope, D., Schoots, N., & Shanahan, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.03813 · Cited by 77
Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and BardPreprint
Borji, A., & Mohammadian, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4476855 · Cited by 76
IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context
Sahoo, N., Kulkarni, P., Ahmad, A., Goyal, T., Asad, N., Garimella, A., & Bhattacharyya, P. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.487 · Cited by 74
The future of standardised assessment: Validity and trust in algorithms for assessment and scoring
Aloisi, C. (2023) · European Journal of Education · doi:10.1111/ejed.12542 · Cited by 73
Efficient utilization of pre-trained models: A review of sentiment analysis via prompt learning
Bu, K., Liu, Y., & Ju, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111148 · Cited by 73
Machine translation and its evaluation: a study
Mondal, S. K., Zhang, H., Kabir, H. M. D., Ni, K., & Dai, H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10423-5 · Cited by 72
“Notably, we can minimize the bias by introducing words or phrases into the context that provide strong positive associations within (Abid et al. 2021; Weidinger et al. 2021).”
BackgroundLarge Language Models for Text Classification: From Zero-Shot Learning to Instruction-TuningPreprint
Chae, Y., & Davidson, T. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/sthwk · Cited by 76
Nofal, A., Ali, H. M., Hadi, M., Ahmad, A., Qayyum, A., Johri, A., Al‐Fuqaha, A., & Qadir, J. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100347 · Cited by 70
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A., & Viviano, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.24 · Cited by 69
“…larger corpora generally result in better models (Kaplan et al., 2020; Sun et al., 2017), data quality and corpora content also plays a major role in the caliber and appropriateness of these models for the various downstream applications (Florez, 2019; Abid et al., 2021; Bhardwaj et al., 2021).”
BackgroundStart Generating: Harnessing Generative Artificial Intelligence for Sociological Research
Davidson, T. (2024) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231241259651 · Cited by 76
“For example, when prompted with content related to Muslims, GPT-3 generated responses related to terrorism (Abid, Farooqi, and Zou 2021).”
On Hate Scaling Laws For Data-SwampsPreprint
Birhane, A., Prabhu, V. U., Han, S., & Boddeti, V. N. (2023) · arXiv.org · doi:10.48550/arxiv.2306.13141 · Cited by 48
“Similarly, [1] studied the outputs generated from GPT-3 when the word “Muslim” is included in the prompt.”
BackgroundSociodemographic Bias in Language Models: A Survey and Forward Path
Gupta, V. K., Venkit, P. N., Wilson, S., & Passonneau, R. J. (2024) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2024.gebnlp-1.19 · Cited by 67
“…2020; Tan and Celis, 2019); ethnic bias, when models are partial towards certain ethnicity (Ahn and Oh, 2021; Garg et al., 2018; Li et al., 2020; Abid et al., 2021; Manzini et al., 2019; Narayanan Venkit et al., 2023); age bias (Nangia et al., 2020; Diaz et al., 2018), sexual-orientation bias…”
BackgroundLarge Language Models in Legal Systems: A Survey
Dehghani, F., Dehghani, R., Ardebili, Y. N., & Rahnamayan, S. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05924-3 · Cited by 66
“This study emphasizes the challenges in addressing deep-seated biases in language models and the necessity for continuous monitoring and adjustment (Abid et al., 2021).”
Jackson, H. (2023) · Media War & Conflict · doi:10.1177/17506352231178148 · Cited by 65
“Studies have found persistent sexist, racist, and Islamophobic bias in highly-used natural language processing toolkits (Abid et al., 2021; Bolukbasi et al., 2016; Bordia and Bowman, 2019; Lu et al., 2020; Nadeem et al., 2020; Shearer et al., 2019; Sheng et al., 2019).”
BackgroundVLStereoSet: A Study of Stereotypical Bias in Pre-trained Vision-Language Models
Zhou, K., Lai, E., & Jiang, J. B. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.40 · Cited by 65
“…example, in NLP, people studied social biases in word embeddings (e.g., Bolukbasi et al., 2016, Zhao et al., 2018b) and language models (e.g., Nadeem et al., 2021,Abid et al., 2021), and in computer vision, people studied social biases in unsupervised vision models (e.g., Steed and Caliskan, 2021).”
BackgroundPosition:TrustLLM: Trustworthiness in Large Language Models
Huang, Y., Sun, L., & Vanschoren, J. (2024) · TU/e Research Portal · Cited by 66
Cognitive bias in decision-making with LLMs
Echterhoff, J. M., Liu, Y., Alessa, A., McAuley, J., & He, Z. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.739 · Cited by 62
“…(Itzhak et al., 2023) have been detected in LLMs (e.g. gender bias (Kotek et al., 2023; Vig et al., 2020; Zhao et al., 2018), religious bias (Abid et al., 2021), stereotype bias (Nadeem et al., 2020), occupational bias (Kirk et al., 2021), sentiment bias (Huang et al., 2019) or bias against…”
BackgroundPlaza-del-Arco, F. M., Curry, A. C., Paoli, S., Curry, A. C., & Hovy, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.251 · Cited by 62
Choudhary, T. (2024) · IEEE Access · doi:10.1109/access.2024.3523764 · Cited by 62
“For instance, Abid et al. [50] established that GPT-3 had serious biases against some religious and ethnic groups due to the data on which this model was trained.”
Mass-Producing Failures of Multimodal Systems with Language Models
Tong, S., Jones, E., & Steinhardt, J. (2023) · neural information processing systems · doi:10.52202/075280-1275 · Cited by 60
An Evaluation of Cultural Value Alignment in LLMPreprint
Sukiennik, N., Chen, G., Xu, F., & Li, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.08863 · Cited by 60
The ROOTS Search Tool: Data Transparency for LLMs
Piktus, A., Akiki, C., Villegas, P., Laurençon, H., Dupont, G., Luccioni, A. S., Jernite, Y., & Rogers, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.29 · Cited by 60
Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection
Zhao, Y., Wang, B., Wang, Y., Zhao, D., He, R., & Hou, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1 · Cited by 60
Demonstrations of the Potential of AI-based Political Issue Polling
Sanders, N. E., Ulinich, A., & Schneier, B. (2023) · Harvard Data Science Review · doi:10.1162/99608f92.1d3cf75d · Cited by 58
Künstliche Intelligenz im DaF-Unterricht? Disruptive Technologien als Herausforderung und Chance
Hartmann, D. (2021) · Informationen Deutsch als Fremdsprache · doi:10.1515/infodaf-2021-0078 · Cited by 57
Mei, K., Fereidooni, S., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594109 · Cited by 56
“Despite enormous efforts in bias evaluation of LMs, prior work extensively focuses on biases related to gender, race, and ethnicity [1, 5, 22, 24, 50, 52].”
BackgroundHandbook of Computational Social Science for Policy
Bertoni, E., Fontana, M., Gabrielli, L., Signorelli, S., & Vespe, M. (2023) · Springer eBooks · doi:10.1007/978-3-031-16624-2 · Cited by 57
Towards Culturally Adaptive Large Language Models in Mental Health: Using ChatGPT as a Case Study
Aleem, M., Zahoor, I., & Naseem, M. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3681858 · Cited by 59
“This includes religious bias, where a 2021 study demonstrated persistent anti-Muslim bias in GPT-3, with 23% of test cases identifying Muslims as terrorists, compared to 5% for other religious groups [3].”
AI safety landscape for large language models: taxonomy, state-of-the-art, and future directions
Chen, C., Gong, X., Liu, Z., Jiang, W. M. W. L. H., Goh, S. Q., & Lam, K. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11590-x · Cited by 56
Ghosh, S., & Caliskan, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10510 · Cited by 55
Veldanda, A. K., Grob, F., Thakur, S., Pearce, H. A., Tan, B., Karri, R., & Garg, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05135 · Cited by 55
“Prior studies in natural language processing identified gender biases Bolukbasi et al. (2016); Nangia et al. (2020); Vig et al. (2020), religious bias Abid et al. (2021) and ethnic bias Ahn & Oh (2021).”
Knowledge of cultural moral norms in large language models
Ramezani, A., & Xu, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.26 · Cited by 54
Editor's Introduction: Humanities in the Loop
Goodlad, L. M. E. (2023) · Critical AI · doi:10.1215/2834703x-10734016 · Cited by 55
Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes
Gallegos, I. O., Aponte, R., Rossi, R. A., Barrow, J., Tanjim, M., Yu, T., Deilamsalehy, H., Zhang, R., Kim, S., Dernoncourt, F., et al. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-short.74 · Cited by 54
Source framing triggers systematic bias in large language models
Germani, F., & Spitale, G. (2025) · Science Advances · doi:10.1126/sciadv.adz2924 · Cited by 54
“Our findings reflect broader concerns in the literature about the ethical risks of LLMs reinforcing harmful or ex-clusionary narratives ( 30–34 ).”
Chen, C., Wu, Z., Lai, Y., Ou, W., Liao, T., & Zheng, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.00419 · Cited by 54
“These biases involve various themes, such as religious stereotypes, gender bias, racism, etc. Abid et al. [5] demonstrated through experiments that the advanced language model GPT-3 has a sustained violent bias against Muslim communities.”
BackgroundOnline Harassment in Majority Contexts: Examining Harms and Remedies across Countries
Schoenebeck, S. Y., Batool, A., Do, G., Darling, S., Grill, G., Wilkinson, D., Khan, M., Toyama, K., & Ashwell, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581020 · Cited by 53
“Many of these problems are deeply embedded in algorithmic systems, which can reinforce Western tropes, such as associating the word "Muslim" with terrorists [4].”
BackgroundVenkit, P. N., Srinath, M., & Wilson, S. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.3 · Cited by 53
Metadata
- Authors
- Abid, A., Farooqi, M., & Zou, J.
- Affiliations
- Stanford University (US)
- McMaster University (CA)
- Venue
- Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2021-07-21
- Pages
- 298-306
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W3184144760
- Cited by
- 476
- References
- 3
- Citations by year
- 2026: 81 · 2025: 124 · 2024: 105 · 2023: 97 · 2022: 36 · 2021: 33
- FWCI
- 61.36
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- Islamophobia, religious bias, large language models, GPT-3, analogical reasoning, adversarial prompts, story generation, bias mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.