Persistent Anti-Muslim Bias in Large Language Models
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
It has been observed that large-scale language models capture undesirable societal biases, e.g. relating to race and gender; yet religious bias has been relatively unexplored. We demonstrate that GPT-3, a state-of-the-art contextual language model, captures persistent Muslim-violence bias. We probe GPT-3 in various ways, including prompt completion, analogical reasoning, and story generation, to understand this anti-Muslim bias, demonstrating that it appears consistently and creatively in different uses of the model and that it is severe even compared to biases about other religious groups. For instance, Muslim is analogized to terrorist in 23% of test cases, while Jewish is mapped to its most common stereotype, money, in 5% of test cases. We quantify the positive distraction needed to overcome this bias with adversarial text prompts, and find that use of the most positive 6 adjectives reduces violent completions for Muslims from 66% to 20%, but which is still higher than for other religious groups.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Pre-trained on massive amounts of text on the Internet, LMs are known to produce linguistically fawed, factually incorrect, or even ethically problematic text at times [1,24].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs and other AI tools may have a number of biases that could arise from various factors, such as unbalanced training datasets74–77.”
From On opportunities and challenges of large multimodal foundation models in education (2025) · cites this paper as [75]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“it is also a liability; all AI systems might inherit the same problematic biases of a few foundation models [Bolukbasi et al. 2016; Caliskan et al. 2017; Abid et al. 2021, inter alia]”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Abid et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Previous work has shown that language models tend to exhibit human-like biases with respect to specific protected groups such as gender, ethnicity or religious orientation3–6.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [4]
In the citing paper
“If a medical LLM has been trained on data that contains misinformation or biased content, LLMs deployed for clinical decision-making are at risk of reproducing these problematic contents [120,124–126].”
From Bias in medical AI: Implications for clinical decision-making (2024) · cites this paper as [126]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundMethodInfluentialSelf-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.131 · Cited by 2,112 · cited 1×
“These descriptions are generated by passing images through a pretrained image captioning model [11], and fine-tuning GPT-3 to transform the short captions into long descriptions, as shown in Fig.”
MethodReformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · cited 1×
“These languagemodels have evolved, from learning individual word vectors with single-layer models [8], to more complex language generation architectures such as recurrent neural networks [5] and most recently transformers [4, 6, 14].”
BackgroundSemi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2020) · arXiv · doi:10.48550/arxiv.2004.09456 · Cited by 643 · cited 2×
“As a result, even though the various language models have different architectures, since they are trained on similar corpora of text, they often learn similar biases [9].”
BackgroundUniversal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642
Identifying and Reducing Gender Bias in Word-Level Language Models
Bordia, S., & Bowman, S. R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop · doi:10.18653/v1/n19-3002 · Cited by 219 · cited 1×
“Previous work has explored the biases that are ultimately learned by language models, with the primary focus being racial and gender stereotypes [2, 3, 7, 12].”
BackgroundThe Woman Worked as a Babysitter: On Biases in Language GenerationPreprint
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01326 · Cited by 205 · cited 2×
“In [12], the authors made similar observations about language generation models.”
Background
Research works printed but not linked: 6
- 7. T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, A. T. Kalai, Advances in neural information processing systems 29 , 4349 (2016).
- 9. S. Bordia, S. R. Bowman, arXiv preprint arXiv:1904.03035 (2019). · arXiv
- 2. A. M. Dai, Q. V. Le, Advances in neural information processing systems 28 , 3079 (2015).
- 10. K. Lu, P. Mardziel, F. Wu, P. Amancharla, A. Datta, Logic, Language, and Security (Springer, 2020), pp. 189-202.
- 3. A. Vaswani, et al. , Advances in neural information processing systems (2017), pp. 59986008.
- 14. E. Wallace, S. Feng, N. Kandpal, M. Gardner, S. Singh, arXiv preprint arXiv:1908.07125 (2019). · arXiv
Cited by
Items 201–300 of 986
Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education
Zeng, Z., Sha, L., Li, Y., Yang, K., Gašević, D., & Chen, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30258 · Cited by 20
The History and Risks of Reinforcement Learning and Human FeedbackPreprint
Lambert, N., Gilbert, T. K., & Zick, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13595 · Cited by 54
“This includes the wide literature on potential sources of harm from LLMs, including a large swath of social biases, such as anti-Muslim bias (Abid, Farooqi, & Zou, 2021; Liang, Wu, Morency, & Salakhutdinov, 2021).”
BackgroundChen, N., Liu, J., Dong, X., Liu, Q., Sakai, T., & Wu, X.-M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698420 · Cited by 51
Human-Centered Agentic Framework for Machine Learning Modeling in Finance
Okpala, I., Golgoon, A., & Kannan, A. R. (2025) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose67019.2025.00030 · Cited by 52
“They noted the research on FATE (Fairness, Accountability, Transparency, and Ethics) [16], [17], which suggests that as programmable systems become more agen-tic, they may amplify biases and inequities, particularly for marginalized groups.”
An Overview of Large Language Models for Statisticians
Ji, W., Yuan, W., Getzen, E. J., Cho, K., Jordan, M. I., Mei, S., Weston, J., Su, W., Xu, J., & Zhang, L. (2026) · The American Statistician · doi:10.1080/00031305.2026.2657480 · Cited by 51
Cantini, R., Orsino, A., Ruggiero, M., & Talia, D. (2025) · Machine Learning · doi:10.1007/s10994-025-06862-6 · Cited by 53
“Complementary studies have highlighted specific phenomena in LLMs, such as anti-Muslim [18], as cognitive [19], gender-occupational [20], and racial biases [21], demonstrating the complexity and multifaceted nature of bias in LLM outputs.”
Intersectional Bias in Causal Language ModelsPreprint
Magee, L., Ghahremanlou, L., Soldatić, K., & Robertson, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07691 · Cited by 51
“Racial and religion-based bias is also increasingly studied in algorithmic systems [18], and in language models specifically [50, 48].”
BackgroundMethodInfluentialThe Ghost in the Machine has an American accent: value conflict in GPT-3
Johnson, R. L., Pistilli, G., Menédez-González, N., Duran, L. D. D., Panai, E., Kalpokiene, J., & Bertulfo, D. J. (2022) · arXiv · doi:10.48550/arxiv.2203.07785 · Cited by 50
“For example, one study shows GPT-3’s stereotyping bias evidenced by association of the word “Muslims” with violent actions in 66% of 100 iterations of a test[2] as opposed to around 15% of the time for the word “Christians”.”
BackgroundLiu, Y., Gautam, S., Ma, J., & Lakkaraju, H. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.198 · Cited by 51
“LLMs, pre-trained on vast natural language datasets, are particularly susceptible to inheriting these social biases and have been shown to exhibit biases related to gender (Lucy and Bamman, 2021), religion (Abid et al., 2021b) and language variants (Ziems et al., 2023; Liu et al., 2023a).”
BackgroundInfluentialSoftware Fairness: An Analysis and Survey
Soremekun, E., Papadakis, M., Cordy, M., & Traon, Y. L. (2025) · ACM Computing Surveys · doi:10.1145/3762170 · Cited by 50
“Similarly, there has been an increasing attention to bias analysis of LLMs [1, 70, 115], especially due to their importance and recent popularity.”
Search Engines Post-ChatGPT: How Generative Artificial Intelligence Could Make Search Less ReliablePreprint
Memon, S. A., & West, J. D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.11707 · Cited by 51
“Research on the evaluation of language models has repeatedly shown that these models can exhibit gender bias 9–12 , racism 13 , antimuslim stereotypes 14 , antisemitism 15 , etc.”
BackgroundFair Text-to-Image Diffusion via Fair Mapping
Li, J., Hu, L., Zhang, J., Zheng, T., Zhang, H., & Wang, D. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i25.34823 · Cited by 53
Das, D., Guha, S., & Semaan, B. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.8 · Cited by 18
“However, two major cultural identity dimensions such as religion and nationality, have not received much attention (Abid et al., 2021; Nadeem et al., 2020; Ousid-houm et al., 2021).”
BackgroundThe long but necessary road to responsible use of large language models in healthcare research
Kwong, J. C. C., Wang, S., Nickel, G., Cacciamani, G. E., & Kvedar, J. C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01180-y · Cited by 49
First-Person Fairness in Chatbots
Eloundou, T., Beutel, A., Robinson, D. G., Gu-Lemberg, K., Brakman, A.-L., Mishkin, P., Shah, M., Heidecke, J., Weng, L., & Kalai, A. T. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.19803 · Cited by 52
Towards Measuring the Representation of Subjective Global Opinions in Language Models
Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16388 · Cited by 48
“Several studies have found harmful biases related to gender, race, religion and other attributes in these models (Sheng et al., 2019; Abid et al., 2021; Bordia & Bowman, 2019; Liang et al., 2021; Srivastava et al., 2022; Narayanan Venkit et al., 2023; Ladhak et al., 2023; Cheng et al., 2023).”
Questioning the Survey Responses of Large Language Models
Dominguez-Olmedo, R., Hardt, M., & Mendler-Dünner, C. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1458 · Cited by 48
Secret Use of Large Language Model (LLM)
Zhang, Z., Shen, C., Yao, B., Wang, D., & Li, T. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711061 · Cited by 48
“Due to the potential issues in LLM-generated content, including bias [1, 83], hallucination [7, 20, 89], invasion of intellectual property [41], and threats to human autonomy [68], it has become crucial for everyday users of LLM-based services to honestly and accurately report when and how…”
On opportunities and challenges of large multimodal foundation models in education
Küchemann, S., Avila, K. E., Dinc, Y., Hortmann, C., Revenga, N., Ruf, V., Stausberg, N., Steinert, S., Fischer, F., Fischer, M., et al. (2025) · npj Science of Learning · doi:10.1038/s41539-025-00301-w · Cited by 47
Mohamed, Y. H., Abdelfattah, M., Alhuwaider, S., Li, F., Zhang, X., Church, K., & Elhoseiny, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.600 · Cited by 47
Survey on AI Ethics: A Socio‐Technical Perspective
Mbiazi, D., Bhange, M., Babaei, M., Sheth, I., Kenfack, P. J., & Kahou, S. E. (2025) · Computational Intelligence · doi:10.1111/coin.70149 · Cited by 48
Mapping the individual, social and biospheric impacts of Foundation Models
Hernández, A. D., Krishna, S., Perini, A., Katell, M. A., Bennett, S., Borda, A., Hashem, Y., Hadjiloizou, S., Mahomed, S., Jayadeva, S., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658939 · Cited by 48
Large Language Models for Psychological Assessment: A Comprehensive Overview
Brickman, J., Gupta, M., & Oltmanns, J. R. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251343582 · Cited by 50
“…for male pronouns for certain pro-fessions (de Vassimon Manela et al., 2021), indicating some religious groups are more violent than others (Abid et al., 2021), favoring majority groups (Zhang et al., 2020), and propagating differential treatment recommendations based on race (Omiye et…”
Structured like a language model: Analysing AI as an automated subject
Magee, L., Arora, V., & Munn, L. (2023) · Big Data & Society · doi:10.1177/20539517231210273 · Cited by 46
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models
Wang, S., Wang, P., Zhou, T., Dong, Y., Tan, Z. W., & Li, J. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2407.02408 · Cited by 46
“Bias Evaluation of LLMs. Fairness concerns of LLMs have increased as they are incorporated into a wider range of real-world applications [25, 39, 1].”
BackgroundMoral Foundations of Large Language Models
Abdulhai, M., Serapio-García, G., Crepy, C., Valter, D., Canny, J., & Jaques, N. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.982 · Cited by 45
Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality PerspectivesPreprint
Liu, H., Chaudhary, M., & Wang, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16851 · Cited by 45
Kroon, A. C., Welbers, K., Trilling, D., & van Atteveldt, W. (2023) · Communication Methods and Measures · doi:10.1080/19312458.2023.2261372 · Cited by 44
“As a consequence, large language models have the potential to embed racial, political, gender, or other harmful societal biases (e.g., Abid et al., 2021; Guo & Caliskan, 2021; Nadeem et al., 2020; Silva et al., 2021).”
BackgroundNatural language processing for social science research: A comprehensive review
Hou, Y., & Huang, J. (2025) · Chinese Journal of Sociology · doi:10.1177/2057150x241306780 · Cited by 45
“Numerous studies have demonstrated that the content generated by LLMs inherits and even exacerbates these social biases (Abid et al., 2021; Bender et al., 2021; Gallegos et al., 2024; Kotek et al., 2023; OpenAI et al., 2024).”
Dewitte, P. (2024) · Computer law & security review · doi:10.1016/j.clsr.2024.106019 · Cited by 45
Formanek, M. (2024) · Journal of Librarianship and Information Science · doi:10.1177/09610006241241066 · Cited by 42
“The highlighted disparities in responses by language models closely align with the findings of current scholarly inquiries conducted by various authors (such as Abid et al., 2021; Bender et al., 2021; Chan, 2023; Li, 2023; Sajan and Kyle, 2023 among others).”
Political biases and inconsistencies in bilingual GPT models—the cases of the U.S. and China
Zhou, D., & Zhang, Y. (2024) · Scientific Reports · doi:10.1038/s41598-024-76395-w · Cited by 42
“Despite these limitations, our study contributes to the existing body of research highlighting various biases in GPT models and AI ethics in general 3–9, 33,”
MethodComputational Modeling of Stereotype Content in Text
Fraser, K., Kiritchenko, S., & Nejadgholi, I. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.826207 · Cited by 42
“More recently, work has focused on detecting stereotypical associations present in large-scale language models (Abid et al., 2021; de Vassimon Manela et al., 2021).”
BackgroundSearching for Computer Vision North Stars
Fei-Fei, L., & Krishna, R. (2022) · Daedalus · doi:10.1162/daed_a_01902 · Cited by 42
“31 In response to these concerns, pioneers in the field are radically rethinking benchmarking.”
BackgroundReclaiming the Digital Commons: A Public Data Trust for Training Data
Chan, A., Bradley, H., & Rajkumar, N. (2023) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604658 · Cited by 26
“characteristics are passed onto models [8, 18, 39, 46, 109].”
Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles
Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T.-H., & Wilson, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604667 · Cited by 42
“Negative [1] 10 8 44 102 Somewhat Negative [2] 36 60 74 117”
BackgroundLUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 41
“For example, recent research indicates that existing LLMs can occasionally generate content that is toxic, biased, insecure, or erroneous [9], [13], [14].”
BackgroundAdvances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review
Rostam, Z. R. K., & Kertész, G. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3763002 · Cited by 42
Albrecht, S. (2023) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000158070 · Cited by 45
Sebastian, G., & Sebastian, S. S. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4461801 · Cited by 41
Unveiling the Implicit Toxicity in Large Language Models
Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., & Huang, M. (2023) · arXiv · doi:10.48550/arxiv.2311.17391 · Cited by 40
“…exhibit various safety issues, such as generating offensive contents (Gehman et al., 2020), reinforcing unfairness/discrimination (Sap et al., 2020; Abid et al., 2021), leaking privacy information (Carlini et al., 2021; Zhang et al., 2023b), and promoting illegal activities (Zhang et al., 2023a).”
BackgroundA survey on fairness of large language models in e-commerce: progress, application, and challengePreprint
Ren, Q., Jiang, Z., Cao, J., Li, S., Li, C., Liu, Y., Huo, S., He, T., & Chen, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.13025 · Cited by 41
“Previous work has shown that LLMs tend to reinforce social biases in their generation outputs due to the bias in the large pre-training corpus, leading to unfair treatment of vulnerable groups [75,76].”
BackgroundUncovering labeler bias in machine learning annotation tasks
Haliburton, L., Leusmann, J., Welsch, R., Ghebremedhin, S., Isaakidis, P., Schmidt, A., & Mayer, S. (2024) · AI and Ethics · doi:10.1007/s43681-024-00572-w · Cited by 40
“With recent developments in generative models, the issue has only been elevated, c.f., Abid et al. [6].”
BackgroundLozoya, D. C., Conway, M., Duro, E. S. D., & D’Alfonso, S. (2025) · JMIR Medical Education · doi:10.2196/68056 · Cited by 39
Exploring Safety-Utility Trade-Offs in Personalized Language Models
Vijjini, A. R., Chowdhury, S. B. R., & Chaturvedi, S. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.565 · Cited by 39
Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge NetworksPreprint
Jin, B., Zhang, Y., Yu, M., & Han, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11050 · Cited by 39
“…C ONSIDERATIONS While it has been demonstrated that PLMs are powerful in language understanding (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020), there are studies pointing out their drawbacks such as containing social bias (Liang et al., 2021) and misinformation (Abid et al., 2021).”
BackgroundLanguage Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
Evaluating Gender Bias of LLMs in Making Morality Judgements
Bajaj, D., Lei, Y., Tong, J., & Huang, R. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.928 · Cited by 40
Baeza-Yates, R. A. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498370 · Cited by 38
“which exhibit many societal biases, including gender [7, 8], race [11, 17] and religious [1] bias.”
BackgroundBerberette, E., Hutchins, J., & Sadovnik, A. (2024) · arXiv.org · doi:10.48550/arxiv.2402.01769 · Cited by 16
“Furthermore, significant research has already been performed in studying bias in LLMs (Vig et al., 2020; Abid et al., 2021; Liang et al., 2021).”
BackgroundAI Risk Management Should Incorporate Both Safety and SecurityPreprint
Qi, X., Huang, Y., Zeng, Y., Debenedetti, E., Geiping, J., He, L., Huang, K., Madhushani, U., Sehwag, V., Shi, W., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.19524 · Cited by 40
“…& Chadha, 2019); AI models deployed in socially critical contexts generating unintended toxic content (Gehman et al., 2020), perpetuating bias (Abid et al., 2021), and producing misleading information (Kreps et al., 2022); as well as long-term concerns like uncontrolled rogue AIs (Hendrycks…”
BackgroundMirowski, P., Love, J., Mathewson, K., & Mohamed, S. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658993 · Cited by 37
“These biases include sexism and racism [11, 94], homophobia and transphobia [31, 84], Islamophobia [1], the perpetuation of Western colonial mindsets [73], Anglocentrism [115], and in-group vs. out-group social identity biases [48].”
BackgroundUnlocking Bias Detection: Leveraging Transformer-Based Models for Content Analysis
Raza, S., Bamgbose, O., Chatrath, V., Ghuge, S., Sidyakin, Y., & Muaad, A. Y. M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3392469 · Cited by 38
“5 [35].”
Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias
Chen, Y., Raghuram, V. C., Mattern, J., Mihalcea, R. F., & Jin, Z. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.281 · Cited by 40
“Although this study focuses on occupational gender bias, the proposed causal formulation and desider-ata can be generalized to other social biases, as long as the stereotype and demographic are objec-tive and observable.”
Quite Good, but Not Enough: Nationality Bias in Large Language Models - a Case Study of ChatGPT
Zhu, S., Wang, W., & Liu, Y. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2ptonpvxx2s2 · Cited by 37
The Attention Economy and the Impact of Artificial Intelligence
Baeza-Yates, R. A., & Fayyad, U. M. (2021) · Springer eBooks · doi:10.1007/978-3-030-86144-5_18 · Cited by 37
“There are examples of gender bias (Caliskan et al. 2017), The Attention Economy and the Impact of Arti fi cial Intelligence race bias (Larson et al. 2016), religious bias (Abid et al. 2021), etc., and their impact has many rami fi cations (Bender et al. 2021).”
BackgroundQuantifying Social Biases Using Templates is UnreliablePreprint
Seshadri, P., Pezeshkpour, P., & Singh, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.04337 · Cited by 37
“For example, Abid et al. [2021] highlight that GPT-3 consistently displays anti-Muslim biases that are much more severe than biases against other religious groups.”
BackgroundWHAT’S IN THE CHATTERBOX? LARGE LANGUAGE MODELS, WHY THEY MATTER, AND WHAT WE SHOULD DO ABOUT THEM
Okerlund, J., Klasky, E., Middha, A., Kim, S., Rosenfeld, H., Kleinman, M., & Parthasarathy, S. (2022) · Deep Blue (University of Michigan) · doi:10.7302/21898 · Cited by 37
Initial policy considerations for generative artificial intelligence
(2023) · OECD Artificial Intelligence Papers · doi:10.1787/fae2d1e6-en · Cited by 36
In the Name of Fairness: Assessing the Bias in Clinical Record De-identification
Xiao, Y., Lim, S., Pollard, T., & Ghassemi, M. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593982 · Cited by 38
“For example, NLP models link the female gender to specific stereotypical occupations [28] and tend to generate violent or negative-toned text when given “Muslim” as a demographic descriptor for input [11].”
BackgroundDoo, F. X., Cook, T. S., Siegel, E. L., Joshi, A., Parekh, V. S., Elahi, A., & Yi, P. H. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.07.007 · Cited by 36
Did the Neurons Read your Book? Document-level Membership Inference for Large Language ModelsPreprint
Meeus, M., Jain, S., Rei, M., & de Montjoye, Y.-A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.15007 · Cited by 37
“This includes questions about what they will learn, e.g. whether modern models will learn the societal biases present in the data or amplify them [1,44,63] or whether they might learn and propagate disinformation [5, 45,67,76].”
BackgroundBeyond Chatbot-K: On Large Language Models, “Generative AI,” and Rise of Chatbots—An Introduction
Goodlad, L. M. E., & Stone, M. (2024) · Critical AI · doi:10.1215/2834703x-11205147 · Cited by 36
Wang, H., & Shu, K. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679821 · Cited by 35
Lazy Data Practices Harm Fairness Research
Simson, J., Fabris, A., & Kern, C. A. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658931 · Cited by 34
“While fairness research has been able to study Muslim bias in language models [2, 32, 72], so far it has neglected allocative harms against Muslim people.”
BackgroundSociotechnical Implications of Generative Artificial Intelligence for Information Access
Mitra, B., Cramer, H. S. M., & Gurevich, O. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_7 · Cited by 36
Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities
Cattell, S., Ghosh, A., & Kaffee, L.-A. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31635 · Cited by 34
“Harmful biases in language models generating toxic outputs against marginalized groups are often observed due to vigilance within directly impacted communities (Cosmos Magazine 2023; Abid, Farooqi, and Zou 2021).”
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
Into the LAION’s Den: Investigating Hate in Multimodal Datasets
Birhane, A., Prabhu, V., Han, S., Boddeti, V., & Luccioni, S. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0930 · Cited by 33
How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studies
Leidinger, A., & Rogers, R. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31684 · Cited by 33
CausalMapper: Challenging designers to think in systems with Causal Maps and Large Language Model
Huang, Z., Quan, K., Chan, J., & MacNeil, S. (2023) · Creativity and Cognition · doi:10.1145/3591196.3596818 · Cited by 24
“However, researchers have also raised concerns about the truthiness of responses [3] and have identified problematic biases [1, 28].”
BackgroundIdentifying and Mitigating Vulnerabilities in LLM-Integrated ApplicationsPreprint
Jiang, F., Xu, Z., Niu, L., Wang, B., Jia, J., Li, B., & Poovendran, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.16153 · Cited by 33
“In (Abid et al., 2021; Brown et al., 2020), it was demonstrated that language models persistently capture religious biases.”
BackgroundPLLuM: A Family of Polish Large Language ModelsPreprint
Kocoń, J., Piasecki, M., Janz, A., Ferdinan, T., Radliński, Ł., Koptyra, B., Oleksy, M., Woźniak, S., Walkowiak, P., Wojtasik, K., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.03823 · Cited by 34
Safe RLHF: Safe Reinforcement Learning from Human Feedback
Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., & Yang, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.12773 · Cited by 32
Lee, B. H., Lim, S., & Chun, S. Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01733 · Cited by 32
“In recent years, the risks of large-scale models in generating inappropriate content have been widely examined [1, 4, 23], and these concerns extend to T2I diffusion models like Stable Diffusion (SD) [10, 47, 49].”
Abrar, A., Oeshy, N. T., Kabir, M., & Ananiadou, S. (2025) · AI & Society · doi:10.1007/s00146-025-02721-z · Cited by 32
“For instance, in 23% of test cases, the model equates “Muslim” with “terrorist”, while in 5% of test cases it linked “Jewish” with “money”(Abid et al., 2021).”
InfluentialAccessEval: Benchmarking Disability Bias in Large Language Models
Panda, S., Agarwal, A., & Patel, H. L. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1653 · Cited by 33
Zhao, Y., Wang, B., Wang, Y., Zhao, D., Jin, X., Zhang, J., He, R., & Hou, Y. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3fpm9f8exw68 · Cited by 32
Using GPT-3 to Build a Lexicon of Drugs of Abuse Synonyms for Social Media Pharmacovigilance
Carpenter, K. A., & Altman, R. B. (2023) · Biomolecules · doi:10.3390/biom13020387 · Cited by 33
HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models
Li, Y., Zhang, G., Yang, B., Lin, C., Ragni, A., Wang, S., & Fu, J. (2022) · doi:10.18653/v1/2022.findings-aacl.32 · Cited by 33
Whose Side are You on: Investigating Political Bias of Large Language Models
Pit, P., Ma, X., Conway, M., Chen, Q., Bailey, J., Pit, P., Keo, P., Diep, W., & Jiang, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4969-6_22 · Cited by 33
Proceedings of the Third Workshop on Narrative Understanding
(2021) · doi:10.18653/v1/2021.nuse-1 · Cited by 31
Debiasing large language models: research opportunities*
Yogarajan, V., Dobbie, G., & Keegan, T. T. (2024) · Journal of the Royal Society of New Zealand · doi:10.1080/03036758.2024.2398567 · Cited by 30
“LLMs inherit stereotypes and misrepresen-tations of societies from the training data (Bender et al. 2021; Yogarajan et al. 2023a), and can also amplify these biases (Abid et al. 2021; Crutchley 2021).”
Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
Purpura, A., Wadhwa, S., Zymet, J., Gupta, A., Luo, A., Rad, M. K., Shinde, S. D., & Sorower, M. S. (2025) · Workshop on Trustworthy NLP (TrustNLP) · doi:10.18653/v1/2025.trustnlp-main.23 · Cited by 30
AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights
Xu, J., Li, G., & Jiang, J. Y. (2025) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v8i3.36755 · Cited by 31
Towards Responsible Natural Language Annotation for the Varieties of Arabic
Bergman, A., & Diab, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.31 · Cited by 30
Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models
Basu, S., Sattigeri, P., Ramamurthy, K. N., Chenthamarakshan, V., Varshney, K. R., Varshney, L. R., & Das, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25832 · Cited by 30
From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards
Chehbouni, K., Roshan, M., Ma, E., Wei, F. A., Taïk, A., Cheung, J. C. K., & Farnadi, G. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.927 · Cited by 30
Semantically Aligned Task Decomposition in Multi-Agent Reinforcement LearningPreprint
Li, W., Qiao, D., Wang, B., Wang, X., Jin, B., & Zha, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10865 · Cited by 30
(2023) · doi:10.52202/075280-1482 · Cited by 29
Wikipedia and AI: Access, representation, and advocacy in the age of large language models
McDowell, Z. J. (2024) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565241238924 · Cited by 29
LePaRD: A Large-Scale Dataset of Judicial Citations to Precedent
Mahari, R., Stammbach, D., Ash, E., & Pentland, A. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.532 · Cited by 29
“Secondly, LLMs have been shown to exhibit biases (Abid et al., 2021; Lucy and Bamman, 2021) and the resulting classification of passages in our application might potentially perpetuate these biases.”
BackgroundDeception for Cyber Defence: Challenges and Opportunities
Liebowitz, D., Nepal, ., Moore, K., Christopher, C. J., Kanhere, S. S., Nguyen, D. D., Timmer, R. C., Longland, M., & Rathakumar, K. (2021) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tpsisa52974.2021.00020 · Cited by 31
“We recognise that content generation approaches that are privacy preserving, and which prevent language models like GPT from generating toxic content [77], [78] are going to be increasingly important issues that we should pay attention to.”
BackgroundSalehi, N., Pakzad, R., Lajevardi, N., & Asad, M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579526 · Cited by 29
“Research has shown how negative sentiment around Islam and Muslims are actually built into these automated systems [1].”
BackgroundDron, J. (2023) · Digital · doi:10.3390/digital3040020 · Cited by 29
Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers
Wang, Y., Wu, X., Wu, H.-T., Tao, Z., & Fang, Y. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.319 · Cited by 29
The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models
Birhane, A., Dehdashtian, S., Prabhu, V., & Boddeti, V. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658968 · Cited by 28
“Racial and gender bias in language models [2, 6, 17] and vision models [15, 34, 46, 76] is a well documented phenomena.”
BackgroundGPT models for text annotation: An empirical exploration in public policy research
Churchill, A., Pichika, S., Xu, C., & Liu, Y. (2025) · Policy Studies Journal · doi:10.1111/psj.70034 · Cited by 29
“Given the source of its training data, GPT models are very likely to be westernized and possess bias (Abid et al., 2021; Bender et al., 2021).”
Introduction to Interpretability and Explainability
Kamath, U., & Liu, J. (2021) · Springer eBooks · doi:10.1007/978-3-030-83356-5_1 · Cited by 28
Metadata
- Authors
- Abid, A., Farooqi, M., & Zou, J.
- Affiliations
- Stanford University (US)
- McMaster University (CA)
- Venue
- Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2021-07-21
- Pages
- 298-306
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W3184144760
- Cited by
- 476
- References
- 3
- Citations by year
- 2026: 81 · 2025: 124 · 2024: 105 · 2023: 97 · 2022: 36 · 2021: 33
- FWCI
- 61.36
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- Islamophobia, religious bias, large language models, GPT-3, analogical reasoning, adversarial prompts, story generation, bias mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.