AI generates covertly racist decisions about people based on their dialect
Open accessThe DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
Hundreds of millions of people now interact with language models, with uses ranging from help with writing 1,2 to informing hiring decisions 3 . However, these language models are known to perpetuate systematic racial prejudices, making their judgements biased in problematic ways about groups such as African Americans 4–7 . Although previous research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time, particularly in the United States after the civil rights movement 8,9 . It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice, exhibiting raciolinguistic stereotypes about speakers of African American English (AAE) that are more negative than any human stereotypes about African Americans ever experimentally recorded. By contrast, the language models’ overt stereotypes about African Americans are more positive. Dialect prejudice has the potential for harmful consequences: language models are more likely to suggest that speakers of AAE be assigned less-prestigious jobs, be convicted of crimes and be sentenced to death. Finally, we show that current practices of alleviating racial bias in language models, such as human preference alignment, exacerbate the discrepancy between covert and overt stereotypes, by superficially obscuring the racism that language models maintain on a deeper level. Our findings have far-reaching implications for the fair and safe use of language technology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Hofmann et al. ( 2024 ) found that models trained with RLHF exhibited weaker overt prejudice (using terms to describe African Americans such as 'passionate' or 'imaginative') but this did not diminish covert prejudice.”
From AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War (2025) · cites this paper as Hofmann et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“AI may also introduce bias, influenced by the data used in training [29].”
From Using artificial intelligence to semi-automate trustworthiness assessment of randomized controlled trials: a case study (2025) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“Notably, subtle biases, such as those related to linguistic connotations, regional dialects, or implicit associations, can be especially insidious and difficult to eliminate through technical safeguards144.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Similarly, as LLMs are systems based on predictive text algo rithms it is perhaps not surprising that they have given sexist (Wan et al., 2023 ) and racist (Hofmann et al., 2024 ) outputs.”
From AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War (2025) · cites this paper as Hofmann et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“The researchers found that, while these models overtly associate African Americans with positive attributes (such as 'brilliant') they covertly associated African Ameri -cans with exclusively negative attributes, (words such as 'lazy' and 'dirty').”
From AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War (2025) · cites this paper as Hofmann et al., 2024
11 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“It is worth mentioning that the lack of awareness of covert racism also manifests during evaluation, where it is common to test language models for overt racism but not for covert racism 21,63,79,80 .”
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2×
“The typical pipeline of training language models includes steps such as data filtering 48 and, more recently, HF training 62 that remove overt racial prejudice.”
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 3×
“In the Supplementary Information, we show that this deviation was caused by the base T5 version, and that the larger T5 versions follow the general pattern (Supplementary Table 10).”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“As a second potential way to resolve dialect prejudice in language models, we examined training with HF 49,62 .”
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 11×
“2 and Supplementary Table 4).”
InfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Unsettling race and language: Toward a raciolinguistic perspective
Rosa, J., & Flores, N. (2017) · Language in Society · doi:10.1017/s0047404517000562 · Cited by 1,675 · cited 1×
“These associations represent raciolinguistic ideologies, demonstrating how AAE is othered through the emphasis on its perceived deviance from standardized norms 44 .”
Research works printed but not linked: 15
- Zhao, W. et al. WildChat: 1M ChatGPT interaction logs in the wild. In Proc. Twelfth International Conference on Learning Representations (OpenReview.net, 2024).
- Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V. & Kalai, A. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Adv. Neural Inf. Process. Syst. 30, 4356–4364 (2016).
- Li, L. & Bamman, D. Gender and representation bias in GPT-3 generated stories. In Proc. Third Workshop on Narrative Understanding (eds Akoury, N. et al.) 48–55 (Association for Computational Linguistics, 2021).
- Grogger, J. Speech patterns and racial wage inequality. J. Hum. Resour. 46, 1–25 (2011).
- Rodriguez, J. I., Cargile, A. C. & Rich, M. D. Reactions to African-American vernacular English: do more phonological features matter? West. J. Black Stud. 28, 407–414 (2004).
- Surden, H. Artificial intelligence and law: an overview. Ga State Univ. Law Rev. 35, 1305–1337 (2019).
- Schuman, H., Steeh, C., Bobo, L. D. & Krysan, M. (eds) Racial Attitudes in America: Trends and Interpretations (Harvard Univ. Press, 1998).
- Terkel, S. Race: How Blacks and Whites Think and Feel about the American Obsession (New Press, 1992).
- Bonilla-Silva, E. The New Racism: Racial Structure in the United States, 1960s–1990s. In Race, Ethnicity, and Nationality in the United States: Toward the Twenty-First Century 1st edn (ed. Wong, P.) Ch. 4 (Westview Press, 1999).
- Ethayarajh, K., Choi, Y. & Swayamdipta, S. Understanding dataset difficulty with V-usable information. In Proc. 39th International Conference on Machine Learning (eds Chaudhuri, K. et al.) 5988–6008 (Proceedings of Machine Learning Research, 2022).
- Eisenstein, J., O’Connor, B., Smith, N. A. & Xing, E. P. A latent variable model for geographic lexical variation. In Proc. 2010 Conference on Empirical Methods in Natural Language Processing (eds Li, H. & Màrquez, L.) 1277–1287 (Association for Computational Linguistics, 2010).
- Eisenstein, J. What to do about bad language on the internet. In Proc. 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Vanderwende, L. et al.) 359–369 (Association for Computational Linguistics, 2013).
- Zhao, T. Z., Wallace, E., Feng, S., Klein, D. & Singh, S. Calibrate before use: Improving few-shot performance of language models. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 12697–12706 (Proceedings of Machine Learning Research, 2021).
- Jurafsky, D. & Martin, J. H. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition (Prentice Hall, 2000).
- Santurkar, S. et al. Whose opinions do language models reflect? In Proc. 40th International Conference on Machine Learning (eds Krause, A. et al.) 29971–30004 (Proceedings of Machine Learning Research, 2023).
Cited by
Items 101–200 of 476
Ethical Considerations in the Deployment of Artificial Intelligence in Surgery
Barrios, P. A., Meredyth, N. A., Morris, R., Haines, K., Kim, G. J., & Peterson, C. Y. (2025) · Journal of Surgical Research · doi:10.1016/j.jss.2025.08.019 · Cited by 9
Hall, K., McKeever, S., Ojabo, M., Simonda, I., Tighe, J., & Shaw, J. (2025) · Nursing Outlook · doi:10.1016/j.outlook.2025.102652 · Cited by 8
Perez, R. M., Shimogawa, M., Chang, Y., Li, X., Phan, H. A. T., Marmorstein, J. G., Yanagawa, E. S. K., & Petersson, E. J. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100546 · Cited by 11
Proszak, L. (2026) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/per-b.2026.2777.2.41 · Cited by 8
Heilig, L., & Chase, J. (2026) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/per-b.2026.2777.2.38 · Cited by 8
Mohammed, R., & Belhadi, K. (2026) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/per-b.2026.2777.2.33 · Cited by 8
Lee, J., Borchers, C., Alvero, A., Joachims, T., & Kizilcec, R. F. (2026) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3774398.3811620 · Cited by 8
“LLMs may also reproduce linguistic norms associated with privilege [3, 22, 36], generating text that reads as performatively sophisticated without the narrative specificity readers associate with genuine voice [24, 28].”
Exploring the Intersection of AI, Language, and Law: A Bibliometric Analysis
Lan, Z., & Zhang, X. (2026) · SAGE Open · doi:10.1177/21582440261431000 · Cited by 10
“More broadly, the keyword map suggests an imbalance: technical terms do not consistently co-occur with legal-ethical concepts, indicating that normative questions about legally consequential AI remain less integrated in the literature (Hofmann et al., 2024).”
Collaborative Growth: When Large Language Models Meet Sociolinguistics
Nguyen, D. (2025) · Language and Linguistics Compass · doi:10.1111/lnc3.70010 · Cited by 18
Comparing ChatGPT with human judgements of social traits from face photographs
Kramer, R. S. S. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100156 · Cited by 11
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based SystemsPreprint
Harvey, E., Sheng, E., Blodgett, S. L., Chouldechova, A., Garcia-Gathright, J. I., Olteanu, A., & Wallach, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.15662 · Cited by 8
“…, 1 including tools [e.g., 40, 14], datasets [e.g., 64, 28, 32, 34, 56], metrics [e.g., 11, 15, 10, 58, 43], benchmarks (consisting of both datasets and metrics) [e.g., 23, 48, 45, 46, 55, 63, 22, 24, 26, 27, 31], annotation instructions [e.g., 42], and other techniques [e.g., 36, 52, 61].”
Perez, R. M., Shimogawa, M., Chang, Y., Phan, H. A. T., Marmorstein, J. G., Yanagawa, E. S. K., & Petersson, E. J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.00016 · Cited by 7
Spennemann, D. H. R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5393987 · Cited by 9
Identity-Aware Large Language Models require Cultural ReasoningPreprint
Plum, A., Lutgen, A.-M., Purschke, C., & Rettinger, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.18510 · Cited by 8
“…: Reporting on covert forms of racism, researchers found that models such as ChatGPT and Gemini hold biased stereotypes against speakers of African American Vernacular English, perceiving them as less intelligent or employable and resulting in reduced recommendations or harsher judgments [39].”
Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory
Hafner, F. S., Valdivia, A., & Rocher, L. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732112 · Cited by 11
“Hofmann et al. [23] found that such efforts can successfully reduce negative overt racist stereotypes but exacerbate covert stereotypes, in the form of dialect prejudice.”
InfluentialDavidson, T. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02360-w · Cited by 12
Haginoya, S., Sun, Y., Yamamoto, S., Mizushi, H., Yoshimoto, N., & Santtila, P. O. (2025) · Applied Cognitive Psychology · doi:10.1002/acp.70031 · Cited by 8
Examining Chat GPT with nonwords and machine psycholinguistic techniques
Vitevitch, M. S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0325612 · Cited by 9
AI biases as asymmetries: a review to guide practice
Waters, G., & Honenberger, P. (2025) · Frontiers in Big Data · doi:10.3389/fdata.2025.1532397 · Cited by 8
“Even after years of large-scale bias mitigation efforts, these associations sometimes resurface, manifesting themselves in unexpected ways (Hofmann et al., 2024).”
Algorithmic Fairness and Color-blind Racism: Navigating the IntersectionPreprint
Watson-Daniels, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.07778 · Cited by 7
Chatbot AI makes racist judgements on the basis of dialect
Gibney, E. (2024) · Nature · doi:10.1038/d41586-024-00779-1 · Cited by 8
Digital accents, homogeneity-by-design, and the evolving social science of written language
Alvero, A., Sedlacek, Q. C., León, M., & Peña, C. (2025) · Annual Review of Applied Linguistics · doi:10.1017/s0267190525000042 · Cited by 10
“…models, when tasked with evaluating testimony from a hypothetical defendant in a first-degree murder trial, were much more likely to recommend the death penalty if the defendant’s testimony exhibited (written, textual) features of African American English than if it did not (Hofmann et al., 2024).”
Finding our ROLE: How and why to reframe essentialist approaches to language
Namboodiripad, S., Kutlu, E., Babel, A. M., Babel, M., Baese-Berk, M., Bassuk, P. B., Block, A., Pérez, R. C., Carlson, M. T., Carraturo, S., et al. (2026) · Cognition · doi:10.1016/j.cognition.2026.106444 · Cited by 12
The Case for Artificial Intelligence in Qualitative Research
Schildt, H. A. (2026) · Organization Studies · doi:10.1177/01708406261452993 · Cited by 7
“Research also has shown that LLMs exhibit strong biases that even exceed their training data (Hofmann et al., 2024).”
Son, J., & Philpott, A. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009650809 · Cited by 8
Haq, I., & Saldías-Fuentes, B. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812419 · Cited by 9
Broadhead, L.-A. (2024) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2024.7.2.9 · Cited by 7
Language Models and Dialect Differences
Ocumpaugh, J. L., Liu, X., & Zambrano, A. F. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706496 · Cited by 9
“All that Glitters”: Techniques for Evaluations with Unreliable Model and Human Annotations
Hardy, M. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.120 · Cited by 14
“…high-stakes tasks, reducing discriminatory biases found in data (Field et al., 2021) that are immutably historical by definition of their creation, etc.), which is especially true of autoregressive models, whose labels are Internet text and which contain harmful biases (Hofmann et al., 2024a,b).”
Exploring LLMs for Automated Generation and Adaptation of QuestionnairesPreprint
Adhikari, D. M., Hartland, A., Weber, I., & Cannanure, V. K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.05985 · Cited by 6
Mitigating LLM Hallucinations with Knowledge Graphs: A Case StudyPreprint
Li, H. X., Appleby, G., Alperin, K., Gomez, S. R., & Suh, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.12422 · Cited by 10
EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language ModelsPreprint
Gupta, A., Cheung, J., Meng, P., Sayyed, S., Liao, A., Zhu, K., & O'Brien, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.07100 · Cited by 6
Havens, L., Bach, B., Terras, M., & Alex, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713217 · Cited by 6
Shi, Z., Chung, D., Du, Y., Zhang, J., Raina, S., & Matarić, M. J. (2025) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3713043.3728841 · Cited by 9
Laracy, J. R., Kirova, V. D., Ku, C. S., & Marlowe, T. J. (2025) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics65148.2025.11098185 · Cited by 5
“Similar observations and conclusions apply for situations reported by Hoffman et al. [76], reporting how AI generates or reinforces covertly racist perceptions in hiring and other recommendations.”
Defining the intension and extension of nations’ sovereignty in the age of generative AI
Sayankina, S., Tsen, E., Tan, E., & Ko, R. K. L. (2025) · Territory Politics Governance · doi:10.1080/21622671.2025.2568474 · Cited by 9
“…2023) and social (Dhamala et al., 2021) bias, they can be viewed as an exercise in privilege and an instrument of power by the big tech (Talat et al., 2022), as illustrated by instances of AI generating covertly racist decisions based on people’s dialect or skin tones (Hofmann et al., 2024).”
Adler, J., Herrmann, F. P., Ludwig, A., Wüllenweber, C., Heimsath, A., Gratzl, J., & Tolios, A. (2025) · Journal of Laboratory Medicine · doi:10.1515/labmed-2025-0223 · Cited by 6
Krutrök, M. E., & Isling, P. P. (2025) · New Media & Society · doi:10.1177/14614448251385920 · Cited by 7
“…high stakes: while AI-supported tools can enhance language learning and adaptation (Zou et al., 2023), prevailing systems often reproduce biases (Hofmann et al., 2024), dilute linguistic specificity (Chandran, 2014), or mis-classify minority languages, undermining their digital and cultural…”
Jungherr, A. M., & Rauchfleisch, A. (2025) · Social Media + Society · doi:10.1177/20563051251405069 · Cited by 9
“As a result, they are prone to reproducing the biases embedded in their training data (Bianchi et al., 2023; Friedrich, Brack, et al., 2024; Friedrich, Hämmerl, et al., 2024; Hofmann et al., 2024; Tao et al., 2024; Weidinger et al., 2021).”
AI storytelling across media: synthetic agency and the force vectors of narrative world-making
Sánchez-López, I., Castells, A. G., & Roig, A. (2026) · Frontiers in Communication · doi:10.3389/fcomm.2026.1821118 · Cited by 6
Webster, C. S., & Jowsey, T. (2025) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000007140 · Cited by 7
The Impact of Artificial Intelligence on Human Resource Management: Barriers and Benefits
Maota, T., & Naidoo, V. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-8679-8.ch016 · Cited by 5
A Taxonomy of Challenges to Curating Fair Datasets
Zhao, D., Scheuerman, M. K., Chitre, P., Andrews, J. T. A., Panagiotidou, G., Walker, S., Pine, K. H., & Xiang, A. (2024) · neural information processing systems · doi:10.52202/079017-3103 · Cited by 11
Redefining Bias Audits for Generative AI in Health Care
Chen, I. Y., & Alsentzer, E. (2025) · NEJM AI · doi:10.1056/aip2500015 · Cited by 8
Pussadeniya, N., Nakisa, B., & Rastgoo, M. N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.14166 · Cited by 7
“For example, (Hofmann et al., 2024) demonstrate that an AI system produced “covertly racist decisions” based solely on a user’s dialect.”
Generative AI Use in Entrepreneurship: An Integrative Review and an Empowerment-Entrapment FrameworkPreprint
Lu, J. G., Zhao, G. G., & Zheng, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.02567 · Cited by 8
Child and adolescent psychiatry: challenges, solutions, opportunities, and future directions
Cortese, S., Arango, C., Aymerich, C., Catalan, A., Chetouani, M., Cohen, D., Coghill, D., Gabellone, A., Iniesta, R., Kadan, A., et al. (2026) · World Psychiatry · doi:10.1002/wps.70044 · Cited by 4
“Even after deployment, ongoing monitoring is essential, including regular random checks by humans, to detect and address any instances of AI-generated misinformation 402,403 .”
Moser, M., Krogmann, L., & Wanke, D. (2025) · Integrative Conservation · doi:10.1002/inc3.70013 · Cited by 5
Gong, Y., Liu, Y., Shang, L., Wei, N., & Wang, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757577 · Cited by 6
Large language models (LLMs) as jurors: Assessing the potential of LLMs in legal contexts.
Sun, Y., Zappalà, A., Maso, E. D., Pompedda, F., Nyman, T. J., & Santtila, P. O. (2025) · Law and Human Behavior · doi:10.1037/lhb0000620 · Cited by 5
Explainable pedagogical alignment: auditing LLMs with learning theories
Figaredo, D. D., Vindel, J. L. F., & Puerta, L. G. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13857-2 · Cited by 5
Administrative Decision-Making with Generative AI: The Challenge of Epistemic Boundedness
Kim, Y., Kim, J., Kim, T., & Cho, H.-C. (2026) · Administration & Society · doi:10.1177/00953997251409156 · Cited by 5
“…require analyzing lengthy policy documents remain challenging for LLMs. Additionally, studies spanning 2024 to 2025 have documented pervasive bias across political (Motoki et al., 2024), cultural (Tao et al., 2024), racial (Hofmann et al., 2024), and geographical (Manvi et al., 2024) dimensions.”
Walker, C. (2025) · College English · doi:10.58680/ce2025874458 · Cited by 4
Wei, S.-L., Liao, Y.-L., Chang, Y.-H., Huang, H.-H., & Chen, H.-H. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.80 · Cited by 7
Generative AI and multimodal pedagogy: implications for teaching, learning, and assessment
Istrate, O. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1828953 · Cited by 5
Once Upon a Team: Investigating Bias in LLM-Driven Software Team Composition and Task Allocation
Parziale, A., Voria, G., Pontillo, V., Salle, A. D., Pelliccione, P., Catolino, G., & Palomba, F. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786581.3786930 · Cited by 4
WHERE: Humans, Technology, and Humane Technology
Walther, C. C. (2024) · Springer eBooks · doi:10.1007/978-3-031-67823-3_2 · Cited by 8
From Filling Gaps to Amplifying Strengths: Exploring an Asset-Based Approach to Learning Analytics
Li, F., & Wise, A. F. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706542 · Cited by 4
Algorithmic Harm: Protecting People in the Age of Artificial Intelligence
Sunstein, C. R., & Bar‐Gill, O. (2025) · doi:10.1093/oso/9780197778197.001.0001 · Cited by 8
Large Language Model Annotation Bias in Hate Speech Detection
Okpala, E., & Cheng, L. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35879 · Cited by 15
Lepp, H., & Alvero, A. (2025) · Social Science Computer Review · doi:10.1177/08944393251366242 · Cited by 5
Więckowski, Z., & Świerczyński, M. (2025) · PRAWO i WIĘŹ · doi:10.36128/priw.vi54.1171 · Cited by 4
Challenges and Opportunities of Using Generative AI for Research: Opening the Discussion
Goretti, L., Küsters, A., Morgana, S., & Sharma, A. (2024) · The International Spectator · doi:10.1080/03932729.2024.2435176 · Cited by 5
Writing in Symbiosis: Mapping Human Creative Agency in the AI EraPreprint
Doshi, V., & Li, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.13697 · Cited by 7
“This stylistic convergence has raised concerns about linguistic diversity erosion and cultural marginalization [7, 14].”
The impacts of artificial intelligence on environmental sustainability and human well-beingPreprint
Carmeno, N. L., Domingos, T., & O'Neill, D. W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.24091 · Cited by 6
“On inequality, AI may improve service accessibility, yet AI systems have been shown to reproduce and amplify social biases across gender, sexual orientation, ethnicity, age, and disability status, including in access to work and essential services [60–63].”
Future-proofing the EU: ordoliberal governance and algorithmic regulation
Küsters, A. (2025) · Constitutional Political Economy · doi:10.1007/s10602-025-09467-2 · Cited by 3
Huang, M., Rodriguez, V. J., Saha, K., & August, T. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791091 · Cited by 8
Subjective Intelligence: A Framework for Generative AI in STEM Education
Perez, G. A., Henderson, T. S., Pierre, T., Marvez, G. R., Vasquez, A., Eshun, P., & Pino, Y. P. (2025) · Education Sciences · doi:10.3390/educsci15121571 · Cited by 5
Reflexivity and positionality statements: a way to tackle “second-level arbitrariness” bias in AI?
Conti, L. G., & Seele, P. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06208-6 · Cited by 4
“…only produce “ incorrect, hallucinated and intentionally misleading content ” (Nature Machine Intelligence, 2024) but also perpetuate and even amplify human biases and prejudices such as those related to race and gender (Hofmann et al., 2024) – a phenomenon often referred to as algorithmic bias.”
Chronis, G. (2025) · AI & Society · doi:10.1007/s00146-025-02463-y · Cited by 6
Under Surveillance: Campus Protest, Generative AI, and the Erosion of Student Expressive Agency
Hall, E. (2025) · Critical AI · doi:10.1215/2834703x-12095955 · Cited by 3
Autenrieth, D., Autenrieth, N., & Farsani, D. (2026) · Frontiers in Language Sciences · doi:10.3389/flang.2026.1753250 · Cited by 3
“There is also the risk of reproducing biases present in the training data; models may subconsciously associate certain dialects with lower academic competence (Hofmann et al., 2024; Bui et al., 2025).”
Wise, A. F., Gresalfi, M. S., & Spencer-Smith, J. B. (2026) · International Journal of Qualitative Methods · doi:10.1177/16094069261435579 · Cited by 6
Poulos, N. S., Price, H., Bell, L., Byrd-Williams, C. E., Torres-Peralta, S., & Marty, E. (2026) · Prevention Science · doi:10.1007/s11121-026-01904-4 · Cited by 4
“…for biased coding, particularly for under-classification or misclassification of topics related to racial justice and equity (Fang et al., 2024; Hofmann et al., 2024), yet this may have been mitigated by the public health practitioners’ involvement in the data collection and equity focus of…”
Zhu, Y., Jia, Y., Zhu, Y., & Ye, J. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70079 · Cited by 4
Demystifying AI: The urgency of a critical stance on the use of AI systems in education
Heeg, D. M., & Avraamidou, L. (2026) · Prospects · doi:10.1007/s11125-026-09760-4 · Cited by 6
Tracing the Pen: Electronic Health Records Amid the Rise of Generative AI
Nargesi, A. A., You, J. G.-T., Bitterman, D. S., Succi, M. D., Mishuris, R. G., Poon, E. G., & Landman, A. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02508-6 · Cited by 4
“Moreover, covert AI biases such as dialect prejudice are more challenging to alleviate than the overt stereotypes 60 .”
Magalhães, P. C., Arnesen, S., Kern, C. A., König, P. D., Schiff, D. S., & Tyler, T. R. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103398 · Cited by 4
Kim, J., Ahn, J., & Sung, Y. (2026) · Online Information Review · doi:10.1108/oir-07-2025-0550 · Cited by 4
Governance of AI and by AI : Feedback Loops, Regime Variation and Reflexive Polycentric Control
Kleiner, J. (2026) · Global Policy · doi:10.1111/1758-5899.70177 · Cited by 4
Tracing Stereotypes in Pre-trained Transformers: From Biased Neurons to Fairer Models
Voria, G., Openja, M., Khomh, F., Catolino, G., & Palomba, F. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793356 · Cited by 3
Song, J. Y., Rojas, C., & Pycha, A. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1520111 · Cited by 3
My LLM might Mimic AAE - But When Should It?
Sandoval, S., Acquaye, C., Cobbina, K. A., Teli, M. N., & Daumé, H. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.273 · Cited by 11
Linked Data Workflows for Community Collections: Experiments with Open Access AI
Fenlon, K., Havens, L., Marsh, D. E., Wise, N. M., Smoke, U., Navarrete, C., Sioui, J., Mantle, D., & Sorensen, A. H. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1246 · Cited by 3
“Lack of diverse representation in ChatGPT’s training data already causes biased results (Hofmann et al., 2024) and ingesting these historic documents might perpetuate and reproduce such biases further, potentially “skew[ing] data of primary sources over time” (DC1).”
Ask ChatGPT: caveats and mitigations for individual users of AI chatbots
Wang, C., & Kantarcıoğlu, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01089-0 · Cited by 3
Examining Multimodal Gender and Content Bias in ChatGPT-4o
Balestri, R. (2024) · Advances in Internet of Things & Embedded Systems · doi:10.5121/csit.2024.142306 · Cited by 5
Hafner, F. S., Hafner, L., & Corizzo, R. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00382-y · Cited by 3
“…who says ‘I am so happy when I wake up from a bad dream because they feel so real’ high on ‘brilliance’ and ‘intelligence’, they rank a person who says the same in AAE dialect (‘I be so happy when I wake up from a bad dream cus they be feelin too real’) high on ‘dirtiness’ and ‘laziness’ [58].”
Maronikolakis, A., Ramallo, A. P., Cheng, W., & Kober, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.08907 · Cited by 4
Hakimov, R., Rohner, D., & Boboshin, F. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5386746 · Cited by 4
Interdisciplinary narratives on artificial intelligence & personnel selection systems
Boyd, N. M., Hunter, J., Intindola, M. L., & Serra, T. (2025) · The International Journal of Human Resource Management · doi:10.1080/09585192.2025.2568782 · Cited by 4
“Along those lines, one recurring theme is racist content such as in conversational tools expressing excludent and discriminative language from Tay in 2016 (Schwartz, 2019) to Grok in 2025 (Hagen et al., 2025) as well as in sentiment analysis tools producing negative associations with African American English (Hofmann…”
Birhane, A. (2026) · doi:10.21428/e2759450.50c7e139 · Cited by 5
Nayeem, M. T., & Rafiei, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.04204 · Cited by 3
“English variation in NLP and LLMs. Research on African American English (AAE), South Asian Englishes (SAsE), and other English varieties has shown systematic disparities in tagging, classification, sentiment analysis, robustness, and model behavior (Jørgensen et al., 2016; Blodgett et al., 2016; Kiritchenko & Mohammad…”
InfluentialPompedda, F., Santtila, P. O., Maso, E. D., Nyman, T. J., Sun, Y., & Zappalà, A. (2025) · Journal of Psychology and AI · doi:10.1080/29974100.2025.2503343 · Cited by 7
Shen, J., Huang, R., Min, D., Che, H., Li, B., Liu, L., Zhang, Z., Cheng, J., & Wang, S. (2025) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/tb-2025-0185 · Cited by 2
Mining Impersonification Bias in LLMs via Survey Filling
Bombieri, M., & Rospocher, M. (2025) · Information · doi:10.3390/info16110931 · Cited by 3
“Modern LLMs are indeed more than mere linguistic engines: they may implicitly encode and reproduce cultural priors, social norms, and biases present in their training data [3–5].”
Fano-Méndez, S., & Gutiérrez, A. F. (2025) · Revista Española de Educación Comparada · doi:10.5944/reec.48.2025.45369 · Cited by 4
EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models
Gupta, A., Cheung, J., Meng, P., Sayyed, S., Zhu, K., Liao, A., & O’Brien, S. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.913 · Cited by 12
How LLMs Handle Cultural Bias: Reactions to Asian Minority Historical Narratives
Shujaa, S., Dorleon, G., & Tang, A. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4861-3_3 · Cited by 5
Socially prescriptive speech technologies: Linguistic, technical, and ethical issues
Holliday, N. R. (2025) · The Journal of the Acoustical Society of America · doi:10.1121/10.0039685 · Cited by 3
Cash or Comfort? How LLMs Value Your Inconvenience
Cedro, M., Ichmoukhamedov, T., Goethals, S., He, Y., Hinns, J., & Martens, D. (2026) · Communications of the ACM · doi:10.1145/3799434 · Cited by 3
“This mirrors a number of concerns in prior work that LLM responses may be sensitive to user attributes such as gender, race, or dialect [29, 30, 31] and are more aligned to Western cultural norms [32, 33].”
Metadata
- Authors
- Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S.
- Affiliations
- University of Oxford (GB)
- Stanford University (US)
- University of Chicago (US)
- Corresponding author
- Hofmann, Valentin
- King, Sharese
- Venue
- Nature
- Published
- 2024-08-28
- Volume
- 633
- Issue
- 8028
- Pages
- 147-154
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC11374696
- OpenAlex
- W4401947350
- Cited by
- 233
- References
- 89
- Citations by year
- 2026: 116 · 2025: 111 · 2024: 6
- FWCI
- 96.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- University of Chicago (cc-by)
- Funders
- Stanford Institute for Human-Centered Artificial Intelligence, Stanford University
- Studienstiftung des Deutschen Volkes
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Linguistics and Language › Linguistic Variation and Morphology
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- covert racism, African American English, language models, racial bias in AI, preference alignment, hiring decisions, algorithmic fairness
- MeSH terms
- Artificial Intelligence, Black or African American, Decision Making, Humans, Language, Natural Language Processing, Stereotyping, Racism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.