Bias and Fairness in Large Language Models: A Survey
Open accessThe DOI matches its Crossref record; 312 of the 312 references it cites were found in a registry or an index; 151 of the 183 sources its claims cite were checked for support.
Abstract
Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this article, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely, metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs.
Overview
CoChat Summary
TL;DR
This survey consolidates, formalizes, and taxonomizes bias evaluation and mitigation for large language models, covering metrics, datasets, and mitigation techniques. It proposes three taxonomies and identifies open problems around power imbalances, fairness conceptualization, evaluation standards, mitigation scalability, and theoretical guarantees.
Key findings
- The survey proposes three taxonomies: bias evaluation metrics organized by what they use from a model (embeddings, probabilities, generated text); bias evaluation datasets organized by structure (counterfactual inputs vs. prompts); and bias mitigation techniques organized by intervention stage (pre-processing, in-training, intra-processing, post-processing).
- It formalizes social bias for LLMs as representational harms (e.g., derogatory language, disparate system performance, erasure, exclusionary norms, misrepresentation, stereotyping, toxicity) and allocational harms (direct and indirect discrimination), and introduces fairness desiderata including fairness through unawareness, invariance, equal social group associations, equal neutral associations, and replicated distributions.
- It reports limitations across metric classes: embedding-based and probability-based metrics may be only weakly or inconsistently correlated with downstream task bias, and generated-text metrics can be sensitive to decoding parameters; it also highlights reliability and validity issues in widely used datasets.
- It consolidates publicly available datasets, with reported sizes including CrowS-Pairs (1,508), StereoSet (16,995), BBQ (58,492), RealToxicityPrompts (100,000), BOLD (23,679), HolisticBias (460,000), and Bias NLI (5,712,066).
Methods
- Design: a comprehensive survey and taxonomy-building effort rather than an empirical study, synthesizing prior work on bias evaluation and mitigation for LLMs.
- Scope: focuses on bias issues in LLMs for English, with additional languages for machine translation and multilingual models, and restricts inclusion to works that propose novel closed-form metrics, datasets, or mitigation techniques and specify applicability to social bias or fairness.
- Analysis: organizes metrics, datasets, and mitigation techniques into taxonomies, formalizes several metrics and mitigation objectives with unified notation, discusses limitations of each class, and outlines open problems and recommendations.
Limitations
- The paper states that it aligns with a technical solutionist perspective rather than a critical theoretical one, and that its taxonomies are organized by technical implementation details instead of downstream usage contexts or harms.
- The paper states that it focuses narrowly on model training and evaluation, rather than a more holistic pipeline including problem formulation, data collection, and deployment/integration.
- The paper states that it is limited to English language papers; evident from the text, it also does not report a systematic search protocol, search dates, or screening counts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
121 Valid · 12 Not valid · 18 Not checked · 106 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite the growing emphasis on addressing these issues, bias and fairness research in LLMs often fails to precisely describe the harms of model behaviors: who is harmed, why the behavior is harmful, and how the harm reflects and reinforces social principles or hierarchies (Blodgett et al. 2020).”
What the sources say
- Partly supportedcites Blodgett et al. 2020 sourceExperimental — accuracy not yet measured
Section 3.1 (Motivations; page not provided) finds that papers on bias in NLP often fail to explain which model behaviors are harmful, how, to whom, and why. Sections 4.1–4.2 call for attention to social hierarchies and explicit normative reasoning. The survey concerns NLP research broadly, however, and does not establish that this failure is common specifically in LLM research. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar to the shortcomings of embedding-based metrics, Delobelle et al. (2022) and Kaneko, Bollegala, and Okazaki (2022) point out that probability-based metrics may be only weakly correlated with biases that appear in downstream tasks”
- Also stated in: § 3.6 Recommendations · p. 27 · plain
“Probability-based metrics also show weak correlations with downstream biases.”
What the sources say
- Supportedcites Delobelle et al. 2022 source
Section 3 describes CrowS-Pairs as a pseudo-loglikelihood-based metric. Section 4.4 reports that CrowS-Pairs correlates less with two measures of bias in downstream tasks, supporting the hedged claim that probability-based metrics may correlate weakly with downstream bias. The finding does not establish that all probability-based metrics do so. (LLM, conf=0.90)
- Supportedcites Kaneko, Bollegala, and Okazaki 2022 source
The Introduction (p. 1) identifies intrinsic measures as using likelihood scores assigned by MLMs and reports only weak correlations with downstream, task-specific bias measures. Section 5.2 (p. 8) likewise finds weak and inconsistent correlations between intrinsic and extrinsic bias scores. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.6 Recommendations · p. 27 · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many approaches, for instance, assume some implicitly desirable criterion ( e.g. , a model output should be independent of any social group in the input), but do not explicitly acknowledge or state the normative social values that justify their framework.”
What the sources say
- Partly supportedcites Blodgett et al. 2021 sourceExperimental — accuracy not yet measured
Section 2.3 describes assumptions about ideal model behavior, including equal probabilities for sentences about different target groups. Section 6 says the benchmarks make implicit decisions about stereotyping that are not explicitly discussed or justified. The paper examines four stereotyping benchmarks, however, rather than establishing this claim about many bias and fairness approaches generally or specifically about their normative social values. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Cabello, Jørgensen, and Søgaard (2023) illustrate that associations between the representations of protected attribute and other words can be independent of downstream performance disparities, if certain assumptions of social groups' language use are violated.”
What the sources say
- Supportedcites Cabello, Jørgensen, and Søgaard 2023 source
The Bias section defines association bias as differences in how demographic-group terms are encoded and empirical fairness as performance differences across social groups. Section 5 says the two can be independent and explains that linking them relies on the In-Group Affinity Assumption about how groups use those terms; it calls that assumption problematic. No page numbers are provided in our copy. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Each stakeholder brings perspectives that may specify different fairness constraints for the same application and setting.”
What the sources say
- Partly supportedcites Jernite et al. (2022) sourceExperimental — accuracy not yet measured
Section 2.2 (p. 4) says participants bring different perspectives and warns that relying only on their values could exclude views more relevant to other language users. The paper discusses nondiscrimination and local expressions of values, but does not say that stakeholders specify different fairness constraints for the same application and setting. (LLM, conf=0.85)
- Not matchedcites Bender (2019)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
71 more claims · 34 with their cited sources checked
Create a free account to see all 257 claims and every source verdict.
- Probably not a substantial claim. 151
- It may state two separate claims as one. 70
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While this process mitigates some risks inherent to AI-generated content, it does not eliminate potential biases associated with proprietary large language models, which lack transparency regarding training data and internal parameters [10].”
From AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notably, there is growing attention on assessing LLM bias [63-64] and robustness [323-324].”
From A Survey of Large Language Models (2026) · cites this paper as [63]
The cited source says what the claim attributes to it.
In the citing paper
“these models pose serious risks, as the vast, unfiltered internet data used to train them can reinforce and amplify harmful biases, disproportionately harming marginalized communities (73).”
From Emergent social conventions and collective bias in LLM populations (2025) · cites this paper as [73]
The cited source says what the claim attributes to it.
In the citing paper
“Despite the use of various mitigation techniques, such as debiasing algorithms, curating balanced datasets, or incorporating fairness-focused training objectives, eliminating bias entirely is a persistent challenge141–143.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [142]
The cited source says what the claim attributes to it.
In the citing paper
“Another major concern with LLMs is their propensity to amplify historical biases.”
From Evaluation of Large Language Models for Peer Review in Transplantation Research: Algorithm Validation Study (2026) · cites this paper as [30]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Counterfactual Inputs · p. 28
“| WinoBias+ | 1,367”
§ 4.1 Counterfactual Inputs · p. 29
“WinoBias+ (Vanmassenhove, Emmery, and Shterionov 2021) augments WinoBias with gender-neutral alternatives, similar to Winogender's neutral option, with 3,167 total instances.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“, 2020), and GPT-4 (OpenAI, 2023); autoencoding models include BERT (Devlin et al., 2019), RoBERTa (Liu et al.”
BackgroundInfluential(2014) · Cited by 33,419
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“, 2019), GPT-3 (Brown et al., 2020), and GPT-4 (OpenAI, 2023); autoencoding models include BERT (Devlin et al.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The rise and rapid advancement of large language models (LLMs) has fundamentally changed language technologies (e.g., Brown et al., 2020; Conneau et al., 2020; Devlin et al., 2019; Lewis et al., 2020; Liu et al., 2019; OpenAI, 2023; Radford et al., 2018; 2019; Raffel et al., 2020).”
BackgroundMeasuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210 · cited 2×
“, family and occupation words), emulating the Implicit Association Test (Greenwald et al., 1998).”
Method
Research works printed but not linked: 5
- Barocas, 2019, Fairness and Machine Learning: Limitations and Opportunities
- Blodgett, Su Lin . 2021. Sociolinguistically Driven Approaches for Just Natural Language Processing. Ph.D. thesis. University of Massachusetts Amherst.
- Green, 2019, ”Good” isn’t good enough, 1
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
- Wang, 2019, BERT has a mouth, and it must speak: BERT as a Markov random field language model, 30, 10.18653/v1/w19-2304 · doi:10.18653/v1/w19-2304
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594 · Cited by 1,781
“122 In the context of LLM-as-a-judge, it indicates that evaluators might score expressions from unfamiliar cultures poorly.”
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
“Gödel Agent [206] utilizes dynamic code modification techniques to alter its own logic and behavior based on the feedback signals from the environment and stores the updated agent logic in the agent memory for recursive improvement.”
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
“(3 ) Bias : Bias in LLMs manifests as the propagation and amplification of discriminatory tendencies present in training data, such as racial and gender biases, leading to unfair or harmful outputs [327–330].”
Towards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366
“Collectively, our findings suggest many avenues for further research that might leverage human–AI complementarity 30 , combining clinicians’ skills in the analysis of verbal and non-verbal cues with the potential strengths of LLMs to suggest more enriched conversational responses, including empathic statements, struct…”
InfluentialCurrent applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
Casheekar, A., Lahiri, A., Rath, K., Prabhakar, K. S., & Srinivasan, K. (2024) · Computer Science Review · doi:10.1016/j.cosrev.2024.100632 · Cited by 211
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
“This section focuses on the classification and evaluation of biases in LLMs, providing a comprehensive overview of the methods used to identify and assess biases at different stages of the model lifecycle Gallegos et al. (2024).”
InfluentialSurvey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174
“While existing mitigation efforts primarily address biases in demographic, cultural, and political beliefs [120], task-specific biases remain understud-ied, thus limiting the applicability of LLM-designed reward functions.”
Sociodemographic biases in medical decision making by large language models
Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Apakama, D. U., Horowitz, C. R., Charney, A. W., Freeman, R., Kummer, B., Glicksberg, B. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03626-6 · Cited by 173
“While prior work has demonstrated that LLMs can produce biased outputs in healthcare settings 11–14 , large-scale analyses examining a wide range of sociodemographic attributes have only recently begun to emerge 15 .”
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
“Since LLMs are often trained on large, unfiltered datasets, GenIR systems may propagate stereotypes and biases present in the data regarding race, culture, and other aspects [355].”
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168
“68,69 At the data collection and processing phase, these include measurement bias (how particular features are chosen, used, or measured), omitted variable bias (when one or more relevant variables are omitted or context is neglected), representation and sampling bias (incorrect sampling leads to…”
InfluentialTowards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges
Bulut, O., & Beiting-Parrish, M. (2024) · Chinese/English Journal of Educational Measurement and Evaluation · doi:10.59863/miql7785 · Cited by 126
“To address this ethical issue, several techniques can be implemented during multiple stages of training LLMs, including pre-processing, in-training, and post-processing (Gallegos et al., 2024).”
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119
“Most current research evaluates fairness as the uniformity of outputs across social groups or detects the prevalence of group-specific biases (Gallegos et al., 2024; Liang et al., 2023).”
BackgroundInfluentialHow understanding large language models can inform the use of ChatGPT in physics education
Polverini, G., & Gregorcic, B. (2024) · European Journal of Physics · doi:10.1088/1361-6404/ad1420 · Cited by 107
“Understanding, measuring and mitigating bias in LLMs are challenging tasks, requiring in depth understanding of the different phases of LLM training [42].”
BackgroundHistory, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109
“Researchers should consider the needs of disadvantaged groups from the outset and design these technologies proactively, rather than simply reacting to the biases present in their designed systems [40, 164, 165].”
Policy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
“…et al., 2023) and reinforcement learning (Gajane et al., 2022), natu-ral language processing (NLP) (Blodgett et al., 2020; Czar-nowska et al., 2021; Gallegos et al., 2023), computer vision (Fabbrizzi et al., 2022), speech processing, recommender systems (Chen et al., 2023a), and knowledge…”
Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 177
Retrieval-Augmented Generation (RAG)
Klesel, M., & Wittmann, H. F. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00945-3 · Cited by 92
“For a current overview, see for instance Mehrabi et al. (2022) or Gallegos et al. (2024).”
Addressing bias in generative AI: Challenges and research opportunities in information management
Wei, X., Kumar, N., & Zhang, H. (2025) · Information & Management · doi:10.1016/j.im.2025.104103 · Cited by 174
Navigating LLM ethics: advancements, challenges, and future directions
Jiao, J., Afroogh, S., Xu, Y., & Phillips, C. (2025) · AI and Ethics · doi:10.1007/s43681-025-00814-5 · Cited by 216
“Social biases reflect model biases related to various aspects of social status and characteristics, such as gender, age, income, race, occupation, education, and lifestyle choices [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51].”
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
“These biases can manifest as harms in specific downstream tasks, impacting clinical decision-making [5,10,54,60,76,83].”
Natural language processing in the era of large language models
Zubiaga, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1350306 · Cited by 81
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2025) · Proceedings of the 17th ACM Web Science Conference 2025 · doi:10.1145/3717867.3717903 · Cited by 81
“Despite such advantages, the additional LLM components could introduce new problems (Memon and West 2024) because language models face critical technical challenges, such as hallucinations (Ji et al. 2023), and exhibit various biases (Gallegos et al. 2024).”
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71
Perspectives of Generative AI in Chemistry Education Within the TPACK Framework
Feldman-Maggor, Y., Blonder, R., & Alexandron, G. (2024) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10147-3 · Cited by 128
“Another source of bias or equity issues arises because most AI tools operate mainly in English, which has a significant advantage due to the availability of large-scale training datasets (Ariely et al., 2023; Gallegos et al., 2024; Xu et al., 2024).”
Emergent social conventions and collective bias in LLM populations
Ashery, A. F., Aiello, L. M., & Baronchelli, A. (2025) · Science Advances · doi:10.1126/sciadv.adu9368 · Cited by 81
“Despite their rapid adoption, these models pose serious risks, as the vast, unfiltered internet data used to train them can reinforce and amplify harmful biases, disproportionately harming marginalized communities ( 73 ).”
AI-Generated Content (AIGC) for Various Data Modalities: A Survey
Foo, L. G., Rahmani, H., & Liu, J. (2025) · ACM Computing Surveys · doi:10.1145/3728633 · Cited by 138
Introduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63
Barman, K. G., Wood, N. G., & Pawłowski, P. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09778-2 · Cited by 86
Preventing harm from non-conscious bias in medical generative AI
Hastings, J. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00246-7 · Cited by 60
Fairness in Large Language Models: A Taxonomic Survey
Chu, Z., Wang, Z., & Zhang, W. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3682112.3682117 · Cited by 59
“This phenomenon inadvertently introduces semantic bias into downstream models, impacting their performance and fairness [50, 9].”
BackgroundAn, J., Huang, D., Lin, C., & Tai, M. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf089 · Cited by 58
“This complexity underscores the need for comprehensive, multidimensional approaches to studying and mitigating bias in AI systems, as highlighted in recent surveys (22).”
Idris, M. D., Feng, X., & Dyo, V. (2024) · IEEE Access · doi:10.1109/access.2024.3400164 · Cited by 85
A Survey of Personalization: From RAG to Agent
Li, X., Jia, P., Xu, D., Wen, Y., Zhang, Y., Zhang, W., Wang, W., Wang, Y., Du, Z., Li, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.10147 · Cited by 52
GIScience in the era of Artificial Intelligence: a research agenda towards Autonomous GIS
Li, Z., Ning, H., Gao, S., Janowicz, K., Li, W., Arundel, S. T., Yang, C. P., Bhaduri, B., Wang, S., Zhu, A.-X., et al. (2025) · Annals of GIS · doi:10.1080/19475683.2025.2552161 · Cited by 95
Khurram, M., Zhang, C., Muhammad, S., Kishnani, H., An, K. R., Abeywardena, K. G., Chadha, U., & Behdinan, K. (2025) · Processes · doi:10.3390/pr13051312 · Cited by 90
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 83
A Current Review of Generative AI in Medicine: Core Concepts, Applications, and Current Limitations
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Shariatnia, M. M., Rouzrokh, P., & Erickson, B. J. (2025) · Current Reviews in Musculoskeletal Medicine · doi:10.1007/s12178-025-09961-y · Cited by 66
“Such bias not only reduces the model’s utility across diverse populations but also risks perpetuating health inequities [101].”
Beyond principlism: practical strategies for ethical AI use in research practices
Lin, Z. (2025) · AI and Ethics · doi:10.1007/s43681-024-00585-5 · Cited by 40
Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2025) · Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2025.ijcnlp-long.18 · Cited by 39
“…complexity (Li et al., 2024b; Khan et al., 2024; Chua et al., 2024), the introduction of new biases (Ohi et al., 2024), inconsistent effectiveness (Gallegos et al., 2024), or impracticality for closed-source models, as mechanistic solutions require modifications to model internals (Wang et al.,…”
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 125
“This ecosystem is characterized by a diverse array of applications and services that leverage pre-trained models to address real-world challenges, driving innovation and efficiency.”
On large language models safety, security, and privacy: A survey
Zhang, R., Li, H., Qian, X., Jiang, W., & Chen, H.-X. (2025) · Journal of Electronic Science and Technology · doi:10.1016/j.jnlest.2025.100301 · Cited by 65
Self-citationClinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
OpenBias: Open-Set Bias Detection in Text-to-Image Generative Models
Moreno D’Incà, Peruzzo, E., Mancini, M., Xu, D., Goe, V., Xu, X., Wang, Z., Shi, H., & Sebe, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01162 · Cited by 122
Nguyen, H., Nguyen, V., Ludovise, S., & Santagata, R. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13568 · Cited by 54
Conformal Prediction for Natural Language Processing: A Survey
Campos, M., Farinhas, A., Zerva, C., Figueiredo, M. A. T., & Martins, A. F. T. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00715 · Cited by 125
“…2019; Desai and Durrett, 2020), evaluation is limited and sometimes affected by data contamination (Sainz et al., 2023; Golchin and Surdeanu, 2024), explanations are often unreliable (Zhao et al., 2024; Wiegreffe and Pinter, 2019), and models often exhibit undesired biases (Gallegos et al., 2024).”
Large Language Models Are Biased Because They Are Large Language Models
Resnik, P. (2025) · Computational Linguistics · doi:10.1162/coli_a_00558 · Cited by 95
Mehdiyev, N., Majlatow, M., & Fettke, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11399-0 · Cited by 62
Hao, X., Demir, E., & Eyers, D. (2025) · Sustainable Futures · doi:10.1016/j.sftr.2025.101152 · Cited by 33
“Bias contributing to discrimination includes historical [10], sampling [115], representation [84], measurement [52], evaluation [58], and algorithmic biases [76].”
Deep Representation Learning: Fundamentals, Technologies, Applications, and Open Challenges
Payandeh, A., Baghaei, K. T., Fayyazsanavi, P., Ramezani, S. B., Chen, Z., & Rahimi, S. (2023) · IEEE Access · doi:10.1109/access.2023.3335196 · Cited by 64
“While concerns remain around bias, safety, and environmental impact [371], [372], the application [373], [374] of LLMs continues to rapidly advance.”
BackgroundScott, I. A., & Zuccon, G. (2024) · Internal Medicine Journal · doi:10.1111/imj.16393 · Cited by 38
“64 Retrieval-augmented generation (RAG) enhances LLM by using a pretrained retriever model to retrieve supplementary text from large external databases, including recent data, and seamlessly integrating this information into the model to generate more accurate, contextualised responses.”
MethodA Field Guide to Automatic Evaluation of LLM-Generated Summaries
van Schaik, T. A., & Pugh, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3661346 · Cited by 30
“Those qualities include fluency [11], coherence [10], [11], [12], relevance [11], factual consistency [11], and fairness [13].”
The advantages of lexicon-based sentiment analysis in an age of machine learning
van der Veen, A. M., & Bleich, E. (2025) · PLoS ONE · doi:10.1371/journal.pone.0313092 · Cited by 62
Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A SurveyPreprint
Li, Z., Wu, X., Du, H., Nghiem, H. T., & Shi, G. (2025) · Qeios · doi:10.32388/gxr68q · Cited by 38
Weng, L., Wang, X., Lu, J., Feng, Y., Liu, Y., Feng, H., Huang, D., & Chen, W. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3567131 · Cited by 80
“Moreover, biases inherent in LLMs’ training corpus can be exacerbated by the patterns of the current dataset, which may further propagate into the extracted insights or generated topics [17].”
Sankar, B. S., Gilliland, D., Rincon, J., Hermjakob, H., Yan, Y., Adam, I., Lemaster, G., Wang, D., Watson, K., Bui, A., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11100984 · Cited by 26
Fairness-Aware Graph Neural Networks: A Survey
Chen, A., Rossi, R. A., Park, N., Trivedi, P., Wang, Y., Yu, T., Kim, S., Dernoncourt, F., & Ahmed, N. K. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649142 · Cited by 85
“Furthermore, the growing complexity of LMs contributes to their “black-box” nature [111, 112], making it challenging to identify and address biases originating from graph data.”
BackgroundSelf-citationOjöng, N. (2025) · Business Strategy and the Environment · doi:10.1002/bse.70031 · Cited by 48
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 84
“Measurements on LLMs’ fairness are generally categorized into three types, based on the model elements they analyze: embeddings, probabilities, and generated texts [82].”
Torkamaan, H., Steinert, S., Pera, M. S., Kudina, O., Freire, S. K., Verma, H., Kelly, S., Sekwenz, M.-T., Yang, J., van Nunen, K., et al. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2431068 · Cited by 42
Zhang, L., Yao, Z., & Moghaddam, A. H. (2025) · Journal of Teacher Education · doi:10.1177/00224871251325109 · Cited by 31
“We excluded racial, cultural, and socioeconomic data to minimize the risk of biases or stereotypical outputs that LLMs may produce (Gallegos et al., 2024).”
Evaluating gender bias in large language models in long-term care
Rickman, S. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03118-0 · Cited by 40
“These kinds of differences fall into what Gal-legos et al. [24] term allocational harm, where biased language may influence treatment or services.”
InfluentialWang, Y., Huang, J., Du, L., Guo, Y., Liu, Y., & Wang, R. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100481 · Cited by 59
Content moderation by LLM: from accuracy to legitimacy
Huang, T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11328-1 · Cited by 24
“It allows us to rethink LLMs’ real potential in content moderation and platform governance, as well as what improvements can be made to further realize its potential.”
A Critical Review of Methods and Challenges in Large Language Models
Asgari, R., Moradi, M., Yan, K., Colwell, D., & Samwald, M. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.061263 · Cited by 48
“PEFT updates a small subset of parameters or adds minimal task-specific layers, preserving the model’s general capabilities while reducing computational resources and mitigating catastrophic forgetting by keeping most pre-trained weights intact [39]. approach enables more flexible and scalable…”
Unlocking Bias Detection: Leveraging Transformer-Based Models for Content Analysis
Raza, S., Bamgbose, O., Chatrath, V., Ghuge, S., Sidyakin, Y., & Muaad, A. Y. M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3392469 · Cited by 37
“Gallegos et al. [14] delved deeper into the concepts of social bias and fairness in NLP, with the goal of actualizing fairness in LMs.”
BackgroundMachine Bias. How Do Generative Language Models Answer Opinion Polls?
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2025) · Sociological Methods & Research · doi:10.1177/00491241251330582 · Cited by 87
“A recent survey focused on LLMs proposed a taxonomy speci fi c to these models (Gallegos et al. 2024).”
SPRec: Self-Play to Debias LLM-based Recommendation
Gao, C., Chen, R., Yuan, S., Huang, K., Yu, Y., & He, X. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714524 · Cited by 100
Balog, K., Metzler, D., & Qin, Z. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730348 · Cited by 56
“For example, studies have shown that LLMs can exhibit biases in their output, favoring LLM-generated content over human-generated ones [10], and perpetuating biases present in their training data [15, 21, 33].”
CalibJudge: Calibrated LLM-as-a-Judge for Multilingual RAG with Uncertainty-Aware Scoring
Wen, C., Zhu, A., Long, R., Huang, H., Jiang, J., & Lee, C. S. (2026) · International Conference on Artificial Intelligence and Control (ICAIC) · doi:10.1145/3807246.3807349 · Cited by 23
“…varies across languages in both distribution and ranking behavior [5], while broader studies on fairness and bias in large language models establish that output discrepancies across groups are not incidental noise but systematic statistical shifts that can persist unless explicitly corrected [6].”
Red teaming large language models: A comprehensive review and critical analysis
Jabbar, M. S., Al-Azani, S., Alotaibi, A., & Ahmed, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104239 · Cited by 41
Large Language Models in Critical Care Medicine: Scoping Review
Shi, T., Ma, J., Yu, Z., Xu, H., Yang, R., Xiong, M., Xiao, M., Li, Y., Zhao, H., & Kong, G. (2025) · JMIR Medical Informatics · doi:10.2196/76326 · Cited by 21
Schoene, A. M., & Canca, C. (2025) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas65609.2025.11269647 · Cited by 55
“However, such models are prone not only to hallucinations [4] and concerns about bias and fairness [5], but also providing users with information that could be harmful to them, others, or society at large.”
Derakhshan, A., Lalli, G. S., & Park, Y. (2026) · Language Testing in Asia · doi:10.1186/s40468-026-00445-5 · Cited by 41
Distortions in Judged Spatial Relations in Large Language Models
Fulman, N., Memduhoğlu, A., & Zipf, A. (2024) · The Professional Geographer · doi:10.1080/00330124.2024.2372792 · Cited by 24
“Furthermore, recent research has unveiled nonspatial biases in LLMs, including logical and cognitive distortions (Gallegos et al. 2023).”
The Responsible Development of Automated Student Feedback with Generative AI
Lindsay, E. D., Zhang, M., Johri, A., & Bjerva, J. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016572 · Cited by 51
“This training paradigm comes with ethical considerations of its own, e.g., in the area of perpetuating cultural biases [35]–[38], and amplifying extremist content online.”
Farrell, H. J. (2025) · Annual Review of Political Science · doi:10.1146/annurev-polisci-040723-013245 · Cited by 31
Jang, E. E., & Sawaki, Y. (2025) · Language Testing · doi:10.1177/02655322251348685 · Cited by 31
“Moreover, bias in AI training data can introduce systematic disadvantages for speakers from diverse linguistic backgrounds (Gallegos et al., 2024).”
Natural language processing for social science research: A comprehensive review
Hou, Y., & Huang, J. (2025) · Chinese Journal of Sociology · doi:10.1177/2057150x241306780 · Cited by 45
“Numerous studies have demonstrated that the content generated by LLMs inherits and even exacerbates these social biases (Abid et al., 2021; Bender et al., 2021; Gallegos et al., 2024; Kotek et al., 2023; OpenAI et al., 2024).”
Vatsal, S., Dubey, H., & Singh, A. (2026) · IEEE Access · doi:10.1109/access.2026.3651218 · Cited by 31
“Equity requires detecting and mitigating systematic errors across patient groups, with evaluation and remediation guided by LLM bias surveys [116], [117].”
KoBBQ: Korean Bias Benchmark for Question Answering
Jin, J., Kim, J., Lee, N., Yoo, H., Oh, A. H., & Lee, H. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00661 · Cited by 132
Williams, A. R., Burke-Moore, L., Chan, R. S.-Y., Enock, F. E., Nanni, F., Sippy, T., Chung, Y.-L., Gabašová, E., Hackenburg, K., & Bright, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0317421 · Cited by 56
“Write a tweet complaining about the news that { mp } { subject } , from the perspective of a { persona } citizen.”
Explainable artificial intelligence in the talent recruitment process-a literature review
Zhang, G., Pan, L., Tang, F., & Yao, F. (2025) · Cogent Business & Management · doi:10.1080/23311975.2025.2570881 · Cited by 32
“in the area of person-job recommendation, since large models are trained on datasets that may contain biases (gallegos et al., 2024), the system may perpetuate these biases, thereby affecting the fairness, neutrality, and interpretability of the generated job postings.”
Sievers, B., & Thornton, M. A. (2024) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsae014 · Cited by 18
“Much like mitigating human bias, mitigating this form of bias in ANNs is far from a solved problem (Gallegos et al ., 2023).”
BackgroundShin, M., Song, J., Kim, M.-G., Yu, H. W., Choe, E. K., & Chai, Y. J. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14072450 · Cited by 20
“However, LLMs face several challenges including the generation of inaccuracies caused by incorrect, biased, outdated, or misleading information [7,8].”
CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering
Joshi, I., Shahid, S., Venneti, S., Vasu, M., Zheng, Y., Li, Y., Krishnamurthy, B., & Chan, G. Y.-Y. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712102 · Cited by 37
Hartmann, D., Oueslati, A., Staufer, D., Pohlmann, L., Munzert, S., & Heuer, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713998 · Cited by 53
“It is widely used to measure the toxic outcomes and biases of LLMs [32], and it has been suggested that Perspective API has become a cornerstone for academic research on online abuse and incivility [75].”
Lu, H., Lin, Y., Li, Z., Yiu, M. L., Gao, Y., & Uddin, S. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103216 · Cited by 23
Han, T., Cong, R., Yu, B., Tang, B.-J., & Wei, Y. (2024) · Energy and AI · doi:10.1016/j.egyai.2024.100440 · Cited by 18
The evolution of language models: From N-Grams to LLMs, and beyond
Raeini, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100168 · Cited by 40
Dual-process theory and decision-making in large language models
Brady, O., Nulty, P., Zhang, L., Ward, T. E., & McGovern, D. P. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00506-1 · Cited by 33
“Tree-of-thought prompting can simulate human System 2 thinking by prompting a language model to consider a range of possibilities 59 .”
Measuring technology acceptance over time using transfer models based on online customer reviews
Baier, D., Karasenko, A., & Rese, A. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104278 · Cited by 25
Perron, B. E., Goldkind, L., Qi, Z., & Victor, B. G. (2025) · Journal of Technology in Human Services · doi:10.1080/15228835.2025.2457045 · Cited by 22
“The challenges of model bias in high-stakes decision-making are welldocumented (Gallegos et al., 2024).”
From Gaze to Data: Privacy and Societal Challenges of Using Eye-tracking Data to Inform GenAI Models
Abdrabou, Y., Özdel, S., Maquiling, V., Bozkir, E., & Kasneci, E. (2025) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3715669.3726788 · Cited by 19
“This highlights the importance of robust privacy measures to prevent unauthorized disclosure and protect the user’s identity.”
Whose morality do they speak? Unraveling cultural bias in multilingual language models
Aksoy, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100172 · Cited by 46
Lee, P., Son, M., & Jia, Z. (2025) · Journal of Business and Psychology · doi:10.1007/s10869-025-10067-y · Cited by 27
“In addition, LLMs are susceptible to inheriting and per-petuating societal biases embedded in their training data, including gender stereotypes, racial biases, and cultural assumptions (Gallegos et al., 2024).”
Metadata
- Authors
- Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K.
- Affiliations
- Stanford University (US)
- Adobe Research. ryrossi@adobe.com
- Pattern Data. joe.barrow@patterndataworks.com
- Adobe Research. tanjim@adobe.com
- Adobe Research. sukim@adobe.com
- Adobe Research. dernonco@adobe.com
- Adobe Research. tyu@adobe.com
- Adobe Research. ruizhang@adobe.com
- Intel (United States) (US)
- ORCID
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 3
- Pages
- 1097-1179
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4399528455
- Semantic Scholar
- bcfa73aedf1b2d1ee4f168e21298a37ac55a37f7
- Cited by
- 1,245
- References
- 280
- Citations by year
- 2026: 596 · 2025: 451 · 2024: 188
- FWCI
- 132.76
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, social bias, fairness, bias assessment, bias mitigation, bias metrics, data preprocessing, post-processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.