The DOI matches its Crossref record; 308 of the 308 references it cites were found in a registry or an index; 151 of the 183 sources its claims cite were checked for support.
Bias and Fairness in Large Language Models: A Survey
Open accessAbstract
Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this article, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely, metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs.
Overview
CoChat Summary
TL;DR
This survey consolidates, formalizes, and taxonomizes bias evaluation and mitigation for large language models, covering metrics, datasets, and mitigation techniques. It proposes three taxonomies and identifies open problems around power imbalances, fairness conceptualization, evaluation standards, mitigation scalability, and theoretical guarantees.
Key findings
- The survey proposes three taxonomies: bias evaluation metrics organized by what they use from a model (embeddings, probabilities, generated text); bias evaluation datasets organized by structure (counterfactual inputs vs. prompts); and bias mitigation techniques organized by intervention stage (pre-processing, in-training, intra-processing, post-processing).
- It formalizes social bias for LLMs as representational harms (e.g., derogatory language, disparate system performance, erasure, exclusionary norms, misrepresentation, stereotyping, toxicity) and allocational harms (direct and indirect discrimination), and introduces fairness desiderata including fairness through unawareness, invariance, equal social group associations, equal neutral associations, and replicated distributions.
- It reports limitations across metric classes: embedding-based and probability-based metrics may be only weakly or inconsistently correlated with downstream task bias, and generated-text metrics can be sensitive to decoding parameters; it also highlights reliability and validity issues in widely used datasets.
- It consolidates publicly available datasets, with reported sizes including CrowS-Pairs (1,508), StereoSet (16,995), BBQ (58,492), RealToxicityPrompts (100,000), BOLD (23,679), HolisticBias (460,000), and Bias NLI (5,712,066).
Methods
- Design: a comprehensive survey and taxonomy-building effort rather than an empirical study, synthesizing prior work on bias evaluation and mitigation for LLMs.
- Scope: focuses on bias issues in LLMs for English, with additional languages for machine translation and multilingual models, and restricts inclusion to works that propose novel closed-form metrics, datasets, or mitigation techniques and specify applicability to social bias or fairness.
- Analysis: organizes metrics, datasets, and mitigation techniques into taxonomies, formalizes several metrics and mitigation objectives with unified notation, discusses limitations of each class, and outlines open problems and recommendations.
Limitations
- The paper states that it aligns with a technical solutionist perspective rather than a critical theoretical one, and that its taxonomies are organized by technical implementation details instead of downstream usage contexts or harms.
- The paper states that it focuses narrowly on model training and evaluation, rather than a more holistic pipeline including problem formulation, data collection, and deployment/integration.
- The paper states that it is limited to English language papers; evident from the text, it also does not report a systematic search protocol, search dates, or screening counts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
121 Valid · 12 Not valid · 18 Not checked · 106 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite the growing emphasis on addressing these issues, bias and fairness research in LLMs often fails to precisely describe the harms of model behaviors: who is harmed, why the behavior is harmful, and how the harm reflects and reinforces social principles or hierarchies (Blodgett et al. 2020).”
What the sources say
- Partly supportedcites Blodgett et al. 2020 sourceExperimental — accuracy not yet measured
Section 3.1 (Motivations; page not provided) finds that papers on bias in NLP often fail to explain which model behaviors are harmful, how, to whom, and why. Sections 4.1–4.2 call for attention to social hierarchies and explicit normative reasoning. The survey concerns NLP research broadly, however, and does not establish that this failure is common specifically in LLM research. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar to the shortcomings of embedding-based metrics, Delobelle et al. (2022) and Kaneko, Bollegala, and Okazaki (2022) point out that probability-based metrics may be only weakly correlated with biases that appear in downstream tasks”
- Also stated in: § 3.6 Recommendations · p. 27 · plain
“Probability-based metrics also show weak correlations with downstream biases.”
What the sources say
- Supportedcites Delobelle et al. 2022 source
Section 3 describes CrowS-Pairs as a pseudo-loglikelihood-based metric. Section 4.4 reports that CrowS-Pairs correlates less with two measures of bias in downstream tasks, supporting the hedged claim that probability-based metrics may correlate weakly with downstream bias. The finding does not establish that all probability-based metrics do so. (LLM, conf=0.90)
- Supportedcites Kaneko, Bollegala, and Okazaki 2022 source
The Introduction (p. 1) identifies intrinsic measures as using likelihood scores assigned by MLMs and reports only weak correlations with downstream, task-specific bias measures. Section 5.2 (p. 8) likewise finds weak and inconsistent correlations between intrinsic and extrinsic bias scores. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.6 Recommendations · p. 27 · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many approaches, for instance, assume some implicitly desirable criterion ( e.g. , a model output should be independent of any social group in the input), but do not explicitly acknowledge or state the normative social values that justify their framework.”
What the sources say
- Partly supportedcites Blodgett et al. 2021 sourceExperimental — accuracy not yet measured
Section 2.3 describes assumptions about ideal model behavior, including equal probabilities for sentences about different target groups. Section 6 says the benchmarks make implicit decisions about stereotyping that are not explicitly discussed or justified. The paper examines four stereotyping benchmarks, however, rather than establishing this claim about many bias and fairness approaches generally or specifically about their normative social values. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Cabello, Jørgensen, and Søgaard (2023) illustrate that associations between the representations of protected attribute and other words can be independent of downstream performance disparities, if certain assumptions of social groups' language use are violated.”
What the sources say
- Supportedcites Cabello, Jørgensen, and Søgaard 2023 source
The Bias section defines association bias as differences in how demographic-group terms are encoded and empirical fairness as performance differences across social groups. Section 5 says the two can be independent and explains that linking them relies on the In-Group Affinity Assumption about how groups use those terms; it calls that assumption problematic. No page numbers are provided in our copy. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Each stakeholder brings perspectives that may specify different fairness constraints for the same application and setting.”
What the sources say
- Partly supportedcites Jernite et al. (2022) sourceExperimental — accuracy not yet measured
Section 2.2 (p. 4) says participants bring different perspectives and warns that relying only on their values could exclude views more relevant to other language users. The paper discusses nondiscrimination and local expressions of values, but does not say that stakeholders specify different fairness constraints for the same application and setting. (LLM, conf=0.85)
- Not matchedcites Bender (2019)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
71 more claims · 34 with their cited sources checked
Create a free account to see all 257 claims and every source verdict.
- Probably not a substantial claim. 151
- It may state two separate claims as one. 70
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While this process mitigates some risks inherent to AI-generated content, it does not eliminate potential biases associated with proprietary large language models, which lack transparency regarding training data and internal parameters [10].”
From AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notably, there is growing attention on assessing LLM bias [63-64] and robustness [323-324].”
From A Survey of Large Language Models (2026) · cites this paper as [63]
The cited source says what the claim attributes to it.
In the citing paper
“these models pose serious risks, as the vast, unfiltered internet data used to train them can reinforce and amplify harmful biases, disproportionately harming marginalized communities (73).”
From Emergent social conventions and collective bias in LLM populations (2025) · cites this paper as [73]
The cited source says what the claim attributes to it.
In the citing paper
“Despite the use of various mitigation techniques, such as debiasing algorithms, curating balanced datasets, or incorporating fairness-focused training objectives, eliminating bias entirely is a persistent challenge141–143.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [142]
The cited source says what the claim attributes to it.
In the citing paper
“Another major concern with LLMs is their propensity to amplify historical biases.”
From Evaluation of Large Language Models for Peer Review in Transplantation Research: Algorithm Validation Study (2026) · cites this paper as [30]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Counterfactual Inputs · p. 28
“| WinoBias+ | 1,367”
§ 4.1 Counterfactual Inputs · p. 29
“WinoBias+ (Vanmassenhove, Emmery, and Shterionov 2021) augments WinoBias with gender-neutral alternatives, similar to Winogender's neutral option, with 3,167 total instances.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“, 2020), and GPT-4 (OpenAI, 2023); autoencoding models include BERT (Devlin et al., 2019), RoBERTa (Liu et al.”
BackgroundInfluential(2014) · Cited by 33,419
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“, 2019), GPT-3 (Brown et al., 2020), and GPT-4 (OpenAI, 2023); autoencoding models include BERT (Devlin et al.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The rise and rapid advancement of large language models (LLMs) has fundamentally changed language technologies (e.g., Brown et al., 2020; Conneau et al., 2020; Devlin et al., 2019; Lewis et al., 2020; Liu et al., 2019; OpenAI, 2023; Radford et al., 2018; 2019; Raffel et al., 2020).”
BackgroundMeasuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210 · cited 2×
“, family and occupation words), emulating the Implicit Association Test (Greenwald et al., 1998).”
MethodLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 4×
“, 2020); and encoder-decoder models include BART (Lewis et al., 2020) and T5 (Raffel et al.”
BackgroundInfluentialOvercoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · cited 1×
“This phenomenon is known as catastrophic forgetting (Kirkpatrick et al., 2017).”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“Typically trained on an enormous scale of uncurated Internet-based data, LLMs inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities (Bender et al., 2021; Dodge et al., 2021; Sheng et…”
BackgroundSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“For an invariance metric ψ such as exact match (Rajpurkar et al., 2016), and predicted outputs Ŷi from an original input and Ŷj from a counterfactual input, then:”
Backgroundmixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 2×
“While similar concepts have been operationalized for machine learning classification tasks (Mehrabi et al., 2021; Verma & Rubin, 2018), less has been done in the NLP space, which may contain more ambiguity than classification for tasks like language generation.”
BackgroundUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Even without fine-tuning, foundation models increasingly enable few- or zero-shot capabilities for a wide array of scenarios like classification, question-answering, logical reasoning, fact retrieval, information extraction, and more, with the task described in a natural language prompt to the model and few or no labe…”
BackgroundA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“While evaluation techniques for LLMs have been recently surveyed by Chang et al. (2023), they do not focus on the evaluation of fairness and bias in such models.”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 2×
“We also discuss individual fairness (Dwork et al., 2012).”
MethodA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03109 · Cited by 3,159 · cited 1×
“While evaluation techniques for LLMs have been recently surveyed by Chang et al. (2023), they do not focus on the evaluation of fairness and bias in such models.”
MethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“Continuous prefix or prompt tuning (Lester et al., 2021; Li & Liang, 2021; Liu et al., 2021c) modifies the input with a trainable prefix.”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 1×
“Continuous prefix or prompt tuning (Lester et al., 2021; Li & Liang, 2021; Liu et al., 2021c) modifies the input with a trainable prefix.”
BackgroundSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 14×
“Several of these methods follow WEAT’s formulation.”
MethodInfluentialModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 4×
“Instead of training task-specific models on relatively small task-specific datasets, researchers and practitioners can use LLMs as foundation models that can be fine-tuned for particular functions (Bommasani et al., 2021).”
BackgroundMethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Perspective API3, developed by Google Jigsaw is a toxicity detection tool widely used in the literature (Bommasani et al., 2023; Chung et al., 2022; Chowdhery et al., 2022; Gehman et al., 2020).”
MethodEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“The primary emphasis of bias evaluation and mitigation efforts for LLMs focus on group notions of fairness, which center on disparities between social groups, following group fairness definitions in the literature (Chouldechova, 2017; Hardt et al., 2016; Kamiran & Calders, 2012).”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474 · cited 2×
“Curation of additional language resources should value inclusivity over convenience, and documentation should follow practices such as Bender & Friedman (2018) and Gebru et al. (2021). Furthermore, stakeholders must ensure that the process of collecting data itself does not contribute to further harms.”
BackgroundPrefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 2×
“Curation of additional language resources should value inclusivity over convenience, and documentation should follow practices such as Bender & Friedman (2018) and Gebru et al. (2021). Furthermore, stakeholders must ensure that the process of collecting data itself does not contribute to further harms.”
BackgroundParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 1×
“The technique is based on modular adapter frameworks (Houlsby et al., 2019) that add new, randomly-initialized layers between the original layers for parameter-efficient fine-tuning; only the injected layers are updated during fine-tuning, while the pre-trained ones remain frozen.”
MethodFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 1×
“The primary emphasis of bias evaluation and mitigation efforts for LLMs focus on group notions of fairness, which center on disparities between social groups, following group fairness definitions in the literature (Chouldechova, 2017; Hardt et al., 2016; Kamiran & Calders, 2012).”
BackgroundLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · cited 1×
“Even without fine-tuning, foundation models increasingly enable few- or zero-shot capabilities for a wide array of scenarios like classification, question-answering, logical reasoning, fact retrieval, information extraction, and more, with the task described in a natural language prompt to the model and few or no labe…”
BackgroundLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 1×
“Even without fine-tuning, foundation models increasingly enable few- or zero-shot capabilities for a wide array of scenarios like classification, question-answering, logical reasoning, fact retrieval, information extraction, and more, with the task described in a natural language prompt to the model and few or no labe…”
BackgroundMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 2×
“Several works have presented word lists associated with social groups for gender (Bolukbasi et al., 2016; Garg et al., 2018; Gupta et al., 2022; Hall Maudslay et al., 2019; Lu et al., 2020; Zhao et al., 2017; 2018), race (Caliskan et al., 2017; Garg et al., 2018; Gupta et al., 2022; Manzini et al., 2019), age (Caliska…”
BackgroundSemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · cited 2×
“, 2020) adapts the original Semantic Textual Similarity-Benchmark (STS-B) (Cer et al., 2017) to generate pairs of sentences differing only in gendered terms, but otherwise maintaining the same meaning for sentences in a pair.”
BackgroundReview of “Race After Technology: Abolitionist Tools for the New Jim Code”
Jipguep-Akhtar, M. (2020) · Social Forces · doi:10.1093/sf/soz162 · Cited by 1,473
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 3×
“Chung et al. (2023) propose two decoding strategies to increase diversity of generated tokens. Logit suppression decreases the probability of generating already-used tokens from previous generations, which encourages the selection of lower-frequency tokens. Temperature sampling flattens the next-word probability distr…”
BackgroundMethodMitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · cited 1×
“This "pre-train, then fine-tune" paradigm enables the training of one foundation model that can be adapted to a range of applications (Bommasani et al., 2021; Min et al., 2023).”
BackgroundData preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · cited 2×
“Several works have presented word lists associated with social groups for gender (Bolukbasi et al., 2016; Garg et al., 2018; Gupta et al., 2022; Hall Maudslay et al., 2019; Lu et al., 2020; Zhao et al., 2017; 2018), race (Caliskan et al.”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 8×
“Blodgett et al. (2021) highlight shortcomings of pseudo-log-likelihood metrics that compare stereotype and anti-stereotype sentences.”
BackgroundInfluentialLiu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 885
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 1×
“Several works have presented word lists associated with social groups for gender (Bolukbasi et al., 2016; Garg et al., 2018; Gupta et al., 2022; Hall Maudslay et al., 2019; Lu et al., 2020; Zhao et al., 2017; 2018), race (Caliskan et al.”
BackgroundThe Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · cited 5×
“• Classification: Toxicity detection models misclassify African-American English tweets as negative more often that those written in Standard American English (Mozafari et al., 2020; Sap et al., 2019).”
BackgroundInfluentialTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
Algorithmic injustice: A relational ethics approach
Birhane, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100205 · Cited by 707
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 10×
“WinoBias+ (Vanmassenhove et al., 2021) augments WinoBias with gender-neutral alternatives, similar to Winogender’s neutral option, with 3,167 total instances.”
BackgroundInfluentialMeasuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · cited 3×
“As an alternative to CDA, Dixon et al. (2018) add non-toxic examples for groups disproportionately represented with toxicity, until the distribution between toxic and non-toxic examples is balanced across groups.”
BackgroundInfluentialStereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · cited 15×
“With αi as the attention associated with si, AULA is given by: AULA(S) = 1 |S| ∑ s∈S αi logP (s|S; θ) (15) For CPS, CAT, AUL, and AULA, and for stereotyping sentences S1 and less- or anti-stereotyping sentences S2, the bias score can be computed as: biasf∈{CPS, CAT, AUL, AULA}(S) = I (f(S1) > f(S2)) (16) where I is th…”
BackgroundMethodInfluentialAdapterFusion: Non-Destructive Task Composition for Transfer Learning
Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., & Gurevych, I. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.39 · Cited by 636 · cited 1×
“(2023a) introduce adapter modules for bias mitigation, based on adapter networks that learn task-specific parameters (Pfeiffer et al., 2021).”
BackgroundDExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.522 · Cited by 634 · cited 4×
“Embeddings Liu et al. (2020) R = λ ∑ (ai,aj )∈A ∥E(ai) − E(aj)∥2 Yang et al.”
InfluentialLiu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2024) · AI Open · doi:10.1016/j.aiopen.2023.08.012 · Cited by 596
A Framework of Potential Sources of Harm Throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 576
The Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 537 · cited 4×
“For instance, Regard Score (Sheng et al., 2019) measures polarity towards and perceptions of social groups, similar to a sentiment and respect score.”
BackgroundInfluentialCalibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 1×
“Even without fine-tuning, foundation models increasingly enable few- or zero-shot capabilities for a wide array of scenarios like classification, question-answering, logical reasoning, fact retrieval, information extraction, and more, with the task described in a natural language prompt to the model and few or no labe…”
BackgroundPlug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508 · cited 5×
“Dathathri et al. (2019) re-ranks outputs using toxicity scores generated by a simple classifier. The gradients of the classifier model can guide generation towards less toxic outputs. Schramowski et al. (2022) identify moral directions aligned with human and societal ethical norms in pre-trained language models. The a…”
BackgroundMethodResultInfluentialHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 4×
“Stereotypical Associations (ST) (Bommasani et al., 2023) measures bias associated with specific terms, defined as:”
BackgroundMethodInfluentialPersistent Anti-Muslim Bias in Large Language Models
Abid, A., Farooqi, M., & Zou, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462624 · Cited by 476 · cited 6×
“In the first example of modified prompting language, positive triggers are added to the input to condition the model to generate more positive outputs (based on Abid et al. (2021) and Narayanan Venkit et al.”
BackgroundMethodInfluentialPreventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 469
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Cheng, M., Durmus, E., & Jurafsky, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.84 · Cited by 464
On Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461 · cited 5×
“To adapt WEAT to contextualized embeddings, Sentence Encoder Association Test (SEAT) (May et al., 2019) generate embeddings of semantically bleached template-based sentences (e.g., "This is [BLANK]", "[BLANK] are things"), replacing the empty slot with social group and neutral attribute words.”
MethodInfluentialGonen, H., & Goldberg, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1061 · Cited by 454 · cited 1×
“Therefore, there is no single set of values that bias and fairness research can assume, yet, as Green (2019) explains, the assumptions and values in scientific and computing research tend to reflect those of dominant groups.”
BackgroundCrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 10×
“Winograd schemas have since been adapted for bias evaluation to measure words’ associations with social groups, most prominently with Winogender (Rudinger et al., 2018) and WinoBias (Zhao et al.”
MethodInfluentialExamining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384 · cited 1×
“For instance, Lauscher et al. (2021) introduce debiasing adapter modules, called ADELE, to mitigate gender bias.”
BackgroundMarked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Cheng, M., Durmus, E., & Jurafsky, D. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.18189 · Cited by 371
Indigenous data, indigenous methodologies and indigenous data sovereignty
Walter, M., & Suina, M. (2019) · International Journal of Social Research Methodology · doi:10.1080/13645579.2018.1531228 · Cited by 367
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362
Measuring Bias in Contextualized Word Representations
Kurita, K., Vyas, N., Pareek, A., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3823 · Cited by 348 · cited 4×
“Log-Probability Bias Score (LPBS) (Kurita et al., 2019) uses a similar template-based approach as DisCo to measure bias in neutral attribute words (e.g., occupations), but normalizes a token’s predicted probability pa (based on a template "[MASK] is a [NEUTRAL ATTRIBUTE]") with the model’s prior probability pprior (ba…”
BackgroundMethodInfluentialPower to the People? Opportunities and Challenges for Participatory AI
Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022) · Equity and Access in Algorithms Mechanisms and Optimization · doi:10.1145/3551624.3555290 · Cited by 330
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · cited 1×
“Bai et al. (2022)’s Constitutional AI uses a similar approach, but with the reward model based on a list of human-specified principles, instead of example prompts and outputs.”
MethodGender Bias in Contextualized Word Embeddings
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., & Chang, K.-W. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1064 · Cited by 319 · cited 10×
“WinoBias+ (Vanmassenhove et al., 2021) augments WinoBias with gender-neutral alternatives, similar to Winogender’s neutral option, with 3,167 total instances.”
BackgroundInfluentialQUARK: Controllable Text Generation with Reinforced Unlearning
Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., & Choi, Y. (2022) · neural information processing systems · doi:10.52202/068431-2001 · Cited by 316
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · cited 1×
“Typically trained on an enormous scale of uncurated Internet-based data, LLMs inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities (Bender et al., 2021; Dodge et al., 2021; Sheng et…”
BackgroundTowards Debiasing Sentence Representations
Liang, P. P., Li, I., Zheng, E., Lim, Y. C., Salakhutdinov, R., & Morency, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.488 · Cited by 299
Quark: Controllable Text Generation with Reinforced UnlearningPreprint
Lu, X., Welleck, S., Jiang, L., Hessel, J., Qin, L., West, P., Ammanabrolu, P., & Choi, Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.13636 · Cited by 299
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295 · cited 5×
“Delineating between social groups is often required to assess disparities, yet can simultaneously legitimize social constructions, reinforce power differentials, and enable systems of oppression (Hanna et al., 2020).”
BackgroundMethodInfluentialLarge pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., & Weston, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.656 · Cited by 286
Recipes for Safety in Open-domain ChatbotsPreprint
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07079 · Cited by 284
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · arXiv · doi:10.48550/arxiv.2304.03738 · Cited by 282 · cited 3×
“, LLM-aided healthcare tools may use proxies associated with demographic factors that exacerbate inequities in patient care (Ferrara, 2023) *African-American English; †Standard American English .”
BackgroundShould ChatGPT be biased? Challenges and risks of bias in large language models
Ferrara, E. (2023) · First Monday · doi:10.5210/fm.v28i11.13346 · Cited by 282 · cited 3×
“, LLM-aided healthcare tools may use proxies associated with demographic factors that exacerbate inequities in patient care (Ferrara, 2023) *African-American English; †Standard American English .”
BackgroundSocial Biases in NLP Models as Barriers for Persons with Disabilities
Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.487 · Cited by 274
Social Biases in NLP Models as Barriers for Persons with Disabilities
Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020) · arXiv · doi:10.48550/arxiv.2005.00813 · Cited by 274 · cited 1×
“From negative sentiment and toxicity directed towards some social groups, to stereotypical linguistic associations, to lack of recognition of certain language dialects, the biases of LLMs have been well-documented (e.g., Blodgett & O’Connor, 2017; Hutchinson et al., 2020; Mei et al., 2023; Měchura, 2022; Mozafari et a…”
BackgroundAuto-Debias: Debiasing Masked Language Models with Automated Biased Prompts
Guo, Y., Yang, Y., & Abbasi, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.72 · Cited by 265
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2021) · Communications of the ACM · doi:10.1145/3433949 · Cited by 262
Hate speech detection and racial bias mitigation in social media based on BERT model
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237861 · Cited by 260 · cited 5×
“• Classification: Toxicity detection models misclassify African-American English tweets as negative more often that those written in Standard American English (Mozafari et al., 2020; Sap et al., 2019).”
BackgroundInfluentialManzini, T., Yao Chong, L., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1062 · Cited by 255 · cited 3×
“, 2017; 2018), race (Caliskan et al., 2017; Garg et al., 2018; Gupta et al., 2022; Manzini et al., 2019), age (Caliskan et al.”
BackgroundHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
Gender Bias in Neural Natural Language Processing
Lu, K., Mardziel, P., Wu, F., Amancharla, P., & Datta, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-62077-6_14 · Cited by 250
Assessing Social and Intersectional Biases in Contextualized Word RepresentationsPreprint
Tan, Y. C., & Celis, L. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01485 · Cited by 250
Automatically Neutralizing Subjective Bias in Text
Pryzant, R., Martinez, R. D., Dass, N., Kurohashi, S., Jurafsky, D., & Yang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5385 · Cited by 249
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244 · cited 11×
“Similar to psycholinguistic norms, the bias score is calculated as a weighted average of bias scores for all words in the text: Gender Polarity(Ŷ) = ∑ Ŷ ∈Ŷ ∑ ŷ∈Ŷ sign(bias-score(ŷ))bias-score(y)2∑ Ŷ ∈Ŷ ∑ ŷ∈Ŷ |bias-score(ŷ)| (29) Cryan et al. (2020) introduces a similar Gender Lexicon Dataset, which also ass…”
BackgroundMethodInfluentialQueens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation
Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., & Weston, J. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.1911.03842 · Cited by 243 · cited 7×
“Dinan et al. (2020), for instance, mitigate gender bias in dialogue generation by binning each training example by the presence or absence of masculine or feminine gendered words, and appending a control token corresponding to the bin to each prompt. Xu et al. (2020) adapt Dinan et al. (2020)’s approach to provide saf…”
BackgroundInfluentialPredictive biases in natural language processing models: A conceptual framework and overview
Shah, D. S., Schwartz, H. A., & Hovy, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.468 · Cited by 241
Linguistic Intergroup Bias: Stereotype Perpetuation Through Language
Maass, A. (1999) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60272-5 · Cited by 235 · cited 1×
“Language can make concrete the categorization and differentiation of social groups, giving voice to generic or derogatory labels, and linking categories of people to stereotypical, unrepresentative, or overly general characteristics (Beukeboom & Burgers, 2019; Maass, 1999).”
BackgroundSelf-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227
Mind the GAP: A balanced corpus of gendered ambiguous pronouns
Webster, K., Recasens, M., Axelrod, V., & Baldridge, J. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00240 · Cited by 223 · cited 6×
“Incorporating subjectivity into gendered ambiguous pronoun (GAP) resolution using style transfer.”
InfluentialUNQOVERing Stereotyping Biases via Underspecified Questions
Li, T., Khashabi, D., Khot, T., Sabharwal, A., & Srikumar, V. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.311 · Cited by 222
GeDi: Generative Discriminator Guided Sequence Generation
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.424 · Cited by 220
Parameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A., & Kim, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.378 · Cited by 220 · cited 3×
“R(t) = λ ∣∣∣∣∣log ∑|Ai| k=1 P (Ai,k)∑|Aj | k=1 P (Aj,k) ∣∣∣∣∣ (41) In Auto-Debias, Guo et al. (2022) extend these ideas to non-binary social groups, encouraging the generated output to be independent of social group.”
BackgroundMethodInfluentialIdentifying and Reducing Gender Bias in Word-Level Language Models
Bordia, S., & Bowman, S. R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop · doi:10.18653/v1/n19-3002 · Cited by 219
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
Davani, A. M., Díaz, M., & Prabhakaran, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00449 · Cited by 219
Movement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
Zmigrod, R., Mielke, S. J., Wallach, H., & Cotterell, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1161 · Cited by 214 · cited 1×
“Counterfactual data augmentation (CDA) is one of the primary of these augmentation techniques (Lu et al., 2020; Qian et al., 2022; Webster et al., 2020; Zmigrod et al., 2019), replacing protected attribute words, such as gendered pronouns, to achieve a balanced dataset.”
BackgroundDon’t ask if artificial intelligence is good or fair, ask how it shifts power
Kalluri, P. (2020) · Nature · doi:10.1038/d41586-020-02003-2 · Cited by 213
Debiasing Pre-trained Contextualised Embeddings
Kaneko, M., & Bollegala, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.107 · Cited by 211
Counterfactual Fairness in Text Classification through Robustness
Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, E. H., & Beutel, A. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3317950 · Cited by 198
Towards Controllable Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.291 · Cited by 197
Measuring and Reducing Gendered Correlations in Pre-trained Models
Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E., & Petrov, S. (2020) · arXiv · doi:10.48550/arxiv.2010.06032 · Cited by 195
Salazar, J., Liang, D., Nguyen, T. Q., & Kirchhoff, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.240 · Cited by 191 · cited 3×
“Several techniques leverage pseudo-log-likelihood (PLL) (Salazar et al., 2020; Wang & Cho, 2019) to score the probability of generating a token given other words in the sentence.”
MethodIt’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution
Maudslay, R. H., Gonen, H., Cotterell, R., & Teufel, S. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1530 · Cited by 190 · cited 3×
“Hall Maudslay et al. (2019) propose a variant of CDA called counterfactual data substitution (CDS) for gender bias mitigation, in which gendered text is randomly substituted with a counterfactual version with 0.5 probability, as opposed to duplicating and reversing the gender of all gendered examples. Hall Maudslay et…”
BackgroundInfluentialSustainable Modular Debiasing of Language Models
Lauscher, A., Lueken, T., & Glavaš, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.411 · Cited by 190
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186
Guo, W., & Caliskan, A. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462536 · Cited by 180
Towards Debiasing NLU Models from Unknown Biases
Utama, P. A., Moosavi, N. S., & Gurevych, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.613 · Cited by 178
Nationality Bias in Text Generation
Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T.-H., & Wilson, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.9 · Cited by 175
Does Gender Matter? Towards Fairness in Dialogue Systems
Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., & Tang, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.390 · Cited by 173
Debiasing Pre-trained Contextualised Embeddings
Kaneko, M., & Bollegala, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2101.09523 · Cited by 171
Social chemistry 101: Learning to reason about social and moral norms
Forbes, M., Hwang, J. D., Shwartz, V., Sap, M., & Choi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.48 · Cited by 169
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 169
Hurtlex: A Multilingual Lexicon of Words to Hurt
Bassignana, E., Basile, V., & Patti, V. (2018) · Proceedings of the Fifth Italian Conference on Computational Linguistics CLiC-it 2018 · doi:10.4000/books.aaccademia.3085 · Cited by 167 · cited 1×
“For identity-related template prompts and the top-k completions Ŷk, the metric calculates how many completions contain words in the HurtLex lexicon (Bassignana et al., 2018), given by:”
BackgroundThe GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159
Unlearning Bias in Language Models by Partitioning Gradients
Yu, C., Jeoung, S., Kasi, A., Yu, P., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.375 · Cited by 158
Blodgett, S. L., & O'Connor, B. (2017) · arXiv · doi:10.48550/arxiv.1707.00061 · Cited by 153
Towards Understanding and Mitigating Social Biases in Language Models
Liang, P. P., Wu, C., Morency, L.-P., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2106.13219 · Cited by 153 · cited 2×
“We provide a non-exhaustive list of examples of settings where bias may manifest in unique forms, depending on the task: • Text Generation: In generated text, bias may appear locally or globally (Liang et al., 2021; Sheng et al., 2019; Yang et al., 2022).”
BackgroundFairFil: Contrastive Neural Debiasing Method for Pretrained Text EncodersPreprint
Cheng, P., Hao, W., Yuan, S., Si, S., & Carin, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06413 · Cited by 150 · cited 7×
“The pairs of sentences are often generated by replacing protected attributes with their opposite or an alternative (Cheng et al., 2021; He et al., 2022a; Oh et al., 2022). Cheng et al. (2021)’s FairFil, for instance, trains a network to maximize the mutual information between an original sentence and its counterfactua…”
BackgroundInfluentialLanguage and discrimination: Generating meaning, perceiving identities, and discriminating outcomes
Craft, J. T., Wright, K. E., Weissler, R. E., & Queen, R. M. (2020) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-011718-011659 · Cited by 147 · cited 1×
“In fact, the determination of which forms of language are considered standard or correct also reinforces social hierarchies that can justify the inferiority of some groups (Blodgett et al., 2020; Craft et al., 2020).”
BackgroundStereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · cited 2×
“Others lack consistency in their definitions of bias, or do not seriously engage with the relevant power dynamics that perpetuate the underlying harm (Blodgett et al., 2021).”
BackgroundAn Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models
Meade, N., Poole-Dayan, E., & Reddy, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.132 · Cited by 136
Should ChatGPT Be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · doi:10.2139/ssrn.4627814 · Cited by 133
Mitigating Political Bias in Language Models through Reinforced Calibration
Liu, R., Jia, C., Wei, J., Xu, G., Wang, L., & Vosoughi, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17744 · Cited by 132
Detoxifying Language Models Risks Marginalizing Minority Voices
Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., & Klein, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.190 · Cited by 131
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131
Cao, Y., Pruksachatkun, Y., Chang, K.-W., Gupta, R., Kumar, V., Dhamala, J., & Galstyan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.62 · Cited by 127
AI and the Everything in the Whole Wide World Benchmark
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021) · arXiv · doi:10.48550/arxiv.2111.15366 · Cited by 127
Beukeboom, C. J., & Burgers, C. (2019) · Review of Communication Research · doi:10.12840/issn.2255-4165.017 · Cited by 126
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J., Maini, V., Yogatama, D., & Kohli, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.7 · Cited by 124
Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function
Qian, Y., Muaz, U., Zhang, B., & Hyun, J. W. (2019) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p19-2031 · Cited by 122
NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints
Lu, X., West, P., Zellers, R., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.339 · Cited by 122
Barikeri, S., Lauscher, A., Vulić, I., & Glavaš, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.151 · Cited by 121 · cited 3×
“RedditBias (Barikeri et al., 2021) introduces a conversational dataset generated from Reddit conversations to assess stereotypes between dominant and minoritized groups along the dimensions of gender, race, religion, and queerness.”
BackgroundInfluential(2018) · International Conference on Machine Learning · Cited by 116
Racial identification by speech
BAUGH, J. (2000) · American Speech · doi:10.1215/00031283-75-4-362 · Cited by 113 · cited 1×
“Linguistic profiling, for instance, can discriminate against speakers of a dialect considered non-standard (Baugh, 2000; Loudermilk, 2015).”
BackgroundRekabsaz, N., Kopeinik, S., & Schedl, M. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462949 · Cited by 111
Unmasking the Mask – Evaluating Social Biases in Masked Language Models
Kaneko, M., & Bollegala, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21453 · Cited by 111
Re-contextualizing Fairness in NLP: The Case of India
Bhatt, S., Dev, S., Talukdar, P. P., Dave, S., & Prabhakaran, V. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.55 · Cited by 107
Steed, R., Panda, S., Kobren, A., & Wick, M. J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.247 · Cited by 104 · cited 1×
“Several works point out that biases in the embedding space have only weak or inconsistent relationships with biases in downstream tasks (Cabello et al., 2023; Cao et al., 2022; Goldfarb-Tarrant et al., 2021; Orgad & Belinkov, 2022; Orgad et al., 2022; Steed et al., 2022).”
BackgroundReducing Sentiment Bias in Language Models via Counterfactual EvaluationPreprint
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J. W., Maini, V., Yogatama, D., & Kohli, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03064 · Cited by 104 · cited 3×
“Huang et al. (2020) alternatively compares counterfactual embeddings with cosine similarity.”
BackgroundInfluentialEntropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists
Attanasio, G., Nozza, D., Hovy, D., & Baralis, E. (2022) · Findings · doi:10.48550/arxiv.2203.09192 · Cited by 101 · cited 8×
“For example, understanding that attention mechanisms play a key role in encoding bias informs attention-targeting mitigations such as Attanasio et al. (2022), Gaci et al. (2022), and Zayed et al. (2023a). Research into how and in which components (e.”
BackgroundInfluentialBringing the People Back In: Contesting Benchmark Machine Learning Datasets
Denton, R., Hanna, A., Amironesei, R., Smart, A., Nicole, H., & Scheuerman, M. K. (2020) · arXiv · doi:10.48550/arxiv.2007.07399 · Cited by 100
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97 · cited 6×
“, Responding "I’m sorry to hear that" to "I’m an autistic dad" conveys a negative misrepresentation of autism (Smith et al., 2022) Stereotyping Negative, generally immutable abstractions about a labeled social group e.”
BackgroundMethodInfluentialHONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95 · cited 4×
“For identity-related template prompts and the top-k completions Ŷk, the metric calculates how many completions contain words in the HurtLex lexicon (Bassignana et al., 2018), given by: HONEST(Ŷ) = ∑ Ŷk∈Ŷk ∑ ŷ∈Ŷk IHurtLex(ŷ) |Ŷ| · k (27) Psycholinguistic Norms (Dhamala et al., 2021), presented with the BOLD dat…”
BackgroundInfluentialBlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.03188 · Cited by 95
Denton, E., Díaz, M., Kivlichan, I., Prabhakaran, V., & Rosen, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.04554 · Cited by 93
Unmasking the Mask - Evaluating Social Biases in Masked Language Models
Kaneko, M., & Bollegala, D. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2104.07496 · Cited by 93
A Simple and Effective Pruning Approach for Large Language Models
Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. (2023) · arXiv · doi:10.48550/arxiv.2306.11695 · Cited by 92
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Solaiman, I., & Dennison, C. (2021) · arXiv · doi:10.48550/arxiv.2106.10328 · Cited by 90
Balancing out Bias: Achieving Fairness Through Balanced Training
Han, X., Baldwin, T., & Cohn, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.779 · Cited by 88
ADEPT: A DEbiasing PrompT Framework
Yang, K., Yu, C., Fung, Y. R., Li, M., & Ji, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26279 · Cited by 87
VALUE: Understanding Dialect Disparity in NLU
Ziems, C., Chen, J., Harris, C., Anderson, J., & Yang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.258 · Cited by 86
Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias
Bartl, M., Nissim, M., & Gatt, A. (2020) · arXiv · doi:10.48550/arxiv.2010.14534 · Cited by 84 · cited 3×
“This work uses the adapter layers to learn debiasing knowledge by fine-tuning on the BEC-Pro gender bias dataset (Bartl et al., 2020).”
MethodInfluentialA Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82
Leavy, S., Σιαπέρα, Ε., & O’Sullivan, B. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462598 · Cited by 81
Garimella, A., Amarnath, A., KUMAR, K. T. K., Yalla, A. P., Natarajan, A., Chhaya, N., & Srinivasan, B. V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.397 · Cited by 80 · cited 4×
“They introduce a regularization term comparing the output softmax probabilities P for binary masculine and feminine words pairs, which was adapted by Garimella et al. (2021) for binary race word pairs.”
BackgroundMethodInfluentialProsocialDialog: A prosocial backbone for conversational agents
Kim, H., Yu, Y., Jiang, L., Lu, X., Khashabi, D., Kim, G., Choi, Y., & Sap, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.267 · Cited by 80
They, Them, Theirs: Rewriting with Gender-Neutral EnglishPreprint
Sun, T., Webster, K., Shah, A., Wang, W. Y., & Johnson, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06788 · Cited by 79 · cited 1×
“To provide gender-neutral alternatives to sentences with gendered pronouns, several works (Jain et al., 2021; Sun et al., 2021; Vanmassenhove et al., 2021) use a rules-based approach to generate parallel debiased sentences from biased sources, and then train a machine translation model to translate from biased sentenc…”
MethodDebiasing Pre-Trained Language Models via Efficient Fine-Tuning
Gira, M., Zhang, R., & Lee, K. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.8 · Cited by 79 · cited 3×
“Gira et al. (2022) freeze over 99% of a model’s parameters before fine-tuning on the WinoBias (Zhao et al., 2019) and CrowS-Pairs (Nangia et al., 2020) datasets, only updating a selective set of parameters, such as layer norm parameters or word positioning embeddings. Ranaldi et al. (2023) only update the attention ma…”
BackgroundInfluentialSelf-citation“Nice Try, Kiddo”: Investigating Ad Hominems in Dialogue Responses
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.60 · Cited by 78
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 75
WinoQueer: A community-in-the-loop benchmark for anti-LGBTQ+ bias in large language models
Felkner, V., Chang, H.-C. H., Jang, E., & May, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.507 · Cited by 74
OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings
Dev, S., Li, T., Phillips, J. M., & Srikumar, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.411 · Cited by 72 · cited 1×
“WEAT* (Dev et al., 2021) presents an alternative, where W1 and W2 are instead defintionally masculine and feminine words (e.”
BackgroundCzarnowska, P., Vyas, Y., & Shah, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00425 · Cited by 72
Chung, J., Kamar, E., & Amershi, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.34 · Cited by 72
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction
Ma, X., Sap, M., Rashkin, H., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.602 · Cited by 71
Perturbation augmentation for fairer NLP
Qian, R., Ross, C., Fernandes, J., Smith, E. M., Kiela, D., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.646 · Cited by 69
Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts
Hallinan, S., Liu, A., Choi, Y., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.21 · Cited by 67
Learning Fair Representation via Distributional Contrastive Disentanglement
Oh, C., Won, H., So, J., Kim, T., Kim, Y., Choi, H., & Song, K. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539232 · Cited by 66 · cited 1×
“The pairs of sentences are often generated by replacing protected attributes with their opposite or an alternative (Cheng et al., 2021; He et al., 2022a; Oh et al., 2022).”
BackgroundSociodemographic Bias in Language Models: A Survey and Forward Path
Gupta, V. K., Venkit, P. N., Wilson, S., & Passonneau, R. J. (2024) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2024.gebnlp-1.19 · Cited by 66 · cited 1×
“As we discuss in Section 4, several widely-used evaluation datasets suffer from reliability and validity issues, including ambiguities about whether instances accurately reflect real-world stereotypes, inconsistent treatment of social groups, assumptions of near-perfect understanding of language, and lack of syntactic…”
BackgroundOn Transferability of Bias Mitigation Effects in Language Model Fine-Tuning
Jin, X., Barbieri, F., Kennedy, B., Davani, A. M., Neves, L., & Ren, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.296 · Cited by 65 · cited 1×
“Other works have since followed this framework (Han et al., 2021b; Jin et al., 2021), training an encoder and discriminator, where the discriminator predicts a protected attribute from a”
BackgroundTheory-Grounded Measurement of U.S. Social Stereotypes in English Language Models
Cao, Y., Sotnikova, A., Daumé, H., Rudinger, R., & Zou, L. X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.92 · Cited by 65
Do neural ranking models intensify gender bias?
Rekabsaz, N., & Schedl, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401280 · Cited by 64
Mitigating Language-Dependent Ethnic Bias in BERT
Ahn, J., & Oh, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.42 · Cited by 64
Critic-Guided Decoding for Controlled Text Generation
Kim, M., Lee, H., Yoo, K. M., Park, J., Lee, H., & Jung, K. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10938 · Cited by 64
Mitigating harm in language models with conditional-likelihood filtrationPreprint
Ngo, H., Raterink, C., Araújo, J. G. M., Zhang, I., Chen, C., Morisot, A., & Frosst, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07790 · Cited by 63
Research works printed but not linked: 5
- Barocas, 2019, Fairness and Machine Learning: Limitations and Opportunities
- Blodgett, Su Lin . 2021. Sociolinguistically Driven Approaches for Just Natural Language Processing. Ph.D. thesis. University of Massachusetts Amherst.
- Green, 2019, ”Good” isn’t good enough, 1
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
- Wang, 2019, BERT has a mouth, and it must speak: BERT as a Markov random field language model, 30, 10.18653/v1/w19-2304 · doi:10.18653/v1/w19-2304
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594 · Cited by 1,781
“122 In the context of LLM-as-a-judge, it indicates that evaluators might score expressions from unfamiliar cultures poorly.”
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
“Gödel Agent [206] utilizes dynamic code modification techniques to alter its own logic and behavior based on the feedback signals from the environment and stores the updated agent logic in the agent memory for recursive improvement.”
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
“(3 ) Bias : Bias in LLMs manifests as the propagation and amplification of discriminatory tendencies present in training data, such as racial and gender biases, leading to unfair or harmful outputs [327–330].”
Metadata
- Authors
- Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K.
- Affiliations
- Stanford University (US)
- Adobe Research. ryrossi@adobe.com
- Pattern Data. joe.barrow@patterndataworks.com
- Adobe Research. tanjim@adobe.com
- Adobe Research. sukim@adobe.com
- Adobe Research. dernonco@adobe.com
- Adobe Research. tyu@adobe.com
- Adobe Research. ruizhang@adobe.com
- Intel (United States) (US)
- ORCID
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 3
- Pages
- 1097-1179
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4399528455
- Semantic Scholar
- bcfa73aedf1b2d1ee4f168e21298a37ac55a37f7
- Cited by
- 1,245
- References
- 280
- Citations by year
- 2026: 596 · 2025: 451 · 2024: 188
- FWCI
- 132.76
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, social bias, fairness, bias assessment, bias mitigation, bias metrics, data preprocessing, post-processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.