The Woman Worked as a Babysitter: On Biases in Language Generation
Open accessAbstract
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces “regard” toward demographic groups as a bias metric for natural language generation, arguing that sentiment is an imperfect proxy for bias. It annotates generated text from GPT-2 and LM 1B, builds a BERT-based regard classifier, and finds demographic biases that are often more pronounced under regard than sentiment.
Key findings
- GPT-2 generations show different levels of bias by demographic: in respect contexts, more negative associations for Black, man, and gay; in occupation contexts, more negative associations for Black, woman, and gay.
- LM 1B samples are overall less biased across demographic pairs than GPT-2 samples.
- Sentiment is a better proxy for bias in respect contexts than in occupation contexts: Spearman correlations between sentiment annotations and regard annotations are 0.95 for respect, 0.70 for occupation, and 0.82 overall; VADER predictions vs. regard annotations are 0.69, 0.54, and 0.61 respectively.
- A BERT-based regard classifier outperforms LSTM and re-purposed sentiment baselines by more than 20% in test accuracy, with averaged accuracy of 78% for respect and 79% for occupation.
Methods
- The authors define two bias contexts (respect and occupation), use five placeholder prefix templates per context, and fill them with gender, race, or sexual-orientation demographics (female/male, Black/White, gay/straight).
- They generate text from OpenAI GPT-2 (small) and Google LM 1B, sample 360 generated sentences for annotation, and have three annotators label sentiment and regard; 302 samples with majority original-category labels are retained.
- They compare VADER and TextBlob sentiment with regard annotations, then train and evaluate LSTM, re-purposed sentiment, and transfer-learning BERT regard classifiers on a 212/60/30 train/dev/test split.
Limitations
- The paper states that it uses manually selected keywords and phrases to generate text, which could be expanded to more automatic methods to generalize findings.
- The paper states that each demographic type is limited to two classes, which is unrepresentative of real-world diversity, and that the annotated dataset is limited/not large.
- Evident from the text: the classifier test set is only 30 samples, and the analysis covers only two bias contexts and two language models, so classifier reliability and broader generalization remain constrained.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
1 of 2 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
- Premisecitation
There has been little analysis on the correlation of sentiment to human judgment of bias.
“there has been little analysis on the correlation of sentiment to human judgment of bias.”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Lu et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Kiritchenko and Mohammad, 2018 source
The Related Work section says that bias in sentiment analysis systems had received limited study. It does not address whether sentiment scores correlate with human judgments of bias, which is the more specific claim attributed to it. (LLM, conf=0.82)
Experimental — accuracy not yet measured
- Findingown data
Some contexts where biases occur include social connotations that are often subtle and difficult to capture in existing sentiment analysis tools.
“We find that some of the contexts where biases occur include social connotations that are often subtle and difficult to capture in existing sentiment analysis tools.”
Quote in the paperSubstantial claim - Findingown data
A re-purposed off-the-shelf sentiment analyzer does better than or is comparable with the LSTM models.
“In fact, a re-purposed off-the-shelf sentiment analyzer (i.e., taking sentiment predictions as regard predictions) does better than or is comparable with the LSTM models.”
Quote in the paperNot a substantial claim - Findingown data
GPT-2 is biased towards certain demographics.
“Table 1 shows that GPT-2, Ope-nAI's publicly available language model, is biased towards certain demographics.”
Quote in the paperSubstantial claim - Findingown data
GPT-2 text generations exhibit different levels of bias towards different demographics.
“Overall, GPT-2 text generations exhibit different levels of bias towards different demographics.”
Quote in the paperSubstantial claim
- Findingown data
Similar trends of regard scores increasing the gap in negative scores across demographic pairs appear in LM 1B-generated samples and annotated samples.
“We see similar trends of the regard scores increasing the gap in negative scores across a corresponding demographic pair in both the LM 1B-generated samples in row (2) and the annotated samples in row (3).”
Quote in the paperNot a substantial claim - Findingown data
When conditioning on context related to respect, there are more negative associations of black, man, and gay demographics.
“Specifically, when conditioning on context related to respect, there are more negative associations of black, man, and gay demographics.”
Quote in the paperSubstantial claim - Findingown data
The BERT model outperforms all other models by more than 20% in test set accuracy.
“As shown in Figure 1, the BERT model outperforms all other models by more than 20% in test set accuracy”
Quote in the paperSubstantial claim - Findingnone given
LM 1B samples are overall less biased across demographic pairs compared to GPT-2.
“Interestingly, we also observe that the LM 1B samples are overall less biased across demographic pairs compared to GPT-2.”
Quote in the paperNot a substantial claim - Findingown data
When conditioning on context related to occupation, there are more negative associations of black, woman, and gay demographics.
“When conditioning on context related to occupation, there are more negative associations of black, woman, and gay demographics.”
Quote in the paperSubstantial claim - Findingown data
The negative regard score gaps across demographic pairs are more pronounced than sentiment score gaps in respect context.
“While the general positive versus negative score trends are preserved across demographic pairs (e.g., Black vs. White) across charts (1a) and (1b), the negative regard score gaps across demographic pairs are more pronounced.”
Quote in the paperNot a substantial claim - Findingown data
LSTM models with random embeddings or pretrained and tunable word embeddings show limited success.
“We find limited success with the LSTM models when using either random embeddings or pretrained and tunable word embeddings.”
Quote in the paperSubstantial claim - Findingown data
Two popular sentiment analyzers predict a neutral sentiment for the sentence 'XYZ worked as a pimp for 15 years'.
“For example, when we run two popular sentiment analyzers on the sentence”
Quote in the paperSubstantial claim - Interpretationown data
Certain bias contexts (e.g., occupation) are not as well-quantified by sentiment scores.
“Additionally, certain bias contexts (e.g., occupation) are not as well-quantified by sentiment scores.”
Quote in the paperSubstantial claim - Interpretationown data
Transfer learning results indicate the feasibility of building a regard classifier.
“Although our dataset is not large, the promising results of transfer learning indicate the feasibility of building a regard classifier.”
Quote in the paperSubstantial claim - Interpretationreasoning
Regard is a more appropriate metric for bias than sentiment.
“Although the correlations between sentiment and regard are all at least moderately high, regard is, by design, a direct measurement of prejudices towards different demographics and thus a more appropriate metric for bias.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Sentences that describe varying levels of respect for a demographic tend to contain more adjectives that are strongly indicative of the overall sentiment.
“Sentences that describe varying levels of respect for a demographic tend to contain more adjectives that are strongly indicative of the overall sentiment.”
Quote in the paperSubstantial claim - Interpretationown data
Using sentiment analysis as the main metric to measure biases in NLG systems may underestimate the magnitude of biases.
“Furthermore, these results indicate that by using sentiment analysis as the main metric to measure biases in NLG systems, we may be underestimating the magnitude of biases.”
Quote in the paperSubstantial claim - Interpretationown data
Sentiment is a better proxy for bias in respect contexts than in occupation contexts.
“In general, the correlations indicate that sentiment is a better proxy for bias in respect contexts than in occupation contexts.”
Quote in the paperSubstantial claim - Interpretationown data
Samples generated from state-of-the-art language models contain biases towards different demographics.
“In this work, we show that samples generated from state-of-the-art language models contain biases towards different demographics, which is problematic for downstream applications that use these language models.”
Quote in the paperSubstantial claim
- May combine two claims. 10
- Classified as probably not a substantial claim. 6
May combine two claims.
Findingown dataUsing Spearman's correlation, the correlations increase to 0.76 for sentiment and 0.80 for regard.
“Using Spearman's correlation, the correlations increase to 0.76 for sentiment and 0.80 for regard.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor the original categories, inter-annotator agreement (Fleiss' kappa) is 0.60 for sentiment and 0.67 for regard.
“When we look at only the original categories, the kappa becomes 0.60 and 0.67 for sentiment and regard, respectively.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe regard classifier labels more occupation samples as neutral, increases the gap between negative scores, and decreases the gap between positive scores.
“Looking at charts (1c) and (1d) in Figure 2, we see that the regard classifier labels more occupation samples as neutral, and also increases the gap between the negative scores and decreases the gap between the positive scores.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe occupation of 'prostitute' appears frequently.
“The occupation of”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe accuracy scores are similar across bias types; BERT has an averaged 78% for respect and 79% for occupation.
“The accuracy scores are similar across bias types; BERT has an averaged 78% for respect and 79% for occupation.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor the complete set of categories, inter-annotator agreement (Fleiss' kappa) is 0.5 for sentiment and 0.49 for regard.
“For the complete set of categories, we measure inter-annotator agreement with fleiss' kappa; the kappa is 0.5 for sentiment and 0.49 for regard.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe number of instances per category is roughly balanced.
“The number of instances per category is roughly balanced, as shown in Table 4.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe correlation scores generally indicate a reasonably high correlation and reliability of the annotation task.
“These correlation scores generally indicate a reasonably high correlation and reliability of the annotation task.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
PremisereasoningEvaluating biases requires a metric that is directed towards a demographic and that relies on additional cues beyond language polarity.
“Evaluating biases requires a metric that is directed towards a demographic and that relies on additional cues beyond language polarity.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductown dataThe manually constructed prefix templates are common phrases that generate a variety of completions.
“We manually verify these templates are common phrases that generate a variety of completions.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataRegard can be reliably annotated and feasibly used to build an automatic classifier.
“Through annotations and classification experiments, we show that regard can be reliably annotated and feasibly used to build an automatic classifier.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningSentences describing occupations are usually more neutrally worded, though some occupations are socially perceived to be more positive or negative than others.
“In contrast, sentences describing occupations are usually more neutrally worded, though some occupations are socially perceived to be more positive or negative than others.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Premisenone givenIf language model-generated sentences cause group A to be more highly thought of than group B, then the language model perpetuates bias towards group B.
“The intuition to understand regard is that if language model-generated sentences cause group A to be more highly thought of than group B, then the language model perpetuates bias towards group B.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe occupations that are typically regarded more negatively are because they are illegal or otherwise explicit.
“The occupations that are typically regarded more negatively are because they are illegal or otherwise explicit.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Working as a 'pimp' generally has a negative social connotation.
“even though working as a”
The paper relies on this without support.
There has not been a detailed study on biases in open-ended natural language generation.
“To the best of our knowledge, there has not been a detailed study on biases in open-ended natural language generation.”
The paper relies on this without support.
VADER is more robust when applied to generated text than other off-the-shelf sentiment analyzers explored.
“VADER is a rule-based sentiment analyzer that is more robust when applied to our domain of generated text than other off-theshelf sentiment analyzers we explore.”
The paper relies on this without support.
Manually selected keywords and phrases are an appropriate scope to quantify the biases that appear in NLG systems.
“In this paper, we use manually selected keywords and phrases to generate text, which, while an appropriate scope to quantify the biases that appear in NLG systems, could be expanded to more automatic methods and help generalize our findings.”
The paper relies on this without support.
Biases can occur in different textual contexts, some biases manifesting more subtly than others.
“Biases can occur in different textual contexts, some biases manifesting more subtly than others.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In generated text, bias may appear locally or globally.
“In generated text, bias may appear locally or globally (Liang et al. 2021; Sheng et al. 2019; Yang et al. 2022).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Sheng et al. 2019
Language models inherit biases and dominant views from their training corpus.
“Text suggestions based on language models have been known to inherit biases and dominant 'views' from the text corpus used to train the language model [22,39,43,55].”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [55]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Figure 2 claims 302 annotated samples per chart, but 302 exist in total
MinorThe paper says it retains 302 annotated samples in total. Figure 2 separately charts respect and occupation samples, so each context-specific chart cannot contain all 302 samples.
“Note that row (3) has 302 total annotated samples per chart.”
“We take the majority annotation as groundtruth, and only keep samples whose groundtruth is an original category, for a total of 302 samples.”
Cites
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 4× · see citing works
“Moreover, we calculate Spearman’s correlation between 1) sentiment annotations and regard an-6 notations, 2) VADER predictions and sentiment annotations, and 3) VADER predictions and regard annotations in Table 5.”
MethodInfluentialBuilding End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9883 · Cited by 1,639 · cited 2× · see citing works
“Language models are a fundamental component of NLG that are widely used in downstream tasks such as machine translation (Koehn, 2009), dialogue generation (Serban et al., 2016), and story generation (Yao et al.”
BackgroundMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 1× · see citing works
“We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
MethodSelf-citationStatistical machine translation
Koehn, P. (2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309 · see citing works
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · cited 1× · see citing works
“We analyze OpenAI’s GPT-2 (small) language model (Radford et al., 2019) and Google’s language model trained on the One Bil-lion Word Benchmark (Jozefowicz et al., 2016).”
MethodStatistical Machine Translation
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829 · Cited by 874 · see citing works
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 2× · see citing works
“…analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
BackgroundMethodSelf-citationGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 1× · see citing works
“We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
MethodAssessing gender bias in machine translation: a case study with Google Translate
Prates, M. O. R., Avelar, P. H., & Lamb, L. C. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04144-6 · Cited by 410 · cited 1× · see citing works
“Recent works in machine translation (Prates et al., 2018) and dialogue systems (Henderson et al., 2018) have brought to attention the perpetuation of biases in natural language generation (NLG) systems.”
BackgroundExamining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384 · cited 1× · see citing works
“- - Regard Although sentiment has been used as a proxy for bias in other works (Lu et al., 2018; Kiritchenko and Mohammad, 2018), there has been little analysis on the correlation of sentiment to human judgment of bias.”
MethodPlan-and-Write: Towards Better Automatic Storytelling
Yao, L., Peng, N., Weischedel, R., Knight, K. K., Zhao, D., & Yan, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017378 · Cited by 328 · cited 2× · see citing works
“…of NLG that are widely used in downstream tasks such as machine translation (Koehn, 2009), dialogue generation (Serban et al., 2016), and story generation (Yao et al., 2019); as such, biases propagated through the language models will have a profound impact on a variety of other NLG tasks.”
BackgroundSelf-citationGender Bias in Neural Natural Language Processing
Lu, K., Mardziel, P., Wu, F., Amancharla, P., & Datta, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62077-6_14 · Cited by 284 · cited 1× · see citing works
“- - Regard Although sentiment has been used as a proxy for bias in other works (Lu et al., 2018; Kiritchenko and Mohammad, 2018), there has been little analysis on the correlation of sentiment to human judgment of bias.”
MethodEthical Challenges in Data-Driven Dialogue Systems
Henderson, P., Sinha, K., Gontier, N., Ke, N. R., Fried, G., Lowe, R., & Pineau, J. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278777 · Cited by 166 · cited 1× · see citing works
“Recent works in machine translation (Prates et al., 2018) and dialogue systems (Henderson et al., 2018) have brought to attention the perpetuation of biases in natural language generation (NLG) systems.”
BackgroundA Practical Guide to Sentiment Annotation: Challenges and Solutions
Mohammad, S. M. (2016) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/w16-0429 · Cited by 104 · cited 1× · see citing works
“The sentiment and regard annotation guidelines are adapted from Mohammad (2016)’s sentiment annotation guidelines.”
MethodExamining Gender Bias in Languages with Grammatical Gender
Zhou, P., Shi, W., Zhao, J., Huang, K.-H., Chen, M., Cotterell, R., & Chang, K.-W. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1531 · Cited by 81 · cited 1× · see citing works
“…these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation contexts.”
BackgroundSelf-citationGender Bias in Neural Natural Language ProcessingPreprint
Lu, K., Mardziel, P., Wu, F., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11714 · Cited by 70 · cited 1× · see citing works
“- - Regard Although sentiment has been used as a proxy for bias in other works (Lu et al., 2018; Kiritchenko and Mohammad, 2018), there has been little analysis on the correlation of sentiment to human judgment of bias.”
MethodExamining Gender Bias in Languages with Grammatical GenderPreprint
Zhou, P., Shi, W., Zhao, J., Huang, K.-H., Chen, M., Cotterell, R., & Chang, K.-W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02224 · Cited by 20 · cited 1× · see citing works
“…these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation contexts.”
BackgroundSelf-citation
Flagged first, then by citations
Cited by
Sorted by citations
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · see citing works
“…et al. (2019); Mehrabi et al. (2019); Nozza et al. (2019); Prabhakaran et al. (2019); Romanov et al. (2019); Ruane et al. (2019); Sap et al. (2019); Sheng et al. (2019); Sun et al. (2019); Sweeney and Najafian (2019); Vaidya et al. (2019); Gaut et al. (2020); Gencoglu (2020); Hovy et al. (2020);…”
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
“For instance, Regard Score (Sheng et al., 2019) measures polarity towards and perceptions of social groups, similar to a sentiment and respect score.”
BackgroundInfluentialRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
“Although they are the backbone of many modern NLP systems (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019), language models (LMs) pretrained on large web text corpora suffer from degenerate and biased behavior (Sheng et al., 2019; Wallace et al., 2019).”
BackgroundMethodInfluentialStereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · see citing works
“For example, GPT2 (Radford et al., 2019), a pretrained language model, has shown to generate unpleasant stereotypical text when prompted with context containing certain races such as African-Americans (Sheng et al., 2019).”
BackgroundBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationLarge language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
“Prior studies on biases in LLMs have revealed both gender and racial bias on general language tasks 6,16,17 , but no work has assessed whether these models may perpetuate race-based medicine.”
BackgroundGender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381 · see citing works
“…has been shown to exist in word embeddings [9, 16, 19, 31, 51, 61, 89, 100, 101, 103], as well as in a broad array of models developed specifically for various NLP tasks, such as auto-captioning, sentiment analysis, toxicity detection, machine translation, and more [46, 58, 71, 80, 83, 87, 91, 95].”
BackgroundRed Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
“Prior work discovers such biases against certain groups by hypothesizing discriminated-against groups in advance (e.g., a particular gender, race, or religion; Bolukbasi et al., 2016; Sheng et al., 2019; Huang et al., 2020).”
BackgroundMethodDocumenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
“Much recent work has found various kinds of bias in fine-tuned models (e.g., Sheng et al., 2019; Gehman et al., 2020; Li et al., 2020), where the bias is assumed to derive from pretraining data, though this has not previously been easy to verify.”
BackgroundGender and Representation Bias in GPT-3 Generated Stories
Lucy, L., & Bamman, D. (2021) · Proceedings of the Third Workshop on Narrative Understanding · doi:10.18653/v1/2021.nuse-1.5 · Cited by 287 · see citing works
“Language models generate different occupations and levels of respect for different genders, races, and sexual orientations (Sheng et al., 2019; Kirk et al., 2021).”
BackgroundToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.234 · Cited by 279 · see citing works
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261 · see citing works
“Text Data Text Classiication [50, 119, 182, 209, 277, 386] Embedding [46, 56, 156, 246, 274, 397] Language Modeling [48, 148, 238, 320, 371] Machine Translation [34, 91, 157, 329, 345] Dialogue Generation [106, 117, 229] Audio Data Speech Recognition [72, 178, 294, 336]”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244 · see citing works
“[31] noted that sentiment and language polarity may not always directly correlate with bias, and defined regard, a metric that directly measures human-annotated bias by measuring polarity towards a demographic, rather than overall language polarity.”
BackgroundSelf-citationFive sources of bias in natural language processing
Hovy, D., & Prabhumoye, S. (2021) · Language and Linguistics Compass · doi:10.1111/lnc3.12432 · Cited by 298 · see citing works
“Additional metrics, approaches and data sets have been proposed to measure the bias inherent in large language models and their sentence completions (Nangia et al., 2020; Nozza et al., 2021; Sheng et al., 2019).”
BackgroundSelf-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227 · see citing works
AI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233 · see citing works
Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 306 · see citing works
Bias of AI-generated content: an examination of news produced by large language models
Fang, X., Che, S., Mao, M., Zhang, H., Zhao, M., & Zhao, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-55686-2 · Cited by 194 · see citing works
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · see citing works
“However, LLMs inherently reflect biases in their training data [161], [162].”
BackgroundHow many data points is a prompt worth?
Le Scao, T., & Rush, A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.208 · Cited by 179 · see citing works
“Prompt completion also suffers from biases already present within the language model (Sheng et al., 2019).”
BackgroundBBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186 · see citing works
“LMs have been found to reproduce social biases in downstream tasks, such as textual entailment (Dev et al., 2020; Sharma et al., 2021), language generation (Sheng et al., 2019), and coreference resolution (Rudinger et al., 2018).”
BackgroundPersonalized Prompt Learning for Explainable Recommendation
Li, L., Zhang, Y., & Chen, L. (2023) · ACM Transactions on Information Systems · doi:10.1145/3580488 · Cited by 172 · see citing works
Gender bias perpetuation and mitigation in AI technologies: challenges and opportunities
O’Connor, S., & Liu, H. K. (2023) · AI & Society · doi:10.1007/s00146-023-01675-4 · Cited by 216 · see citing works
“Discussions on gender bias often naturally fall into two categories: studies which explore or attempt to measure gender bias in AI techniques (Stanovsky et al. 2019; Sheng et al. 2019;)”
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., & Weston, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.656 · Cited by 157 · see citing works
“sentence debiasing work up until this point foregrounds measuring bias (Lee et al., 2019b; Sheng et al., 2019).”
BackgroundInfluentialToxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178 · see citing works
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.522 · Cited by 199 · see citing works
“Given that large pretrained LMs are at risk of producing toxic content (Sheng et al., 2019; Gehman et al., 2020), steering away from toxic “degener-ation” is crucial for their safe deployment.”
BackgroundInfluentialStereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · see citing works
“…sentence templates where terms pertaining to groups or their attributes are perturbed (Rudinger et al., 2018; Zhao et al., 2018; Kurita et al., 2019), prompts designed to elicit problematic responses (Gehman et al., 2020; Sheng et al., 2019; Example Sentences Context I really like Norweigan salmon.”
BackgroundKnowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
Agarwal, O., Ge, H., Shakeri, S., & Al-Rfou, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.278 · Cited by 161 · see citing works
“Moreover, facts may not be expressed as explicitly in text as they are in KGs, and the variability in the quality of text can eventually cause biases in the resulting models (Bolukbasi et al., 2016; Sheng et al., 2019; Manzini et al., 2019).”
Background“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters
Wan, Y., Pu, G., Sun, J., Garimella, A., Chang, K.-W., & Peng, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.243 · Cited by 175 · see citing works
Self-citationThe Importance of Modeling Social Factors of Language: Theory and Practice
Hovy, D., & Yang, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.49 · Cited by 136 · see citing works
“predictions by large pretrained language models like “the black woman who worked as a babysitter” (Sheng et al., 2019).”
BackgroundSocietal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131 · see citing works
“Language models are the core components for many NLG and NLU tasks, and this task enables directly quantifying biases in large, pre-trained language models (Bordia and Bowman, 2019; Sheng et al., 2019; Solaiman et al., 2019; Brown et al., 2020).”
BackgroundMethodInfluentialSelf-citationReducing Sentiment Bias in Language Models via Counterfactual Evaluation
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J., Maini, V., Yogatama, D., & Kohli, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.7 · Cited by 124 · see citing works
“, 2018), and demographic biases in language generation (Sheng et al., 2019), text classification”
BackgroundMethodInfluentialWANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 131 · see citing works
“We acknowledge that text generated from large pretrained language models is susceptible to perpetuating social harms and containing toxic language (Sheng et al., 2019; Gehman et al., 2020).”
BackgroundAuto-Debias: Debiasing Masked Language Models with Automated Biased Prompts
Guo, Y., Yang, Y., & Abbasi, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.72 · Cited by 120 · see citing works
“Prompts have been an effective tool in probing the internal knowledge relations of language models (Petroni et al., 2019), and they can also reflect the stereotypes encompassed in PLMs (Ousidhoum et al., 2021; Sheng et al., 2019).”
BackgroundMethodSqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Iandola, F., Shaw, A., Krishna, R., & Keutzer, K. (2020) · Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2020.sustainlp-1.17 · Cited by 114 · see citing works
“However, Sheng et al. investigate biases regarding race, gender, and sexual orientation in a GPT-2 self-attention model trained on a language-modeling (text-generation) task [63].”
BackgroundGarrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., & Ureña-López, L. A. (2021) · Applied Sciences · doi:10.3390/app11073184 · Cited by 123 · see citing works
Frozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113 · see citing works
“For high stakes applications in the real-world, there are potential concerns with transfer of harmful biases from one modality to one another using pretrained transformer models (Sheng et al. 2019; Bender et al. 2021).”
BackgroundRethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112 · see citing works
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112 · see citing works
“…we discuss one important dimension, but other potential harms have been discussed in prior work, such as, but not limited to, statistical biases (Sheng et al., 2019; Huang et al., 2020; Abid et al., 2021), privacy concerns (Carlini et al., 2020), and environmental impact (Strubell et al.,…”
BackgroundInfluentialMeasuring and Reducing Gendered Correlations in Pre-trained Models
Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E., & Petrov, S. (2020) · arXiv · doi:10.48550/arxiv.2010.06032 · Cited by 195 · see citing works
“However, more recent work has suggested that such analogy-based measurement techniques are unstable and may not generalize (May et al., 2019), leading to new measurement techniques such as template-based (Kurita et al., 2019) and generation-based (Sheng et al., 2019) tests.”
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
“2020), sentences about black or gay people (Sheng et al. 2019), or sentences about nonbinary, disabled, or”
BackgroundFairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 100 · see citing works
“Similarly, Sheng et al. [102], Huang et al. [103], Dhamala et al. [104] designed templates for detecting fairness bugs in natural language generation systems.”
BackgroundMethodInfluentialThe Security of Using Large Language Models: A Survey with Emphasis on ChatGPT
Zhou, W., Zhu, X., Han, Q., Li, L., Chen, X., Wen, S., & Xiang, Y. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124983 · Cited by 101 · see citing works
Explicitly unbiased large language models still form biased associations
Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T. L. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2416228122 · Cited by 184 · see citing works
Pretrained Transformers as Universal Computation EnginesPreprint
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05247 · Cited by 95 · see citing works
“For high stakes applications in the real-world, there are potential concerns with transfer of harmful biases from one modality to one another using pretrained transformer models (Sheng et al. 2019; Bender et al. 2021).”
BackgroundHONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95 · see citing works
Towards Controllable Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.291 · Cited by 95 · see citing works
“The notion of controllable biases necessitates a quantitative metric for evaluating biases; we use the metric of regard defined by Sheng et al. (2019).”
BackgroundMethodResultInfluentialSelf-citationSecurity and Privacy on Generative Data in AIGC: A Survey
Wang, T., Zhang, Y., Qi, S., Zhao, R., Xia, Z., & Weng, J. (2024) · ACM Computing Surveys · doi:10.1145/3703626 · Cited by 105 · see citing works
“Some works [104, 105] have found stronger associations between males and occupations in language models, verifying the gender bias in generative data.”
BackgroundMetamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models
Ma, P., Wang, S., & Liu, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/64 · Cited by 80 · see citing works
“…severe legal or ethical issues, when the decisions are un-fairly made by being biased against certain subpopulations within sensitive attributes (e.g., those of a particular race, gender, nationality, or sexual orientation) [Kusner et al. , 2017; Bolukbasi et al. , 2016; Sheng et al. , 2019].”
BackgroundA Survey of Race, Racism, and Anti-Racism in NLP
Field, A., Blodgett, S. L., Talat, Z., & Tsvetkov, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.149 · Cited by 94 · see citing works
“Analyze Corpus Preoţiuc-Pietro and Ungar (2018) 2018 COLING Social Science/Media Develop Model Sheng et al. (2019) 2018 EMNLP Text Generation Detect Bias Wojatzki et al. (2018) 2018 LREC Social Science/Media Collect Corpus Wood-Doughty et al. (2018) 2018 Workshop Social Science/Media Develop Model Clarke and Grieve (…”
BackgroundAbramski, K., Citraro, S., Lombardi, L., Rossetti, G., & Stella, M. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030124 · Cited by 81 · see citing works
“Sentiment scores and measurements of “regard” towards a demographic have been applied to assess stereotypes related to gender and sexual orientation in output produced by GPT-2 [37].”
MethodAriai, F., Mackenzie, J., & Demartini, G. (2025) · ACM Computing Surveys · doi:10.1145/3777009 · Cited by 79 · see citing works
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97 · see citing works
“, 2021) or from slotting a set of terms into templates (Kurita et al., 2019; May et al., 2019; Sheng et al., 2019; Webster et al., 2020).”
BackgroundCultural Bias in Large Language Models: A Comprehensive Analysis and Mitigation Strategies
Liu, Z. (2025) · Journal of Transcultural Communication · doi:10.1515/jtc-2023-0019 · Cited by 77 · see citing works
“…are exploring a variety of methods to reduce cultural bias in LLMs, such as training on more diverse datasets and developing new evaluation and mitigation approaches (Abid et al., 2021a, 2021b; Ahn and Oh, 2021; Cao et al., 2022; Nadeem et al., 2020; Nozza et al., 2021; Sheng et al., 2019).”
StarCoder 2 and The Stack v2: The Next Generation
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.19173 · Cited by 74 · see citing works
WangchanBERTa: Pretraining transformer-based Thai Language ModelsPreprint
Lowphansirikul, L., Polpanumas, C., Jantrakulchai, N., & Nutanong, S. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09635 · Cited by 73 · see citing works
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 74 · see citing works
“…et al., 2020; Garg et al., 2018), PTLMs have also been shown to recreate stereotypical content due to the nature of their training data (Sheng et al., 2019) Different probing experiments have been proposed to study the drawbacks of PTLMs in areas such as the biomedical domain (Jin et…”
BackgroundDemystifying ChatGPT: An In-depth Survey of OpenAI’s Robust Large Language Models
Bhattacharya, P., Prasad, V. K., Verma, A., Gupta, D. K., Sapsomboon, A., Viriyasitavat, W., & Dhiman, G. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10115-5 · Cited by 78 · see citing works
Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection
Nozza, D. (2021) · doi:10.18653/v1/2021.acl-short.114 · Cited by 68 · see citing works
“We are aware that the inherent (gender) biases of sentence and word embeddings are affecting the model’s performance on detecting hate speech towards women (Bolukbasi et al., 2016; Sheng et al., 2019; Nangia et al., 2020; Nozza et al., 2021).”
BackgroundQuantifying and alleviating political bias in language models
Liu, R., Jia, C., Wei, J., Xu, G., & Vosoughi, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2021.103654 · Cited by 68 · see citing works
Envisioning Information Access Systems: What Makes for Good Tools and a Healthy Web?
Shah, C. S., & Bender, E. M. (2024) · ACM Transactions on the Web · doi:10.1145/3649468 · Cited by 72 · see citing works
“Specifically, we will review LLM-based generative IA in comparison to classical discriminative systems with an eye toward the dangers of the generative systems as well as future directions for system development.”
Mitigating Political Bias in Language Models through Reinforced Calibration
Liu, R., Jia, C., Wei, J., Xu, G., Wang, L., & Vosoughi, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17744 · Cited by 70 · see citing works
“The ethical implications of bias in NLG have started to receive considerable attention in discussions around the social impact of AI ( Sheng et al. 2020, 2019; Wallace et al. 2019; Bordia and Bowman 2019).”
BackgroundUNQOVERing Stereotyping Biases via Underspecified Questions
Li, T., Khashabi, D., Khot, T., Sabharwal, A., & Srikumar, V. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.311 · Cited by 72 · see citing works
“Prior works (e.g., Sheng et al., 2019; Rudinger et al., 2018) have shown that gender-occupation bias is predominant in textual corpora, and consequently in learned representations.”
BackgroundPowerTransformer: Unsupervised Controllable Revision for Biased Language Correction
Ma, X., Sap, M., Rashkin, H., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.602 · Cited by 61 · see citing works
“Additionally, future work could consider alternative views of portrayal biases (e.g., “regard” or bias directed at different demographic groups; Sheng et al., 2019; Sap et al., 2020), or use more holistic views of gender roles (e.g., “mascu-line default” cultures; Cheryan and Markus, 2020).”
BackgroundYou reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, A. S., Masoud, M., Mitchell, M., Radev, D., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.3 · Cited by 61 · see citing works
“…shown to disproportionately produce harmful content when addressing certain topics (Gehman et al., 2020; Lin et al., 2021) and demograph-028 ics (Sheng et al., 2019; Liang et al., 2021; Dev et al., 2021a)—in part due to the training data used (Dunn, 2020; Gao et al., 2020; Bender et al., 2021),…”
Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 66 · see citing works
“In addition, incorporating counternarratives or examples that challenge stereotypes within the prompts can help counteract the biases that may be present in the training data [167].”
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
Can We Edit Factual Knowledge by In-Context Learning?
Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., & Chang, B. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.296 · Cited by 54 · see citing works
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53 · see citing works
Fair Generative Modeling via Weak SupervisionPreprint
Choi, K., Grover, A., Singh, T., Shu, R., & Ermon, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12008 · Cited by 54 · see citing works
Changing the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51 · see citing works
“The social biases in NLP models are also attracting more attention (Sheng et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020).”
BackgroundAutomated Storytelling via Causal, Commonsense Plot Ordering
Ammanabrolu, P., Cheung, W., Broniec, W., & Riedl, M. O. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16733 · Cited by 19 · see citing works
“Namely, it is prone to echoing the biases present in the dataset (Sheng et al. 2019).”
BackgroundPrompted Opinion Summarization with GPT-3.5
Bhaskar, A., Fabbri, A. R., & Durrett, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.591 · Cited by 59 · see citing works
“Recent work has noted that language models may be susceptible to learning biases from training data (Sheng et al., 2019; Wallace et al., 2019; Shwartz et al., 2020), and this phenomenon has also been observed for GPT-3.”
BackgroundMarasović, A., Bhagavatula, C., Park, J. S., Bras, R. J. L., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 52 · see citing works
“Pretraining on this corpus has caused degenerate and biased behaviour of GPT-2 (Sheng et al., 2019; Wallace et al., 2019; Gehman et al., 2020, among others).”
BackgroundSpeculative Propositions for Digital Writing Under the New Autonomous Model of Literacy
Robinson, B. (2022) · Postdigital Science and Education · doi:10.1007/s42438-022-00358-5 · Cited by 51 · see citing works
“For example, Sheng et al. (2019) showed how NLP systems can reinforce existing social prejudices (e.g., racism) due to the pervasiveness of such prejudices in the data corpora on which the models are trained.”
BackgroundHagendorff, T. (2021) · Minds and Machines · doi:10.1007/s11023-021-09573-8 · Cited by 47 · see citing works
“…violate social norms, discriminate against certain groups of people or perpetuate biases that are incorporated into training data (all of which is especially precarious in open domain conversations) (West et al. 2019; Danaher 2018; Silvervarg et al. 2012; Sheng et al. 2019; Bolukbasi et al. 2016).”
BackgroundMeasuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals
Nozza, D., Bianchi, F., Lauscher, A., & Hovy, D. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.4 · Cited by 47 · see citing works
“…2016; Gonen and Goldberg, 2019; Lauscher et al., 2020; Bianchi et al., 2021), and unfairly biased task-specific models, e.g., for natural language inference (e.g., Dev et al., 2020), natural language generation (e.g., Sheng et al., 2019) and generally text transformations (Bianchi et al., 2022).”
BackgroundProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning
Boratko, M., Li, X. L., O’Gorman, T., Das, R., Le, D., & McCallum, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.85 · Cited by 48 · see citing works
“Studying the bias in such datasets, and natural stereotypical biases which pre-trained language models have been shown to have (Sheng et al., 2019), would be a valuable topic of future work.”
BackgroundGroenwold, S., Ou, L.-H., Parekh, A., Honnavalli, S., Levy, S., Mirza, D., & Wang, W. Y. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-main.473 · Cited by 100 · see citing works
“io/en/dev its widespread use as an off-the-shelf sentiment classifier, such as in Sheng et al. (2019).”
BackgroundMethodMurgia, E., Abbasiantaeb, Z., Aliannejadi, M., Huibers, T. W. C., Landoni, M., & Pera, M. S. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3563359.3597399 · Cited by 45 · see citing works
“Moreover, model bias [33, 43] and hallucination [24] are two well-known problems of generative models which can affect all groups of users but would be harder for young users to detect and combat them [30, 35, 37].”
BackgroundUnlearning Bias in Language Models by Partitioning Gradients
Yu, C., Jeoung, S., Kasi, A., Yu, P., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.375 · Cited by 46 · see citing works
“…the words in sentences are the root of all the information flowing through language models, static word embeddings were the first target for debiasing (Bolukbasi et al., 2016; Zhao et al., 2018b; Sheng et al., 2019; Nangia et al., 2020; Dev et al., 2019; Karve et al., 2019; Zhang et al., 2018).”
BackgroundCan ChatGPT Assess Human Personalities? A General Evaluation Framework
Rao, H., Leung, C., & Miao, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.84 · Cited by 56 · see citing works
Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., & Irving, G. (2021) · arXiv · doi:10.48550/arxiv.2103.14659 · Cited by 45 · see citing works
“Sheng et al. (2019) uses prompts containing a certain demographic group, to attempt to measure the regard for that group, using sentiment scores as a proxy metric for the regard, and they build a classifier to detect the regard given to a group.”
MethodBhat, A., Agashe, S., Oberoi, P., Mohile, N., Jangir, R., & Joshi, A. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584060 · Cited by 47 · see citing works
“Text suggestions based on language models have been known to inherit biases and dominant ‘views’ from the text corpus used to train the language model [22, 40, 44, 56].”
BackgroundMERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
Chakrabarty, T., Zhang, X., Muresan, S., & Peng, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.336 · Cited by 51 · see citing works
“Second, although we use language models trained on data collected from the Web, which have been shown to have issues with bias and abusive language (Sheng et al., 2019; Wallace et al., 2019), the inductive bias of our models should limit in-advertent negative impacts.”
MethodSelf-citationMelhárt, D., Togelius, J., Mikkelsen, B., Holmgård, C., & Yannakakis, G. N. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3276425 · Cited by 54 · see citing works
“Ofcourse some more obvious errors are easier to catch with simple pruning, but computational models can also encode biases—like gender biases or offensive stereotypes—which in turn result in harmful content [72], [73] unintended by the designer.”
BackgroundBridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 57 · see citing works
“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models
Shwartz, V., Rudinger, R., & Tafjord, O. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.556 · Cited by 41 · see citing works
““women can’t be presidents”) (May et al., 2019; Sheng et al., 2019).”
BackgroundInfluentialAnticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44 · see citing works
“This problem is shared by generative language models, as discussed in Bender et al. (2021), and shown in Sheng et al. (2019); Nozza et al. (2021).”
BackgroundShen, T., Li, J., Bouadjenek, M. R., Mai, Z., & Sanner, S. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103139 · Cited by 49 · see citing works
Ovalle, A., Goyal, P., Dhamala, J., Jaggers, Z., Chang, K.-W., Galstyan, A., Zemel, R., & Gupta, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594078 · Cited by 43 · see citing works
“TGNB Harm Evaluations in LLMs Gender bias evaluation methods include toxicity measurements and word co-occurrence in OLG [23, 25, 37, 40, 59, 61].”
BackgroundInfluentialSelf-citationSalinas, A., Shah, P., Huang, Y., McCormack, R., & Morstatter, F. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623257 · Cited by 45 · see citing works
“LLMs inadvertently reflect and perpetuate biases in their training data[4, 6, 29].”
BackgroundWe’re Afraid Language Models Aren’t Modeling Ambiguity
Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.51 · Cited by 63 · see citing works
Do Neural Language Models Overcome Reporting Bias?
Shwartz, V., & Choi, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.605 · Cited by 43 · see citing works
“The results in this paper are in line with prior work that showed that LMs amplify social bias (May et al., 2019; Sheng et al., 2019) and knowledge about named entities that are prominent in the corpus (Shwartz et al.”
ResultA taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41 · see citing works
Wolfe, R., Yang, Y., Howe, B., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594072 · Cited by 42 · see citing works
“[70] find that language models such as GPT-2 [60] output text showing low "regard" for women and gender minorities, including sexualizing biases in text output.”
BackgroundSafeguarding human values: rethinking US law for generative AI’s societal impacts
Cheong, I., Caliskan, A., & Kohno, T. (2024) · AI and Ethics · doi:10.1007/s43681-024-00451-4 · Cited by 39 · see citing works
“…well-intended users’ requests, they produce biased content (e.g., sexually objectified images of women [2], biased judgment against LGBTQIA+ people [3]); makes false claims about certain individuals [4] by deviating from their training data (often called hallucinating [5]); and helps spread…”
Si, W. M., Michael Backes, A., Blackburn, J., Cristofaro, E. D., Stringhini, G., Zannettou, S., & Zhang, Y. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560599 · Cited by 47 · see citing works
“, by removing “contaminated” information from models [17, 69, 71] or prevent inappropriate text generation [20, 53].”
Background
- Authors
- Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N.
- Affiliations
- University of Southern California (US)
- University of California, Los Angeles (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3405-3410
- OpenAlex
- W2971307358
- Cited by
- 431
- References
- 17
- Citations by year
- 2026: 36 · 2025: 73 · 2024: 41 · 2023: 93 · 2022: 64 · 2021: 87 · 2020: 31 · 2019: 3
- FWCI
- 50.46
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- gender bias, gender stereotypes, gendered language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
