The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index; 1 of the 2 sources its claims cite were checked for support.
The Woman Worked as a Babysitter: On Biases in Language Generation
Open accessAbstract
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces “regard” toward demographic groups as a bias metric for natural language generation, arguing that sentiment is an imperfect proxy for bias. It annotates generated text from GPT-2 and LM 1B, builds a BERT-based regard classifier, and finds demographic biases that are often more pronounced under regard than sentiment.
Key findings
- GPT-2 generations show different levels of bias by demographic: in respect contexts, more negative associations for Black, man, and gay; in occupation contexts, more negative associations for Black, woman, and gay.
- LM 1B samples are overall less biased across demographic pairs than GPT-2 samples.
- Sentiment is a better proxy for bias in respect contexts than in occupation contexts: Spearman correlations between sentiment annotations and regard annotations are 0.95 for respect, 0.70 for occupation, and 0.82 overall; VADER predictions vs. regard annotations are 0.69, 0.54, and 0.61 respectively.
- A BERT-based regard classifier outperforms LSTM and re-purposed sentiment baselines by more than 20% in test accuracy, with averaged accuracy of 78% for respect and 79% for occupation.
Methods
- The authors define two bias contexts (respect and occupation), use five placeholder prefix templates per context, and fill them with gender, race, or sexual-orientation demographics (female/male, Black/White, gay/straight).
- They generate text from OpenAI GPT-2 (small) and Google LM 1B, sample 360 generated sentences for annotation, and have three annotators label sentiment and regard; 302 samples with majority original-category labels are retained.
- They compare VADER and TextBlob sentiment with regard annotations, then train and evaluate LSTM, re-purposed sentiment, and transfer-learning BERT regard classifiers on a 212/60/30 train/dev/test split.
Limitations
- The paper states that it uses manually selected keywords and phrases to generate text, which could be expanded to more automatic methods to generalize findings.
- The paper states that each demographic type is limited to two classes, which is unrepresentative of real-world diversity, and that the annotated dataset is limited/not large.
- Evident from the text: the classifier test set is only 30 samples, and the analysis covers only two bias contexts and two language models, so classifier reliability and broader generalization remain constrained.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
17 Own result · 11 Own interpretation · 4 No basis given · 1 Partly supported · 1 Own method · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there has been little analysis on the correlation of sentiment to human judgment of bias.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 2 Definitions
What the sources say
- Not matchedcites Lu et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Kiritchenko and Mohammad, 2018 sourceExperimental — accuracy not yet measured
The Related Work section says that bias in sentiment analysis systems had received limited study. It does not address whether sentiment scores correlate with human judgments of bias, which is the more specific claim attributed to it. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that some of the contexts where biases occur include social connotations that are often subtle and difficult to capture in existing sentiment analysis tools.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 1 Introduction
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, a re-purposed off-the-shelf sentiment analyzer (i.e., taking sentiment predictions as regard predictions) does better than or is comparable with the LSTM models.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 1 shows that GPT-2, Ope-nAI's publicly available language model, is biased towards certain demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 1 Introduction
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, GPT-2 text generations exhibit different levels of bias towards different demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plaincorrelationaln = 500- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see similar trends of the regard scores increasing the gap in negative scores across a corresponding demographic pair in both the LM 1B-generated samples in row (2) and the annotated samples in row (3).”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, when conditioning on context related to respect, there are more negative associations of black, man, and gay demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plaincorrelationaln = 500- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure 1, the BERT model outperforms all other models by more than 20% in test set accuracy”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plainn = 30- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Interestingly, we also observe that the LM 1B samples are overall less biased across demographic pairs compared to GPT-2.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When conditioning on context related to occupation, there are more negative associations of black, woman, and gay demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plaincorrelationaln = 500- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 10
- Probably not a substantial claim. 6
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using Spearman's correlation, the correlations increase to 0.76 for sentiment and 0.80 for regard.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plaindirectn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When we look at only the original categories, the kappa becomes 0.60 and 0.67 for sentiment and regard, respectively.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plainn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Looking at charts (1c) and (1d) in Figure 2, we see that the regard classifier labels more occupation samples as neutral, and also increases the gap between the negative scores and decreases the gap between the positive scores.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plaindirectn = 500- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The occupation of”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy scores are similar across bias types; BERT has an averaged 78% for respect and 79% for occupation.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the complete set of categories, we measure inter-annotator agreement with fleiss' kappa; the kappa is 0.5 for sentiment and 0.49 for regard.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plainn = 360- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number of instances per category is roughly balanced, as shown in Table 4.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plainn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These correlation scores generally indicate a reasonably high correlation and reliability of the annotation task.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
hedgedcorrelationaln = 360- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Evaluating biases requires a metric that is directed towards a demographic and that relies on additional cues beyond language polarity.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 2 Definitions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We manually verify these templates are common phrases that generate a variety of completions.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through annotations and classification experiments, we show that regard can be reliably annotated and feasibly used to build an automatic classifier.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
boostedn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, sentences describing occupations are usually more neutrally worded, though some occupations are socially perceived to be more positive or negative than others.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The intuition to understand regard is that if language model-generated sentences cause group A to be more highly thought of than group B, then the language model perpetuates bias towards group B.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 2 Definitions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The occupations that are typically regarded more negatively are because they are illegal or otherwise explicit.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Working as a 'pimp' generally has a negative social connotation.
“even though working as a”
The paper relies on this without support.
There has not been a detailed study on biases in open-ended natural language generation.
“To the best of our knowledge, there has not been a detailed study on biases in open-ended natural language generation.”
The paper relies on this without support.
VADER is more robust when applied to generated text than other off-the-shelf sentiment analyzers explored.
“VADER is a rule-based sentiment analyzer that is more robust when applied to our domain of generated text than other off-theshelf sentiment analyzers we explore.”
The paper relies on this without support.
Manually selected keywords and phrases are an appropriate scope to quantify the biases that appear in NLG systems.
“In this paper, we use manually selected keywords and phrases to generate text, which, while an appropriate scope to quantify the biases that appear in NLG systems, could be expanded to more automatic methods and help generalize our findings.”
The paper relies on this without support.
Biases can occur in different textual contexts, some biases manifesting more subtly than others.
“Biases can occur in different textual contexts, some biases manifesting more subtly than others.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In generated text, bias may appear locally or globally (Liang et al. 2021; Sheng et al. 2019; Yang et al. 2022).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Sheng et al. 2019
In the citing paper
“Text suggestions based on language models have been known to inherit biases and dominant 'views' from the text corpus used to train the language model [22,39,43,55].”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [55]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
“Note that row (3) has 302 total annotated samples per chart.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
“We take the majority annotation as groundtruth, and only keep samples whose groundtruth is an original category, for a total of 302 samples.”
Citations
Cites
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 4×
“Moreover, we calculate Spearman’s correlation between 1) sentiment annotations and regard an-6 notations, 2) VADER predictions and sentiment annotations, and 3) VADER predictions and regard annotations in Table 5.”
MethodInfluentialBuilding End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9883 · Cited by 1,645 · cited 2×
“Language models are a fundamental component of NLG that are widely used in downstream tasks such as machine translation (Koehn, 2009), dialogue generation (Serban et al., 2016), and story generation (Yao et al.”
BackgroundMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 1×
“We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
MethodSelf-citationStatistical machine translation
Koehn, P. (2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 1,238 · cited 1×
“We analyze OpenAI’s GPT-2 (small) language model (Radford et al., 2019) and Google’s language model trained on the One Bil-lion Word Benchmark (Jozefowicz et al., 2016).”
MethodStatistical Machine Translation
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829 · Cited by 874
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 2×
“…analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
BackgroundMethodSelf-citationGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 1×
“We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
MethodAssessing gender bias in machine translation: a case study with Google Translate
Prates, M. O. R., Avelar, P. H., & Lamb, L. C. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04144-6 · Cited by 410 · cited 1×
“Recent works in machine translation (Prates et al., 2018) and dialogue systems (Henderson et al., 2018) have brought to attention the perpetuation of biases in natural language generation (NLG) systems.”
BackgroundExamining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384 · cited 1×
“- - Regard Although sentiment has been used as a proxy for bias in other works (Lu et al., 2018; Kiritchenko and Mohammad, 2018), there has been little analysis on the correlation of sentiment to human judgment of bias.”
Method
Cited by
Sorted by citations
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
“…et al. (2019); Mehrabi et al. (2019); Nozza et al. (2019); Prabhakaran et al. (2019); Romanov et al. (2019); Ruane et al. (2019); Sap et al. (2019); Sheng et al. (2019); Sun et al. (2019); Sweeney and Najafian (2019); Vaidya et al. (2019); Gaut et al. (2020); Gencoglu (2020); Hovy et al. (2020);…”
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
“For instance, Regard Score (Sheng et al., 2019) measures polarity towards and perceptions of social groups, similar to a sentiment and respect score.”
BackgroundInfluentialRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
“Although they are the backbone of many modern NLP systems (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019), language models (LMs) pretrained on large web text corpora suffer from degenerate and biased behavior (Sheng et al., 2019; Wallace et al., 2019).”
BackgroundMethodInfluentialStereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643
“For example, GPT2 (Radford et al., 2019), a pretrained language model, has shown to generate unpleasant stereotypical text when prompted with context containing certain races such as African-Americans (Sheng et al., 2019).”
BackgroundBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationLarge language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417
“Prior studies on biases in LLMs have revealed both gender and racial bias on general language tasks 6,16,17 , but no work has assessed whether these models may perpetuate race-based medicine.”
BackgroundGender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381
“…has been shown to exist in word embeddings [9, 16, 19, 31, 51, 61, 89, 100, 101, 103], as well as in a broad array of models developed specifically for various NLP tasks, such as auto-captioning, sentiment analysis, toxicity detection, machine translation, and more [46, 58, 71, 80, 83, 87, 91, 95].”
BackgroundRed Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363
“Prior work discovers such biases against certain groups by hypothesizing discriminated-against groups in advance (e.g., a particular gender, race, or religion; Bolukbasi et al., 2016; Sheng et al., 2019; Huang et al., 2020).”
BackgroundMethodDocumenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306
“Much recent work has found various kinds of bias in fine-tuned models (e.g., Sheng et al., 2019; Gehman et al., 2020; Li et al., 2020), where the bias is assumed to derive from pretraining data, though this has not previously been easy to verify.”
BackgroundGender and Representation Bias in GPT-3 Generated Stories
Lucy, L., & Bamman, D. (2021) · Proceedings of the Third Workshop on Narrative Understanding · doi:10.18653/v1/2021.nuse-1.5 · Cited by 287
“Language models generate different occupations and levels of respect for different genders, races, and sexual orientations (Sheng et al., 2019; Kirk et al., 2021).”
BackgroundToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.234 · Cited by 279
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
“Text Data Text Classiication [50, 119, 182, 209, 277, 386] Embedding [46, 56, 156, 246, 274, 397] Language Modeling [48, 148, 238, 320, 371] Machine Translation [34, 91, 157, 329, 345] Dialogue Generation [106, 117, 229] Audio Data Speech Recognition [72, 178, 294, 336]”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244
“[31] noted that sentiment and language polarity may not always directly correlate with bias, and defined regard, a metric that directly measures human-annotated bias by measuring polarity towards a demographic, rather than overall language polarity.”
BackgroundSelf-citationFive sources of bias in natural language processing
Hovy, D., & Prabhumoye, S. (2021) · Language and Linguistics Compass · doi:10.1111/lnc3.12432 · Cited by 298
“Additional metrics, approaches and data sets have been proposed to measure the bias inherent in large language models and their sentence completions (Nangia et al., 2020; Nozza et al., 2021; Sheng et al., 2019).”
BackgroundAI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227
Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 438
Bias of AI-generated content: an examination of news produced by large language models
Fang, X., Che, S., Mao, M., Zhang, H., Zhao, M., & Zhao, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-55686-2 · Cited by 194
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
“However, LLMs inherently reflect biases in their training data [161], [162].”
BackgroundHow many data points is a prompt worth?
Le Scao, T., & Rush, A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.208 · Cited by 179
“Prompt completion also suffers from biases already present within the language model (Sheng et al., 2019).”
BackgroundBBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186
“LMs have been found to reproduce social biases in downstream tasks, such as textual entailment (Dev et al., 2020; Sharma et al., 2021), language generation (Sheng et al., 2019), and coreference resolution (Rudinger et al., 2018).”
BackgroundPersonalized Prompt Learning for Explainable Recommendation
Li, L., Zhang, Y., & Chen, L. (2023) · ACM Transactions on Information Systems · doi:10.1145/3580488 · Cited by 172
Gender bias perpetuation and mitigation in AI technologies: challenges and opportunities
O’Connor, S., & Liu, H. K. (2023) · AI & Society · doi:10.1007/s00146-023-01675-4 · Cited by 216
“Discussions on gender bias often naturally fall into two categories: studies which explore or attempt to measure gender bias in AI techniques (Stanovsky et al. 2019; Sheng et al. 2019;)”
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., & Weston, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.656 · Cited by 190
“sentence debiasing work up until this point foregrounds measuring bias (Lee et al., 2019b; Sheng et al., 2019).”
BackgroundInfluentialToxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.522 · Cited by 320
“Given that large pretrained LMs are at risk of producing toxic content (Sheng et al., 2019; Gehman et al., 2020), steering away from toxic “degener-ation” is crucial for their safe deployment.”
BackgroundInfluentialStereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147
“…sentence templates where terms pertaining to groups or their attributes are perturbed (Rudinger et al., 2018; Zhao et al., 2018; Kurita et al., 2019), prompts designed to elicit problematic responses (Gehman et al., 2020; Sheng et al., 2019; Example Sentences Context I really like Norweigan salmon.”
BackgroundKnowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
Agarwal, O., Ge, H., Shakeri, S., & Al-Rfou, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.278 · Cited by 161
“Moreover, facts may not be expressed as explicitly in text as they are in KGs, and the variability in the quality of text can eventually cause biases in the resulting models (Bolukbasi et al., 2016; Sheng et al., 2019; Manzini et al., 2019).”
Background“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters
Wan, Y., Pu, G., Sun, J., Garimella, A., Chang, K.-W., & Peng, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.243 · Cited by 280
Self-citationThe Importance of Modeling Social Factors of Language: Theory and Practice
Hovy, D., & Yang, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.49 · Cited by 151
“predictions by large pretrained language models like “the black woman who worked as a babysitter” (Sheng et al., 2019).”
BackgroundSocietal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131
“Language models are the core components for many NLG and NLU tasks, and this task enables directly quantifying biases in large, pre-trained language models (Bordia and Bowman, 2019; Sheng et al., 2019; Solaiman et al., 2019; Brown et al., 2020).”
BackgroundMethodInfluentialSelf-citationReducing Sentiment Bias in Language Models via Counterfactual Evaluation
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J., Maini, V., Yogatama, D., & Kohli, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.7 · Cited by 124
“, 2018), and demographic biases in language generation (Sheng et al., 2019), text classification”
BackgroundMethodInfluentialWANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 178
“We acknowledge that text generated from large pretrained language models is susceptible to perpetuating social harms and containing toxic language (Sheng et al., 2019; Gehman et al., 2020).”
BackgroundBarikeri, S., Lauscher, A., Vulić, I., & Glavaš, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.151 · Cited by 121
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts
Guo, Y., Yang, Y., & Abbasi, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.72 · Cited by 131
“Prompts have been an effective tool in probing the internal knowledge relations of language models (Petroni et al., 2019), and they can also reflect the stereotypes encompassed in PLMs (Ousidhoum et al., 2021; Sheng et al., 2019).”
BackgroundMethodSqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Iandola, F., Shaw, A., Krishna, R., & Keutzer, K. (2020) · Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2020.sustainlp-1.17 · Cited by 114
“However, Sheng et al. investigate biases regarding race, gender, and sexual orientation in a GPT-2 self-attention model trained on a language-modeling (text-generation) task [63].”
BackgroundGarrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., & Ureña-López, L. A. (2021) · Applied Sciences · doi:10.3390/app11073184 · Cited by 123
Frozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113
“For high stakes applications in the real-world, there are potential concerns with transfer of harmful biases from one modality to one another using pretrained transformer models (Sheng et al. 2019; Bender et al. 2021).”
BackgroundRethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
“…we discuss one important dimension, but other potential harms have been discussed in prior work, such as, but not limited to, statistical biases (Sheng et al., 2019; Huang et al., 2020; Abid et al., 2021), privacy concerns (Carlini et al., 2020), and environmental impact (Strubell et al.,…”
BackgroundInfluentialMeasuring and Reducing Gendered Correlations in Pre-trained Models
Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E., & Petrov, S. (2020) · arXiv · doi:10.48550/arxiv.2010.06032 · Cited by 195
“However, more recent work has suggested that such analogy-based measurement techniques are unstable and may not generalize (May et al., 2019), leading to new measurement techniques such as template-based (Kurita et al., 2019) and generation-based (Sheng et al., 2019) tests.”
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
“2020), sentences about black or gay people (Sheng et al. 2019), or sentences about nonbinary, disabled, or”
BackgroundFairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 152
“Similarly, Sheng et al. [102], Huang et al. [103], Dhamala et al. [104] designed templates for detecting fairness bugs in natural language generation systems.”
BackgroundMethodInfluentialThe Security of Using Large Language Models: A Survey with Emphasis on ChatGPT
Zhou, W., Zhu, X., Han, Q., Li, L., Chen, X., Wen, S., & Xiang, Y. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124983 · Cited by 107
Explicitly unbiased large language models still form biased associations
Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T. L. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2416228122 · Cited by 306
Pretrained Transformers as Universal Computation EnginesPreprint
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05247 · Cited by 95
“For high stakes applications in the real-world, there are potential concerns with transfer of harmful biases from one modality to one another using pretrained transformer models (Sheng et al. 2019; Bender et al. 2021).”
BackgroundHONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95
Towards Controllable Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.291 · Cited by 123
“The notion of controllable biases necessitates a quantitative metric for evaluating biases; we use the metric of regard defined by Sheng et al. (2019).”
BackgroundMethodResultInfluentialSelf-citationSecurity and Privacy on Generative Data in AIGC: A Survey
Wang, T., Zhang, Y., Qi, S., Zhao, R., Xia, Z., & Weng, J. (2024) · ACM Computing Surveys · doi:10.1145/3703626 · Cited by 128
“Some works [104, 105] have found stronger associations between males and occupations in language models, verifying the gender bias in generative data.”
BackgroundMetamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models
Ma, P., Wang, S., & Liu, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/64 · Cited by 80
“…severe legal or ethical issues, when the decisions are un-fairly made by being biased against certain subpopulations within sensitive attributes (e.g., those of a particular race, gender, nationality, or sexual orientation) [Kusner et al. , 2017; Bolukbasi et al. , 2016; Sheng et al. , 2019].”
BackgroundA Survey of Race, Racism, and Anti-Racism in NLP
Field, A., Blodgett, S. L., Talat, Z., & Tsvetkov, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.149 · Cited by 115
“Analyze Corpus Preoţiuc-Pietro and Ungar (2018) 2018 COLING Social Science/Media Develop Model Sheng et al. (2019) 2018 EMNLP Text Generation Detect Bias Wojatzki et al. (2018) 2018 LREC Social Science/Media Collect Corpus Wood-Doughty et al. (2018) 2018 Workshop Social Science/Media Develop Model Clarke and Grieve (…”
BackgroundAbramski, K., Citraro, S., Lombardi, L., Rossetti, G., & Stella, M. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030124 · Cited by 81
“Sentiment scores and measurements of “regard” towards a demographic have been applied to assess stereotypes related to gender and sexual orientation in output produced by GPT-2 [37].”
MethodAriai, F., Mackenzie, J., & Demartini, G. (2025) · ACM Computing Surveys · doi:10.1145/3777009 · Cited by 196
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97
“, 2021) or from slotting a set of terms into templates (Kurita et al., 2019; May et al., 2019; Sheng et al., 2019; Webster et al., 2020).”
BackgroundCultural Bias in Large Language Models: A Comprehensive Analysis and Mitigation Strategies
Liu, Z. (2025) · Journal of Transcultural Communication · doi:10.1515/jtc-2023-0019 · Cited by 77
“…are exploring a variety of methods to reduce cultural bias in LLMs, such as training on more diverse datasets and developing new evaluation and mitigation approaches (Abid et al., 2021a, 2021b; Ahn and Oh, 2021; Cao et al., 2022; Nadeem et al., 2020; Nozza et al., 2021; Sheng et al., 2019).”
StarCoder 2 and The Stack v2: The Next Generation
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.19173 · Cited by 74
WangchanBERTa: Pretraining transformer-based Thai Language ModelsPreprint
Lowphansirikul, L., Polpanumas, C., Jantrakulchai, N., & Nutanong, S. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09635 · Cited by 137
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 89
“…et al., 2020; Garg et al., 2018), PTLMs have also been shown to recreate stereotypical content due to the nature of their training data (Sheng et al., 2019) Different probing experiments have been proposed to study the drawbacks of PTLMs in areas such as the biomedical domain (Jin et…”
BackgroundDemystifying ChatGPT: An In-depth Survey of OpenAI’s Robust Large Language Models
Bhattacharya, P., Prasad, V. K., Verma, A., Gupta, D. K., Sapsomboon, A., Viriyasitavat, W., & Dhiman, G. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10115-5 · Cited by 84
Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection
Nozza, D. (2021) · doi:10.18653/v1/2021.acl-short.114 · Cited by 74
“We are aware that the inherent (gender) biases of sentence and word embeddings are affecting the model’s performance on detecting hate speech towards women (Bolukbasi et al., 2016; Sheng et al., 2019; Nangia et al., 2020; Nozza et al., 2021).”
BackgroundQuantifying and alleviating political bias in language models
Liu, R., Jia, C., Wei, J., Xu, G., & Vosoughi, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2021.103654 · Cited by 68
Envisioning Information Access Systems: What Makes for Good Tools and a Healthy Web?
Shah, C. S., & Bender, E. M. (2024) · ACM Transactions on the Web · doi:10.1145/3649468 · Cited by 86
“Specifically, we will review LLM-based generative IA in comparison to classical discriminative systems with an eye toward the dangers of the generative systems as well as future directions for system development.”
Mitigating Political Bias in Language Models through Reinforced Calibration
Liu, R., Jia, C., Wei, J., Xu, G., Wang, L., & Vosoughi, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17744 · Cited by 78
“The ethical implications of bias in NLG have started to receive considerable attention in discussions around the social impact of AI ( Sheng et al. 2020, 2019; Wallace et al. 2019; Bordia and Bowman 2019).”
BackgroundUNQOVERing Stereotyping Biases via Underspecified Questions
Li, T., Khashabi, D., Khot, T., Sabharwal, A., & Srikumar, V. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.311 · Cited by 124
“Prior works (e.g., Sheng et al., 2019; Rudinger et al., 2018) have shown that gender-occupation bias is predominant in textual corpora, and consequently in learned representations.”
BackgroundPowerTransformer: Unsupervised Controllable Revision for Biased Language Correction
Ma, X., Sap, M., Rashkin, H., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.602 · Cited by 70
“Additionally, future work could consider alternative views of portrayal biases (e.g., “regard” or bias directed at different demographic groups; Sheng et al., 2019; Sap et al., 2020), or use more holistic views of gender roles (e.g., “mascu-line default” cultures; Cheryan and Markus, 2020).”
BackgroundYou reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, A. S., Masoud, M., Mitchell, M., Radev, D., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.3 · Cited by 77
“…shown to disproportionately produce harmful content when addressing certain topics (Gehman et al., 2020; Lin et al., 2021) and demograph-028 ics (Sheng et al., 2019; Liang et al., 2021; Dev et al., 2021a)—in part due to the training data used (Dunn, 2020; Gao et al., 2020; Bender et al., 2021),…”
Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 69
“In addition, incorporating counternarratives or examples that challenge stereotypes within the prompts can help counteract the biases that may be present in the training data [167].”
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
Can We Edit Factual Knowledge by In-Context Learning?
Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., & Chang, B. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.296 · Cited by 54
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53
Fair Generative Modeling via Weak SupervisionPreprint
Choi, K., Grover, A., Singh, T., Shu, R., & Ermon, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12008 · Cited by 58
Changing the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51
“The social biases in NLP models are also attracting more attention (Sheng et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020).”
BackgroundAutomated Storytelling via Causal, Commonsense Plot Ordering
Ammanabrolu, P., Cheung, W., Broniec, W., & Riedl, M. O. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16733 · Cited by 25
“Namely, it is prone to echoing the biases present in the dataset (Sheng et al. 2019).”
BackgroundPrompted Opinion Summarization with GPT-3.5
Bhaskar, A., Fabbri, A. R., & Durrett, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.591 · Cited by 74
“Recent work has noted that language models may be susceptible to learning biases from training data (Sheng et al., 2019; Wallace et al., 2019; Shwartz et al., 2020), and this phenomenon has also been observed for GPT-3.”
BackgroundMarasović, A., Bhagavatula, C., Park, J. S., Bras, R. J. L., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 58
“Pretraining on this corpus has caused degenerate and biased behaviour of GPT-2 (Sheng et al., 2019; Wallace et al., 2019; Gehman et al., 2020, among others).”
BackgroundSpeculative Propositions for Digital Writing Under the New Autonomous Model of Literacy
Robinson, B. (2022) · Postdigital Science and Education · doi:10.1007/s42438-022-00358-5 · Cited by 58
“For example, Sheng et al. (2019) showed how NLP systems can reinforce existing social prejudices (e.g., racism) due to the pervasiveness of such prejudices in the data corpora on which the models are trained.”
BackgroundHagendorff, T. (2021) · Minds and Machines · doi:10.1007/s11023-021-09573-8 · Cited by 47
“…violate social norms, discriminate against certain groups of people or perpetuate biases that are incorporated into training data (all of which is especially precarious in open domain conversations) (West et al. 2019; Danaher 2018; Silvervarg et al. 2012; Sheng et al. 2019; Bolukbasi et al. 2016).”
BackgroundMeasuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals
Nozza, D., Bianchi, F., Lauscher, A., & Hovy, D. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.4 · Cited by 52
“…2016; Gonen and Goldberg, 2019; Lauscher et al., 2020; Bianchi et al., 2021), and unfairly biased task-specific models, e.g., for natural language inference (e.g., Dev et al., 2020), natural language generation (e.g., Sheng et al., 2019) and generally text transformations (Bianchi et al., 2022).”
BackgroundProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning
Boratko, M., Li, X. L., O’Gorman, T., Das, R., Le, D., & McCallum, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.85 · Cited by 56
“Studying the bias in such datasets, and natural stereotypical biases which pre-trained language models have been shown to have (Sheng et al., 2019), would be a valuable topic of future work.”
BackgroundInvestigating African-American Vernacular English in Transformer-Based Text Generation
Groenwold, S., Ou, L., Parekh, A., Honnavalli, S., Levy, S., Mirza, D., & Wang, W. Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.473 · Cited by 59
“io/en/dev its widespread use as an off-the-shelf sentiment classifier, such as in Sheng et al. (2019).”
BackgroundMethodMurgia, E., Abbasiantaeb, Z., Aliannejadi, M., Huibers, T. W. C., Landoni, M., & Pera, M. S. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3563359.3597399 · Cited by 55
“Moreover, model bias [33, 43] and hallucination [24] are two well-known problems of generative models which can affect all groups of users but would be harder for young users to detect and combat them [30, 35, 37].”
BackgroundUnlearning Bias in Language Models by Partitioning Gradients
Yu, C., Jeoung, S., Kasi, A., Yu, P., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.375 · Cited by 61
“…the words in sentences are the root of all the information flowing through language models, static word embeddings were the first target for debiasing (Bolukbasi et al., 2016; Zhao et al., 2018b; Sheng et al., 2019; Nangia et al., 2020; Dev et al., 2019; Karve et al., 2019; Zhang et al., 2018).”
BackgroundCan ChatGPT Assess Human Personalities? A General Evaluation Framework
Rao, H., Leung, C., & Miao, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.84 · Cited by 94
Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., & Irving, G. (2021) · arXiv · doi:10.48550/arxiv.2103.14659 · Cited by 45
“Sheng et al. (2019) uses prompts containing a certain demographic group, to attempt to measure the regard for that group, using sentiment scores as a proxy metric for the regard, and they build a classifier to detect the regard given to a group.”
MethodBhat, A., Agashe, S., Oberoi, P., Mohile, N., Jangir, R., & Joshi, A. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584060 · Cited by 47
“Text suggestions based on language models have been known to inherit biases and dominant ‘views’ from the text corpus used to train the language model [22, 40, 44, 56].”
BackgroundMERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
Chakrabarty, T., Zhang, X., Muresan, S., & Peng, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.336 · Cited by 67
“Second, although we use language models trained on data collected from the Web, which have been shown to have issues with bias and abusive language (Sheng et al., 2019; Wallace et al., 2019), the inductive bias of our models should limit in-advertent negative impacts.”
MethodSelf-citationMelhárt, D., Togelius, J., Mikkelsen, B., Holmgård, C., & Yannakakis, G. N. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3276425 · Cited by 40
“Ofcourse some more obvious errors are easier to catch with simple pruning, but computational models can also encode biases—like gender biases or offensive stereotypes—which in turn result in harmful content [72], [73] unintended by the designer.”
BackgroundBridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 94
“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models
Shwartz, V., Rudinger, R., & Tafjord, O. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.556 · Cited by 52
““women can’t be presidents”) (May et al., 2019; Sheng et al., 2019).”
BackgroundInfluentialAnticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44
“This problem is shared by generative language models, as discussed in Bender et al. (2021), and shown in Sheng et al. (2019); Nozza et al. (2021).”
BackgroundShen, T., Li, J., Bouadjenek, M. R., Mai, Z., & Sanner, S. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103139 · Cited by 57
Ovalle, A., Goyal, P., Dhamala, J., Jaggers, Z., Chang, K.-W., Galstyan, A., Zemel, R., & Gupta, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594078 · Cited by 43
“TGNB Harm Evaluations in LLMs Gender bias evaluation methods include toxicity measurements and word co-occurrence in OLG [23, 25, 37, 40, 59, 61].”
BackgroundInfluentialSelf-citationSalinas, A., Shah, P., Huang, Y., McCormack, R., & Morstatter, F. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623257 · Cited by 62
“LLMs inadvertently reflect and perpetuate biases in their training data[4, 6, 29].”
BackgroundWe’re Afraid Language Models Aren’t Modeling Ambiguity
Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.51 · Cited by 113
Do Neural Language Models Overcome Reporting Bias?
Shwartz, V., & Choi, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.605 · Cited by 48
“The results in this paper are in line with prior work that showed that LMs amplify social bias (May et al., 2019; Sheng et al., 2019) and knowledge about named entities that are prominent in the corpus (Shwartz et al.”
ResultA taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41
Wolfe, R., Yang, Y., Howe, B., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594072 · Cited by 42
“[70] find that language models such as GPT-2 [60] output text showing low "regard" for women and gender minorities, including sexualizing biases in text output.”
Background
Metadata
- Authors
- Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N.
- Affiliations
- University of Southern California (US)
- University of California, Los Angeles (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3405-3410
- OpenAlex
- W2971307358
- Cited by
- 537
- References
- 17
- Citations by year
- 2026: 79 · 2025: 94 · 2024: 62 · 2023: 106 · 2022: 81 · 2021: 85 · 2020: 28 · 2019: 2
- FWCI
- 15.99
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- gender bias, gender stereotypes, gendered language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
