The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index; 1 of the 2 sources its claims cite were checked for support.
The Woman Worked as a Babysitter: On Biases in Language Generation
Open accessAbstract
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces “regard” toward demographic groups as a bias metric for natural language generation, arguing that sentiment is an imperfect proxy for bias. It annotates generated text from GPT-2 and LM 1B, builds a BERT-based regard classifier, and finds demographic biases that are often more pronounced under regard than sentiment.
Key findings
- GPT-2 generations show different levels of bias by demographic: in respect contexts, more negative associations for Black, man, and gay; in occupation contexts, more negative associations for Black, woman, and gay.
- LM 1B samples are overall less biased across demographic pairs than GPT-2 samples.
- Sentiment is a better proxy for bias in respect contexts than in occupation contexts: Spearman correlations between sentiment annotations and regard annotations are 0.95 for respect, 0.70 for occupation, and 0.82 overall; VADER predictions vs. regard annotations are 0.69, 0.54, and 0.61 respectively.
- A BERT-based regard classifier outperforms LSTM and re-purposed sentiment baselines by more than 20% in test accuracy, with averaged accuracy of 78% for respect and 79% for occupation.
Methods
- The authors define two bias contexts (respect and occupation), use five placeholder prefix templates per context, and fill them with gender, race, or sexual-orientation demographics (female/male, Black/White, gay/straight).
- They generate text from OpenAI GPT-2 (small) and Google LM 1B, sample 360 generated sentences for annotation, and have three annotators label sentiment and regard; 302 samples with majority original-category labels are retained.
- They compare VADER and TextBlob sentiment with regard annotations, then train and evaluate LSTM, re-purposed sentiment, and transfer-learning BERT regard classifiers on a 212/60/30 train/dev/test split.
Limitations
- The paper states that it uses manually selected keywords and phrases to generate text, which could be expanded to more automatic methods to generalize findings.
- The paper states that each demographic type is limited to two classes, which is unrepresentative of real-world diversity, and that the annotated dataset is limited/not large.
- Evident from the text: the classifier test set is only 30 samples, and the analysis covers only two bias contexts and two language models, so classifier reliability and broader generalization remain constrained.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 of 2 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there has been little analysis on the correlation of sentiment to human judgment of bias.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 2 Definitions
What the sources say
- Not matchedcites Lu et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Kiritchenko and Mohammad, 2018 sourceExperimental — accuracy not yet measured
The Related Work section says that bias in sentiment analysis systems had received limited study. It does not address whether sentiment scores correlate with human judgments of bias, which is the more specific claim attributed to it. (LLM, conf=0.82)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that some of the contexts where biases occur include social connotations that are often subtle and difficult to capture in existing sentiment analysis tools.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 1 Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, a re-purposed off-the-shelf sentiment analyzer (i.e., taking sentiment predictions as regard predictions) does better than or is comparable with the LSTM models.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 1 shows that GPT-2, Ope-nAI's publicly available language model, is biased towards certain demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 1 Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, GPT-2 text generations exhibit different levels of bias towards different demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see similar trends of the regard scores increasing the gap in negative scores across a corresponding demographic pair in both the LM 1B-generated samples in row (2) and the annotated samples in row (3).”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, when conditioning on context related to respect, there are more negative associations of black, man, and gay demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure 1, the BERT model outperforms all other models by more than 20% in test set accuracy”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Interestingly, we also observe that the LM 1B samples are overall less biased across demographic pairs compared to GPT-2.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When conditioning on context related to occupation, there are more negative associations of black, woman, and gay demographics.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- It may state two separate claims as one. 10
- Probably not a substantial claim. 6
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using Spearman's correlation, the correlations increase to 0.76 for sentiment and 0.80 for regard.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When we look at only the original categories, the kappa becomes 0.60 and 0.67 for sentiment and regard, respectively.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Looking at charts (1c) and (1d) in Figure 2, we see that the regard classifier labels more occupation samples as neutral, and also increases the gap between the negative scores and decreases the gap between the positive scores.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The occupation of”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy scores are similar across bias types; BERT has an averaged 78% for respect and 79% for occupation.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the complete set of categories, we measure inter-annotator agreement with fleiss' kappa; the kappa is 0.5 for sentiment and 0.49 for regard.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number of instances per category is roughly balanced, as shown in Table 4.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Strength:plain· not yet calibratedn = 302Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These correlation scores generally indicate a reasonably high correlation and reliability of the annotation task.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Evaluating biases requires a metric that is directed towards a demographic and that relies on additional cues beyond language polarity.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 2 Definitions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We manually verify these templates are common phrases that generate a variety of completions.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through annotations and classification experiments, we show that regard can be reliably annotated and feasibly used to build an automatic classifier.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, sentences describing occupations are usually more neutrally worded, though some occupations are socially perceived to be more positive or negative than others.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > Annotation task
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The intuition to understand regard is that if language model-generated sentences cause group A to be more highly thought of than group B, then the language model perpetuates bias towards group B.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 2 Definitions
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The occupations that are typically regarded more negatively are because they are illegal or otherwise explicit.”
§ The Woman Worked as a Babysitter: On Biases in Language Generation > 5 Biases in language generation systems
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Working as a 'pimp' generally has a negative social connotation.
“even though working as a”
The paper relies on this without support.
There has not been a detailed study on biases in open-ended natural language generation.
“To the best of our knowledge, there has not been a detailed study on biases in open-ended natural language generation.”
The paper relies on this without support.
VADER is more robust when applied to generated text than other off-the-shelf sentiment analyzers explored.
“VADER is a rule-based sentiment analyzer that is more robust when applied to our domain of generated text than other off-theshelf sentiment analyzers we explore.”
The paper relies on this without support.
Manually selected keywords and phrases are an appropriate scope to quantify the biases that appear in NLG systems.
“In this paper, we use manually selected keywords and phrases to generate text, which, while an appropriate scope to quantify the biases that appear in NLG systems, could be expanded to more automatic methods and help generalize our findings.”
The paper relies on this without support.
Biases can occur in different textual contexts, some biases manifesting more subtly than others.
“Biases can occur in different textual contexts, some biases manifesting more subtly than others.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In generated text, bias may appear locally or globally (Liang et al. 2021; Sheng et al. 2019; Yang et al. 2022).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Sheng et al. 2019
In the citing paper
“Text suggestions based on language models have been known to inherit biases and dominant 'views' from the text corpus used to train the language model [22,39,43,55].”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [55]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Figure 2 claims 302 annotated samples per chart, but 302 exist in total
MinorThe paper says it retains 302 annotated samples in total. Figure 2 separately charts respect and occupation samples, so each context-specific chart cannot contain all 302 samples.
“Note that row (3) has 302 total annotated samples per chart.”
“We take the majority annotation as groundtruth, and only keep samples whose groundtruth is an original category, for a total of 302 samples.”
Citations
Cites
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 4×
“Moreover, we calculate Spearman’s correlation between 1) sentiment annotations and regard an-6 notations, 2) VADER predictions and sentiment annotations, and 3) VADER predictions and regard annotations in Table 5.”
MethodInfluentialBuilding End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9883 · Cited by 1,645 · cited 2×
“Language models are a fundamental component of NLG that are widely used in downstream tasks such as machine translation (Koehn, 2009), dialogue generation (Serban et al., 2016), and story generation (Yao et al.”
BackgroundMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 1×
“We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
MethodSelf-citationStatistical machine translation
Koehn, P. (2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 1,238 · cited 1×
“We analyze OpenAI’s GPT-2 (small) language model (Radford et al., 2019) and Google’s language model trained on the One Bil-lion Word Benchmark (Jozefowicz et al., 2016).”
MethodStatistical Machine Translation
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829 · Cited by 874
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 2×
“…analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
BackgroundMethodSelf-citationGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 1×
“We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
MethodAssessing gender bias in machine translation: a case study with Google Translate
Prates, M. O. R., Avelar, P. H., & Lamb, L. C. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04144-6 · Cited by 410 · cited 1×
“Recent works in machine translation (Prates et al., 2018) and dialogue systems (Henderson et al., 2018) have brought to attention the perpetuation of biases in natural language generation (NLG) systems.”
BackgroundExamining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384 · cited 1×
“- - Regard Although sentiment has been used as a proxy for bias in other works (Lu et al., 2018; Kiritchenko and Mohammad, 2018), there has been little analysis on the correlation of sentiment to human judgment of bias.”
Method
Flagged first, then by citations
Cited by
Sorted by citations
Items 101–200 of 439
Safeguarding human values: rethinking US law for generative AI’s societal impacts
Cheong, I., Caliskan, A., & Kohno, T. (2024) · AI and Ethics · doi:10.1007/s43681-024-00451-4 · Cited by 61
“…well-intended users’ requests, they produce biased content (e.g., sexually objectified images of women [2], biased judgment against LGBTQIA+ people [3]); makes false claims about certain individuals [4] by deviating from their training data (often called hallucinating [5]); and helps spread…”
Natural Language Processing in Marketing
Hartmann, J., & Netzer, O. (2023) · Artificial Intelligence in Marketing · doi:10.1108/s1548-643520230000020011 · Cited by 38
Si, W. M., Michael Backes, A., Blackburn, J., Cristofaro, E. D., Stringhini, G., Zannettou, S., & Zhang, Y. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560599 · Cited by 52
“, by removing “contaminated” information from models [17, 69, 71] or prevent inappropriate text generation [20, 53].”
Background“Nice Try, Kiddo”: Investigating Ad Hominems in Dialogue Responses
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.60 · Cited by 48
“For analysis, Sheng et al. (2019) evaluate societal biases in language generation, Curry and Rieser (2018) study how conversational systems respond to sexual harassment, and Khatri et al. (2018) detect offensive content with a semi-supervised approach.”
BackgroundSelf-citationGDPR and Large Language Models: Technical and Legal Obstacles
Feretzakis, G., Vagena, E., Kalodanis, K., Peristera, P., Kalles, D., & Anastasiou, A. (2025) · Future Internet · doi:10.3390/fi17040151 · Cited by 38
Investigating machine moral judgement through the Delphi experiment
Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. J. L., Liang, J. T., Levine, S., Dodge, J., Sakaguchi, K., Forbes, M., Hessel, J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00969-6 · Cited by 58
A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis
Hosseini-Asl, E., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.58 · Cited by 44
Hartmann, J., Schwenzow, J., & Witte, M. (2023) · arXiv · doi:10.48550/arxiv.2301.01768 · Cited by 37
Ding, Z., Srinivasan, A., Macneil, S., & Chan, J. (2023) · Creativity and Cognition · doi:10.1145/3591196.3593516 · Cited by 35
“Analyzing the bias and stereotype-based categories of harm typically involves consideration of specific social categories at risk of harm from biases, such as gender [84], or disability [52].”
BackgroundNavigating the Ethical Terrain of AI-Generated Text Tools: A Review
Mohamed, Y. A., Mohamed, A. H. H. M., Khanan, A., Bashir, M., Adiel, M. A. E., & Elsadig, M. A. (2024) · IEEE Access · doi:10.1109/access.2024.3521945 · Cited by 44
An evaluation on large language model outputs: Discourse and memorization
de Wynter, A., Wang, X., Sokolov, A., Gu, Q., & Chen, S. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100024 · Cited by 55
“Finally, work has been done in characterizing LLMs in terms of fairness and their potential social impact [11, 13, 14, 19, 27, 45, 47]; this area is even more critical as they become more accessible.”
Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources
Mehrabi, N., Zhou, P., Morstatter, F., Pujara, J., Ren, X., & Galstyan, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.410 · Cited by 35
“our targets into: people, locations, professions, and others and manually come up with 30 templates inspired by the prefix templates for bias in NLG (Sheng et al., 2019).”
BackgroundMethodResultInfluentialIntrinsic bias metrics do not correlate with application bias
Goldfarb-Tarrant, S., Marchant, R., Muñoz Sánchez, R., Pandya, M., & Lopez, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.150 · Cited by 33
“However, we know that bias is still a problem for large contextual embeddings (Zhao et al., 2019, 2020; Gehman et al., 2020; Sheng et al., 2019), so our work remains important.”
BackgroundMetaphor Generation with Conceptual Mappings
Stowe, K., Chakrabarty, T., Peng, N., Muresan, S., & Gurevych, I. (2021) · doi:10.18653/v1/2021.acl-long.524 · Cited by 47
“Although we use language models trained on data collected from the Web, which have been shown to have issues with bias and abusive language (Sheng et al., 2019; Wallace et al., 2019), the inductive bias of our models should limit inadvertent negative impacts.”
MethodSelf-citationGarimella, A., Amarnath, A., KUMAR, K. T. K., Yalla, A. P., Natarajan, A., Chhaya, N., & Srinivasan, B. V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.397 · Cited by 37
“Unstructured data often contain several biases, and natural language processing (NLP) models trained on them learn and sometimes amplify them (Bolukbasi et al., 2016; Kurita et al., 2019; Sheng et al., 2019).”
BackgroundMarkedness in Visual Semantic AI
Wolfe, R., & Caliskan, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533183 · Cited by 37
“Intersectional biases have been discovered in both computer vision [6, 49] and in language models [25, 34, 47, 50].”
BackgroundDebiasing Pre-Trained Language Models via Efficient Fine-Tuning
Gira, M., Zhang, R., & Lee, K. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.8 · Cited by 38
“Sheng et al. (2019) defined and measured a concept of regard and sentiment for GPT-2 output.”
BackgroundAligning Generative Language Models with Human Values
Liu, R., Zhang, G., Feng, X., & Vosoughi, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.18 · Cited by 37
Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models
Ungless, E. L., Roß, B., & Lauscher, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.502 · Cited by 53
Dictionary-based Debiasing of Pre-trained Word Embeddings
Kaneko, M., & Bollegala, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.16 · Cited by 36
“Unfair biases have adversely affected several NLP tasks such as machine translation (Vanmassenhove et al., 2018) and language generation (Sheng et al., 2019).”
BackgroundPipelines for Social Bias Testing of Large Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2022) · doi:10.18653/v1/2022.bigscience-1.6 · Cited by 33
“Then, it is possible to exploit lexicons (Nozza et al., 2021, 2022), or hate speech (Ousidhoum et al., 2021; Sheng et al., 2019) and sentiment classifiers Hutchinson et al. (2020); Huang et al. (2020) to obtain a social bias score from the template-based generated text.”
BackgroundArakawa, R., Yakura, H., & Goto, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581133 · Cited by 29
“For example, stereotypes and negative propagation of particular social groups have been observed in their outputs [9, 19, 64].”
BackgroundReward modeling for mitigating toxicity in transformer-based language models
Faal, F., Schmitt, K., & Yu, J. Y. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03944-z · Cited by 35
Song, D., Gao, S., He, B., & Schilder, F. (2022) · IEEE Access · doi:10.1109/access.2022.3190408 · Cited by 34
gScoreCAM: What Objects Is CLIP Looking At?
Chen, P., Li, Q., Biaz, S., Bui, T., & Nguyen, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26316-3_35 · Cited by 28
They, Them, Theirs: Rewriting with Gender-Neutral EnglishPreprint
Sun, T., Webster, K., Shah, A., Wang, W. Y., & Johnson, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06788 · Cited by 65
“…Gonen and Goldberg, 2019) or whether there is the potential for harm when applied to downstream applications (Blodgett et al., 2020), including coreference resolution (Zhao et al., 2018; Rudinger et al., 2018; Webster et al., 2018) and language generation (Sheng et al., 2019; Nadeem et al., 2020).”
BackgroundStylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Lyu, Y., Liang, P. P., Pham, H., Hovy, E. H., Póczos, B., Salakhutdinov, R., & Morency, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.171 · Cited by 33
“Finally, we performed a qualitative evaluation: we sampled 49 sentence completions using prompts from Sheng et al. (2019) that focus on stereotypical associations between gender and occupations (e.”
MethodInfluentialProceedings of the Third Workshop on Narrative Understanding
(2021) · doi:10.18653/v1/2021.nuse-1 · Cited by 30
Estimating the Personality of White-Box Language ModelsPreprint
Karra, S. R., Nguyen, S. T., & Tulabandhula, T. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4598766 · Cited by 77
Towards Understanding Gender Bias in Relation Extraction
Gaut, A., Sun, T., Tang, S., Huang, Y., Qian, J., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., et al. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.265 · Cited by 26
“Existing studies have revealed gender bias in various NLP tasks (Zhao et al., 2017; Rudinger et al., 2018; Zhao et al., 2018; Dixon et al., 2018; Lu et al., 2018; Kir-itchenko and Mohammad, 2018; Romanov et al., 2019; Sheng et al., 2019; Sun et al., 2019).”
BackgroundSelf-citationExplaining Toxic Text via Knowledge Enhanced Text Generation
Sridhar, R., & Yang, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.59 · Cited by 28
“Following Sheng et al. (2019), we use the predicted target minority corresponding to each input post and a set of prompts to induce biased prompt completions from GPT models.”
MethodGradient-based Constrained Sampling from Language Models
Kumar, S., Paria, B., & Tsvetkov, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.144 · Cited by 57
“Prior work have shown that large pre-trained LMs are at risk of producing toxic content even when given innocuous prompts (Sheng et al., 2019; Gehman et al., 2020).”
BackgroundPosition is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
Neumann, A., Kirsten, E., Zafar, M. B., & Singh, J. P. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732038 · Cited by 47
Evaluating gender bias in large language models in long-term care
Rickman, S. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03118-0 · Cited by 27
“Regard is based on BERT and fine-tuned on data designed to evaluate demographic bias [38].”
Looking for a handsome carpenter! Debiasing GPT-3 job advertisements
Borchers, C., Gala, D., Gilburt, B., Oravkin, E., Bounsi, W., Asano, Y. M., & Kirk, H. (2022) · Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.22 · Cited by 29
Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
Lam, M. S., Ma, Z., Li, A., Freitas, I., Wang, D., Landay, J. A., & Bernstein, M. S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581290 · Cited by 31
“Additionally, as with any technology that builds on top of large language models and other pretrained models, sketch models will be subject to biases embedded in these models [36, 56].”
BackgroundArtificial intelligence for global health: cautious optimism with safeguards
Pujari, S., Reis, A. A., Zhao, Y., Alsalamah, S., Serhan, F., Reeder, J. C., & Labrique, A. B. (2023) · Bulletin of the World Health Organization · doi:10.2471/blt.23.290215 · Cited by 24
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models
Xie, Z., & Lukasiewicz, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.876 · Cited by 41
Investigating Societal Biases in a Poetry Composition SystemPreprint
Sheng, E., & Uthus, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02686 · Cited by 26
“The annotations use sentiment annotation guidelines described by Sheng et al. (2019).”
BackgroundMethodSelf-citationJackson, H. (2023) · Media War & Conflict · doi:10.1177/17506352231178148 · Cited by 23
“Studies have found persistent sexist, racist, and Islamophobic bias in highly-used natural language processing toolkits (Abid et al., 2021; Bolukbasi et al., 2016; Bordia and Bowman, 2019; Lu et al., 2020; Nadeem et al., 2020; Shearer et al., 2019; Sheng et al., 2019).”
BackgroundFAtiMA Toolkit: Toward an Accessible Tool for the Development of Socio-emotional Agents
Mascarenhas, S. F., Guimarães, M., Prada, R., Santos, P. A., Dias, J., & Paiva, A. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3510822 · Cited by 23
“Dataheavy systems are prone to echoing the biases present in the datasets [59], limiting their ability to produce generic and independent results.”
BackgroundThe Limits of Word Level Differential Privacy
Mattern, J., Weggenmann, B., & Kerschbaum, F. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.65 · Cited by 49
The workweek is the best time to start a family – A Study of GPT-2 Based Claim Generation
Gretz, S., Bilu, Y., Cohen-Karlik, E., & Slonim, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.47 · Cited by 29
“In addition, GPT-2 text generations have been shown to exhibit different levels of bias towards different demographics (Sheng et al., 2019).”
BackgroundReducing Sentiment Bias in Language Models via Counterfactual EvaluationPreprint
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J. W., Maini, V., Yogatama, D., & Kohli, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03064 · Cited by 98
“, 2018), and demographic biases in language generation (Sheng et al., 2019), text classification”
BackgroundMethodInfluentialEvaluation of African American Language Bias in Natural Language Generation
Deas, N., Grieser, J. A., Kleiner, S., Patton, D. U., Turcan, E., & McKeown, K. R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.421 · Cited by 41
Self-Supervised Test-Time Learning for Reading Comprehension
Banerjee, P., Gokhale, T., & Baral, C. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.95 · Cited by 30
“We believe that this is a possible way to avoid learning spurious correlations or linguistic priors, especially when it comes to socio-cultural and historical biases that have been shown to percolate into models for various NLP tasks (Hendricks et al., 2018; Kurita et al., 2019; Sheng et al., 2019).”
BackgroundHypoGen: Hyperbole Generation with Commonsense and Counterfactual Knowledge
Tian, Y., Sridhar, A. K., & Peng, N. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.136 · Cited by 25
“It is known that pretrained language models could capture the bias reflected in the training data (Sheng et al., 2019; Wallace et al., 2019).”
BackgroundSelf-citationInspecting the Geographical Representativeness of Images from Text-to-Image Models
Basu, A., Babu, R. V., & Pruthi, D. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00474 · Cited by 21
“There is a growing body of work that critically analyzes the outputs of deep learning models in an attempt to discover and measure societal biases for various downstream applications including image classification [41, 22, 46], image captioning [47, 13], language generation [34, 33], face recognition [5, 9], image sea…”
BackgroundEvaluating Biased Attitude Associations of Language Models in an Intersectional Context
Sabbaghi, S. O., Wolfe, R., & Caliskan, A. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604666 · Cited by 25
“[63] in the text output of language models can be traced back to the contextualized embedding space, where the occurrence of the”
BackgroundReducing Non-Normative Text Generation from Language Models
Peng, X., Li, S., Frazier, S., & Riedl, M. O. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.43 · Cited by 28
“…descriptions of behavior (murder, crime, suicide, racist actions, rude behavior, etc.), exhibit biases against certain demographics groups (Sheng et al., 2019; Solaiman et al., 2019), stereotypical biases (Nadeem et al., 2020) or racist text when prompted with trigger phrases (Wallace et…”
BackgroundBeyond Chatbot-K: On Large Language Models, “Generative AI,” and Rise of Chatbots—An Introduction
Goodlad, L. M. E., & Stone, M. (2024) · Critical AI · doi:10.1215/2834703x-11205147 · Cited by 33
Designing Robot Identity: The Role of Voice, Clothing, and Task on Robot Gender Perception
Dennler, N. S., Kian, M., Nikolaidis, S., & Matarić, M. J. (2025) · International Journal of Social Robotics · doi:10.1007/s12369-025-01209-6 · Cited by 24
“Large Language Models (LLMs) have been shown to generate text that perpetuates gender biases [67], homophobia [68], transphobia [69], ableism [70], racism [71], hate speech [72], abusive lan-gauge [73], dehumanization [74], and microaggressions [75] against marginalized populations.”
Aligning to Social Norms and Values in Interactive Narratives
Ammanabrolu, P., Jiang, L., Sap, M., Hajishirzi, H., & Choi, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.439 · Cited by 28
Evidence for Hypodescent in Visual Semantic AI
Wolfe, R., Banaji, M. R., & Caliskan, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533185 · Cited by 19
“[68] show that the text output of language models like GPT-2 contains biases demonstrating low regard for members of marginalized groups, including racial minorities.”
BackgroundTowards an open-domain chatbot for language practice
Tyen, G., Brenchley, M., Caines, A., & Buttery, P. J. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.28 · Cited by 23
“These models, especially ones trained on internet data, are known to produce outputs that are toxic (e.g. Gehman et al., 2020) or that contain harmful biases (e.g. Nadeem et al., 2021; Sheng et al., 2019).”
BackgroundUnderstanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots
Chen, B., Wang, G., Guo, H., Wang, Y., & Yan, Q. (2023) · International Symposium on Research in Attacks, Intrusions and Defenses (RAID) · doi:10.1145/3607199.3607237 · Cited by 26
“[36] study the bias problem in autoregressive language models.”
BackgroundA DSL for Testing LLMs for Fairness and Bias
Morales, S., Clarisó, R., & Cabot, J. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3640310.3674093 · Cited by 22
“Typically, LLMs undergo training using data obtained from web crawls and tend to reproduce and even amplify the unfairness and toxicity found within their data sources [3, 7, 16, 33, 34, 40].”
BackgroundUsing Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching
Katz, A., Gerhardt, M., & Soledad, M. M. (2024) · International Journal of Qualitative Methods · doi:10.1177/16094069241293283 · Cited by 24
“Additionally, models may introduce biases embedded in their training data and underlying algorithms (Mao et al., 2023; Sheng et al., 2019; Vig et al., 2020).”
Uncovering Implicit Gender Bias in Narratives through Commonsense Inference
Huang, T., Brahman, F., Shwartz, V., & Chaturvedi, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.326 · Cited by 24
“Most existing methods on quantifying bias recognize explicit manifestation of bias in the surfacelevel text (Dinan et al., 2020a; Lucy et al., 2020; Gala et al., 2020) or collected human annotations (Sheng et al., 2019; Dinan et al., 2020b).”
Gender Bias in Natural Language Processing and Computer Vision: A Comparative Survey
Bartl, M., Mandal, A., Leavy, S., & Little, S. (2024) · ACM Computing Surveys · doi:10.1145/3700438 · Cited by 23
“Sheng et al. [114] created sentence templates with placeholders for identity terms in contexts related to respect for a person and occupations.”
Using Natural Sentence Prompts for Understanding Biases in Language Models
Alnegheimish, S., Guo, A., & Sun, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.203 · Cited by 16
“Re-027 cently, many works have shown the language mod-028 els are susceptible to biases contained in the train-029 ing dataset (Sheng et al., 2019).”
BackgroundDifferentially Private Language Models for Secure Data Sharing
Mattern, J., Jin, Z., Weggenmann, B., Schölkopf, B., & Sachan, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.323 · Cited by 49
The Hazards of Using ChatGPT: A Call to Action for Medical Education Researchers
Liaw, W. R., Chavez, S., Pham, C., et al. (2023) · PRiMER · doi:10.22454/primer.2023.295710 · Cited by 16
Automated Journalism as a Source of and a Diagnostic Device for Bias in Reporting
Leppänen, L., Tuulonen, H., & Sirén‐Heikel, S. (2020) · Media and Communication · doi:10.17645/mac.v8i3.3022 · Cited by 19
“The language model could then be interrogated for bias, for example using the method of Sheng et al. (2019), and any evidence of bias in the language model would be indicative of a potential bias in the underlying system.”
MethodKoSBI: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applications
Lee, H., Hong, S., Park, J., Kim, T., Kim, G., & Ha, J.-W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track) · doi:10.18653/v1/2023.acl-industry.21 · Cited by 15
AI image generation technology in ophthalmology: Use, misuse and future applications
Phipps, B., Hadoux, X., Sheng, B., Campbell, J. P., Liu, T. Y. A., Keane, P. A., Cheung, C. Y.-L., Chung, T. Y., Wong, T. Y., & van Wijngaarden, P. (2025) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2025.101353 · Cited by 18
“Moreover, generative AI algorithms, while adept at generating smooth, artefact-free images, can sometimes succumb to over-smoothing, which obfuscates the fine textural details that are often crucial for discerning ocular pathologies (Ali et al., 2024; Alvi et al., 2018; Sheng et al., 2019).”
Hartmann, D., Oueslati, A., Staufer, D., Pohlmann, L., Munzert, S., & Heuer, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713998 · Cited by 29
“Second, research revealed an over-moderation of neutral statements mentioning target groups such as “woman” [96], which was coined as systematic offensive stereotyping (SOS) bias [25].”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models
Saeed, M., Ahmadi, N., Nakov, P., & Papotti, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.110 · Cited by 16
“Biases We are aware of (i) the biases and abusive language patterns (Sheng et al., 2019; Zhang et al., 2020; Bender et al., 2021; Liang et al., 2021) that PLMs impose, and (ii) the imperfectness and the biases of our rules as data from Wikipedia has been used to mine the rules and compute their…”
BackgroundMethodShihadeh, J., Ackerman, M., Troske, A., Lawson, N. M., & Gonzalez, E. (2022) · IEEE Global Humanitarian Technology Conference (GHTC) · doi:10.1109/ghtc55712.2022.9910995 · Cited by 18
“Sheng et al. [21] has focused on biases in multiple models including BERT and GPT-2, across gender, race, and sexual orientation in the context of different occupations.”
MethodAnoop, K. R., Gangan, M. P., Deepak, P., & Lajish, V. L. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-4453-6_2 · Cited by 17
“, to generate evaluation sentences that can represent real-world context better than the corpus in [33,52,76,101,128].”
BackgroundSeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models
Jha, A., Davani, A. M., Reddy, C. K. A., Dave, S., Prabhakaran, V., & Dev, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.548 · Cited by 43
Ocampo, N. B., Cabrio, E., & Villata, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.173 · Cited by 21
“In this work, we propose a new framework for generating on-scale close-to-human-like adversarial implicit HS texts using the pre-trained language model (PLM) GPT3 (Brown et al., 2020), which is known to output biased and hateful content (Sheng et al., 2019; Gehman et al., 2020).”
The Origins of Generative AI in Transcription and Machine Translation, and Why That Matters
Stone, M., Goodlad, L. M. E., & Sammons, M. (2024) · Critical AI · doi:10.1215/2834703x-11256853 · Cited by 21
Learning Knowledge Graph-based World Models of Textual EnvironmentsPreprint
Ammanabrolu, P., & Riedl, M. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09608 · Cited by 18
“World modeling for text environments is more suited for domains in which change in the world is affected via language, which mitigates physical risks—downstream lines of work are not directly relevant to robotics—but not cognitive and emotional risks, as any system capable of generating natural language is capable of…”
BackgroundLeashing the Inner Demons: Self-Detoxification for Language Models
Xu, C., He, Z., He, Z., & McAuley, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21406 · Cited by 17
Ashktorab, Z., Hoover, B., Agarwal, M., Dugan, C., Geyer, W., Yang, H., & Yurochkin, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581227 · Cited by 21
“These NLP tasks include question answering [38], relation extraction [25], text classifcation [16], autocomplete generation [57] and machine translation [60].”
BackgroundA Robust Bias Mitigation Procedure Based on the Stereotype Content Model
Ungless, E. L., Rafferty, A., Nag, H., & Roß, B. (2022) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/2022.nlpcss-1.23 · Cited by 16
Tang, K., Zhou, W., Zhang, J., Liu, A., Deng, G., Li, S., Qi, P., Zhang, W., Zhang, T., & Yu, N. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670284 · Cited by 26
“This dimension ensures the benchmarks encompass various facets of gender bias, thus capturing the intricate and multifaceted nature of gendered experiences.”
LOGAN: Local Group Bias Detection by Clustering
Zhao, J., & Chang, K.-W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.155 · Cited by 15
“Other applications such as cross-lingual transfer learning (Zhao et al., 2020) and natural language generation (Sheng et al., 2019) also exhibit unintended biases.”
BackgroundSelf-citationOn Measuring Social Biases in Prompt-Based Multi-Task Learning
Akyürek, A. F., Paik, S., Kocyigit, M. Y., Akbıyık, S., Runyun, Ş. L., & Wijaya, D. T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.42 · Cited by 21
TuringAdvice: A Generative and Dynamic Evaluation of Language Use
Zellers, R., Holtzman, A., Clark, E., Qin, L., Farhadi, A., & Choi, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.386 · Cited by 30
“These qualitative and quantitative results confirm a pattern observed by many others, that large language models like GPT3 often generate explicitly racist and sexist language out-of-the-box Sheng et al., 2019; Gehman et al., 2020; Bender et al., 2021, among others).”
ResultFairPrism: Evaluating Fairness-Related Harms in Text Generation
Fleisig, E., Amstutz, A., Atalla, C., Blodgett, S. L., Daumé III, H., Olteanu, A., Sheng, E., Vann, D., & Wallach, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.343 · Cited by 14
“Sheng et al. (2019) asked annotators to label 360 template-generated examples for “regard,” which measures language polarity toward and social perceptions of demographic groups (i.e., whether an AI system causes “group A to be more highly thought of than group B”).”
BackgroundMethodSelf-citationCiccone, G. (2024) · Heritage · doi:10.3390/heritage7100256 · Cited by 13
Parameter-efficient Modularised Bias Mitigation via AdapterFusion
Kumar, D., Lesota, O., Zerveas, G., Cohen, D., Eickhoff, C., Schedl, M., & Rekabsaz, N. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.201 · Cited by 13
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models
Zhao, J., Fang, M., Shi, Z., Li, Y., Chen, L., & Pechenizkiy, M. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.757 · Cited by 14
Bridging Fairness and Environmental Sustainability in Natural Language Processing
Hessenthaler, M., Strubell, E., Hovy, D., & Lauscher, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.533 · Cited by 16
Debiasing large language models: research opportunities*
Yogarajan, V., Dobbie, G., & Keegan, T. T. (2024) · Journal of the Royal Society of New Zealand · doi:10.1080/03036758.2024.2398567 · Cited by 18
“For the regard score, we utilise pre-trained models 16 from Sheng et al. (2019).”
Parity benchmark for measuring bias in LLMs
Simpson, S., Nukpezah, J., Brooks, K., & Pandya, R. (2024) · AI and Ethics · doi:10.1007/s43681-024-00613-4 · Cited by 12
“For instance, a recent study found that when specific prompt tasks are given to state-of-the-art language models, biases pertaining to demographic groups based on race and gender were apparent [3 ].”
InfluentialSocietal Biases in Language Generation: Progress and ChallengesPreprint
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04054 · Cited by 12
“Language models are the core components for many NLG and NLU tasks, and this task enables directly quantifying biases in large, pre-trained language models (Bordia and Bowman, 2019; Sheng et al., 2019; Solaiman et al., 2019; Brown et al., 2020).”
BackgroundMethodInfluentialSelf-citationValidating Large Language Models with ReLM
Kuchnik, M., Smith, V., & Amvrosiadis, G. (2022) · arXiv · doi:10.48550/arxiv.2211.15458 · Cited by 12
Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks
Xing, W., Li, M., Li, M., & Han, M. (2026) · ACM Computing Surveys · doi:10.1145/3806048 · Cited by 67
“Effective strategies often combine automated techniques with human oversight to probe different failure modes.”
Fair and Argumentative Language Modeling for Computational Argumentation
Holtermann, C., Lauscher, A., & Ponzetto, S. P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.541 · Cited by 16
Exploring Social Biases of Large Language Models in a College Artificial Intelligence Course
Kolisko, S., & Anderson, C. J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26879 · Cited by 12
“A number of probe tasks have been proposed: Rudinger, May, and Van Durme (2017); Sheng et al. (2019); Bordia and Bowman (2019); Lee, Madotto, and Fung (2019); Liu et al. (2019a); May et al. (2019); Nadeem, Bethke, and Reddy (2021); Sotnikova et al. (2021) and others.”
Mitigate Gender Bias Using Negative Multi-task Learning
Gao, L., Zhan, H., Chen, A., & Sheng, V. S. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11368-0 · Cited by 14
Layered Bias: Interpreting Bias in Pretrained Large Language Models
Prakash, N., & Lee, R. K.-W. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.22 · Cited by 15
“Some studies, such as those by (Sheng et al., 2019; Gehman et al., 2020; Dhamala et al., 2021; Kirk et al., 2021; Nozza et al., 2021), employ open-ended text generation.”
BackgroundLewis, A., Dangol, A., Suh, H., Olszewski, A., Fogarty, J. A., & Kientz, J. A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714131 · Cited by 18
“Sheng et al. [75] assessed demographic biases (i.e., race, gender, sexuality) in natural language generation across two language models using pre-trained classifiers.”
ENTRUST: Argument Reframing with Language Models and Entailment
Chakrabarty, T., Hidey, C., & Muresan, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.394 · Cited by 14
“been shown to have issues with bias and abusive language (Sheng et al., 2019; Wallace et al., 2019), the inductive bias of our models should limit inadvertent negative impacts.”
BackgroundMethodAnalyzing Stereotypes in Generative Text Inference Tasks
Sotnikova, A., Cao, Y., Daumé, H., & Rudinger, R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.355 · Cited by 11
“…a variety of tasks including language modeling, coreference resolution, natural language inference, machine translation, and sentiment analysis (Sheng et al., 2019; Rudinger et al., 2018; Lu et al., 2018; Dinan et al., 2019; Rudinger et al., 2017; Kiritchenko and Mohammad, 2018); Blodgett et…”
BackgroundEvaluating Natural Language Inference Models: A Metamorphic Testing Approach
Jiang, M., Bao, H., Tu, K., Zhang, X., & Ding, Z. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00033 · Cited by 11
Metadata
- Authors
- Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N.
- Affiliations
- University of Southern California (US)
- University of California, Los Angeles (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3405-3410
- OpenAlex
- W2971307358
- Cited by
- 537
- References
- 17
- Citations by year
- 2026: 79 · 2025: 94 · 2024: 62 · 2023: 106 · 2022: 81 · 2021: 85 · 2020: 28 · 2019: 2
- FWCI
- 15.99
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- gender bias, gender stereotypes, gendered language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
