CoChat
Sign Up
Cited by 644Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionTopic Modeling

StereoSet: Measuring stereotypical bias in pretrained language models

Open access
Nadeem, Moin, Bethke, Anna, Reddy, Siva·2021·Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
DOI 10.18653/v1/2021.acl-long.416
Open this paper in CoChat — free accountOpen PDF

Abstract

Moin Nadeem, Anna Bethke, Siva Reddy. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

AI snapshot

Written from the paper’s full text.

TL;DR

StereoSet introduces a large-scale natural English benchmark of 16,995 Context Association Tests across gender, profession, race, and religion to measure stereotypical bias alongside language modeling ability in pretrained language models. It finds that popular models such as BERT, GPT2, ROBERTA, and XLNET show strong stereotypical biases, and that stronger language modeling correlates with greater bias (Spearman ρ = 0.87).

Key findings

  • StereoSet contains 321 target terms and 16,995 test instances (triplets): 8,498 intrasentence CATs and 8,497 intersentence CATs across four domains.
  • All pretrained models evaluated have higher language modeling scores (lms) than RANDOMLM but also exhibit more stereotypical behavior than RANDOMLM (ss = 50). In Table 4, GPT2-large reaches lms = 89.6 and ss = 62.7; ROBERTA-base has ss = 58.5.
  • Language modeling ability and stereotype score are strongly correlated: Spearman rank correlation ρ = 0.87. The ENSEMBLE model is reported as the most biased model (ss = 62.5 in the text; 62.2 in Table 4).
  • SENTIMENTLM achieves a high stereotype score (ss = 60.8) but a lower language modeling score (lms = 65.1), indicating sentiment can help detect stereotypical associations but is insufficient for distinguishing meaningful from meaningless sentences.

Methods

  • The paper designs the Context Association Test (CAT) with two formats: intrasentence fill-in-the-blank and intersentence discourse-level association. Each instance includes a target context and three associations: stereotype, anti-stereotype, and meaningless.
  • Data were crowdsourced via Amazon Mechanical Turk from workers in the United States. Target terms were curated from Wikidata relations for profession, race, and religion, plus gender terms. Each CAT was written by one worker and validated by five others; CATs were retained when at least three validators agreed on labels, yielding 83% of collected CATs.
  • Models evaluated include BERT, ROBERTA, XLNET, and GPT2. Evaluation uses likelihood-based and pseudo-likelihood-based scoring, with metrics: language modeling score (lms), stereotype score (ss), and idealized CAT score (icat). For intersentence tasks, Next Sentence Prediction heads were trained for models lacking them.

Limitations

  • The paper states that evaluating full language modeling ability is beyond its scope, that StereoSet has no training set by design, and that neutral associations were discarded because crowdworkers had low inter-annotator agreement and often produced implicitly stereotypical sentences.
  • The paper states that achieving ideal StereoSet performance does not guarantee a model is unbiased, that icat is not intended as the sole criterion for model selection, and that further research is needed; it also notes lack of computing resources to train the same model on increasing amounts of corpora.
  • Evident from the text: the dataset is US English and collected only from US crowdworkers, so the stereotypes are tailored to the United States; it covers only four domains; validation retains CATs with as few as three of five agreeing validators; and some reported numbers differ between the narrative and tables (e.g., GPT2-large lms/ss and ENSEMBLE ss), which may complicate interpretation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2021.acl-long.416

Cites

26 of 35 references verified0 citation errors · 9 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 17 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Existing bias evaluations mainly evaluate bias of pretrained language models on a small set of artificial sentences.

    “they mainly evaluate bias of pretrained language models on a small set of artificial sentences, even though these models are trained on natural data”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites May et al., 2019 source
    • cites Kurita et al., 2019 source

      The Introduction and Table 1 describe evaluating BERT bias with a small set of simple template sentences, and the paper discusses an earlier sentence-embedding bias test. But it does not establish that existing bias evaluations mainly use artificial sentences; it also examines bias on a Wikipedia-derived pronoun-resolution dataset. Page numbers are unavailable in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    Contextual word embedding bias studies focus on single word terms and ignore multiword terms like construction worker.

    “Third, they focus on single word terms and ignore multiword terms like construction worker.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites May et al., 2019 source
    • cites Kurita et al., 2019 source

      The Introduction (page unavailable) describes this paper’s bias measure as querying BERT for a particular token and illustrates it with single-word targets and attributes. It does not establish that contextual-embedding bias studies generally ignore multiword terms, or specifically discuss “construction worker.” (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    Current evaluations focus on measuring bias without considering the language modeling ability of a model.

    “current evaluations focus on measuring bias without considering the language modeling ability of a model, which could lead to misleading trust on a model even if it is a poor language model.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites May et al., 2019 source
    • cites Kurita et al., 2019 source

      The Introduction and §5 Real World Implications discuss existing bias-evaluation methods and say they can give inconsistent or unclear results for contextualized embeddings. But the paper does not say those evaluations neglect a model’s language-modeling ability, or that they could inspire misplaced trust in a poor language model. Its proposed method uses masked-token probabilities to measure bias, not to assess overall language-modeling quality. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    Contextual word embedding bias studies study bias of a model independent of its language modeling ability.

    “Lastly, they study bias of a model independent of its language modeling ability which could lead to undeserved trust in a model if it is a poor language model.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites May et al., 2019 source
    • cites Kurita et al., 2019 source

      The Introduction describes measuring BERT’s bias using masked-token predictions and normalizing those predictions against a prior probability. It does not establish that the bias measure is independent of language-modeling ability or discuss undeserved trust in a poor language model. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Contextual word embedding bias studies use artificial context that does not reflect natural usage of a word.

    “First, the context does not reflect the natural usage of a word.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites May et al., 2019 source
    • cites Kurita et al., 2019 source

      The Introduction (page unavailable) says the authors create simple template sentences to measure bias, supporting the claim that the study uses constructed contexts. It does not say that those contexts fail to reflect a word’s natural usage; that criticism is not established by this source. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 17
  • May combine two claims. 14

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Word embeddings such as word2vec and GloVe contain stereotypical biases.

    “The seminal works of Bolukbasi et al. (2016) and Caliskan et al. (2017) show that word embeddings such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) contain stereotypical biases using diagnostic methods like word analogies and association tests.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Bolukbasi et al. (2016) source
    • cites Caliskan et al. (2017) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Aribandi et al. (2021) has shown that pseudo-likelihood has higher variance than likelihood scoring.

    “Moreover, Aribandi et al. (2021) has shown that psuedo-likelihood has higher variance than likelihood scoring.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Aribandi et al. (2021) source

      Section 3.1 reports that CrowS-Pairs has far higher variance than StereoSet. Section 2.2 identifies CrowS-Pairs’ pseudo-likelihood as one possible reason, but also cites its smaller dataset and greater number of categories. The paper does not establish that pseudo-likelihood generally has higher variance than likelihood scoring. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    This filtering results in selecting 83% of the CATs, indicating that there is regularity in stereotypical views among the workers.

    “This filtering results in selecting 83% of the CATs, indicating that there is regularity in stereotypical views among the workers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT2-large is the best performing language model (lms 88.3) followed by GPT2-medium (85.9).

    “Among models, GPT2-large is the best performing language model (88.3) followed by GPT2-medium (85.9).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Multiple annotators use similar attributes.

    “This aligns with expectations and indicates that multiple annotators use similar attributes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    StereoSet contains 4 target domains, 321 target terms and 16,995 test instances (triplets).

    “We crowdsource Stere-oSet, a dataset for associative contexts in English containing 4 target domains, 321 target terms and 16,995 test instances (triplets).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    As model size increases, language modeling ability (lms) increases, and correspondingly its stereotypical score.

    “As the model size increases, we see that its language modeling ability (lms) increases, and correspondingly its stereotypical score.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The NSP classifier achieves 94.6% accuracy with ROBERTA-base, 97.1% with ROBERTA-large, 93.4% with XLNET-base, and 94.1% with XLNET-large.

    “Our NSP classifier achieves an 94.6% accuracy with ROBERTA-base, a 97.1% accuracy with ROBERTA-large, a 93.4% accuracy with XLNET-base and 94.1% accuracy with XL-NET-large.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The NSP classifier obtains 92.5% accuracy on GPT2-small, 94.2% on GPT2-medium, and 96.1% on GPT2-large.

    “Our NSP classifier obtains a 92.5% accuracy on GPT2-small, 94.2% on GPT2-medium, and 96.1% on GPT2-large.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The fine-tuned BERT sentiment classifier achieves 92% test accuracy on the Large Movie Review dataset.

    “Our fine-tuned model achieves an 92% test accuracy on the Large Movie Review dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The NSP classifier results are comparable to Devlin et al. (2019), with each model obtaining 93-98% accuracy against the test set of 3.5 million examples.

    “Our results are comparable to Devlin et al. (2019), with each model obtaining 93-98% accuracy against the test set of 3.5 million examples.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Devlin et al. (2019) source

  • May combine two claims.

    Findingown data

    Collecting neutral contexts is prone to implicit biases and has low inter-annotator agreement.

    “In practice, we found that collecting neutral contexts are prone to implicit biases and has low inter-annotator agreement (Section 4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The ENSEMBLE model is relatively less biased on race than on other domains, with a stereotype score (ss) of 61.8 on the test set.

    “The model is relatively less biased on race than on others (ss = 61.8).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The stereotypes elicited from US crowdworkers are representative of stereotypes in the United States.

  • May combine two claims.

    Findingown data

    Corpora size does not correlate with either lms or ss.

    “Surprisingly, the corpora size does not correlate with either lms or ss.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The results exhibit similar trends on the development and test sets.

    “The results exhibit similar trends on the development and test sets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The sentiment classifier also classifies neutral sentiment such as "My housekeeper is a Mexican" as positive.

    “It is important to note that the classifier also classifies neutral sentiment such as My housekeeper is a Mexican as positive.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The term Muslim is an outlier in domain-wise bias results, despite having established stereotypes indicated by high human agreement.

    “The outlier is Muslim, although it has established stereotypes indicated by the high human agreement (see Table 10).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The stereotypes elicited from US crowdworkers are representative of stereotypes in the United States.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The authors restrict themselves to crowdworkers in USA.

    “We restrict ourselves to crowdworkers in USA since stereotypes could change based on the country.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Stereotypes could change based on the country.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The authors validate the data with additional workers and retain CATs where at least three validators agree on labels.

    “For each context and its associations, we ask five validators to classify each association into a stereotype, an anti-stereotype or a meaningless association.We only retain CATs where at least three validators agree on the labels.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Male names are more likely to be associated with career terms than female names in word embeddings.

    “Caliskan et al. show that male names are more likely to be associated with career terms than female names where the association is measured using embedding similarity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Caliskan et al.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Sentiment alone isn't sufficient to distinguish meaningful and meaningless sentences.

    “However, its language model performance is worse, which is expected, since sentiment alone isn't sufficient to distinguish meaningful and meaningless sentences.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The authors split StereoSet based on target terms: 25% for development and 75% for test, with disjoint terms.

    “We split StereoSet based on the target terms: 25% of the target terms and their instances for the development set and 75% for the test set.We ensure terms in the development set and test set are disjoint.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Some of the collected neutral sentences are not truly neutral; for example, "the chess player was bearded" may inadvertently conceal stereotypes because both chess players and bearded men are commonly seen as wise.

    “Furthermore, we hold that some of these neutral sentences aren't truly neutral; the chess player was bearded may inadvertently conceal stereotypes, since both chess players and bearded men are commonly seen as wise.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All crowdworkers were required to be in the United States and have a HIT acceptance rate greater than 97%.

    “We required all crowdworkers to be in the United States and have a HIT acceptance rate greater than 97%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Each example in StereoSet was written by one crowdworker and validated by four other crowdworkers.

    “We collected this data via Amazon Mechanical Turk (AMT), where each example was written by one crowdworker and validated by four other crowdworkers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The authors train identical Next Sentence Prediction heads on identical datasets for all models.

    “For all models, we train identical Next Sentence Prediction heads on identical datasets (details given in Appendix A.5), and compute the log likelihood that any given target sentence follows the context.”
    Quote in the paperSubstantial claim
26 of 26

  • The paper relies on this without support.

    Stereotypes could change based on the country.

    “We restrict ourselves to crowdworkers in USA since stereotypes could change based on the country.”

  • The authors state this as an assumption.

    Language modeling ability and stereotypical bias are of equal importance for model comparison.

    “we introduce a simple point-estimate called the idealized CAT (icat) score for model comparison assuming equal importance to language modeling ability and stereotypical bias.”

  • The paper relies on this without support.

    The real world distribution of corpora likely reflects stereotypes.

    “The correlation between lms and ss is unfortunate and perhaps un- avoidable as long as we rely on the real world distribution of corpora to train language models since these corpora are likely to reflect stereotypes.”

  • The paper relies on this without support.

    BERT has been trained with a Next Sentence Prediction classification head to provide p(s|c), while the other models have not.

    “While BERT has been trained with a Next Sentence Prediction classification head to provide p(s|c), the other models have not.”

  • The paper relies on this without support.

    The stereotypes elicited from US crowdworkers are representative of stereotypes in the United States.

    “StereoSet is in English and is tailored for the stereotypes that exist in the United States.”

  • The paper relies on this without support.

    The NSP classifier's accuracy on a Wikipedia-derived corpus means it can measure language modeling ability on StereoSet.

    “Our results are comparable to Devlin et al. (2019), with each model obtaining 93-98% accuracy against the test set of 3.5 million examples.”

  • The paper relies on this without support.

    The sentiment classifier's accuracy on movie reviews means it can classify sentiment in StereoSet.

    “Our fine-tuned model achieves an 92% test accuracy on the Large Movie Review dataset.”

  • The paper relies on this without support.

    Human agreement scores in Table 10 are a proxy for most and least biased terms.

    “see Table 10 for humanagreement scores, a proxy for most and least biased terms”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • StereoSet presents for each bias type a context sentence with three options: a stereotype, an anti-stereotype, and an unrelated option.

    “For each type of bias, the dataset presents a context sentence with three options: one with a stereotype, one with a neutral or positive connotation (”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nadeem, Bethke, and Reddy 2021

  • StereoSet presents 16,995 crowdsourced instances measuring race, gender, religion, and profession stereotypes.

    “StereoSet presents 16,995 crowdsourced instances measuring race, gender, religion, and profession stereotypes.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nadeem, Bethke, and Reddy 2021

  • StereoSet evaluates intrasentence bias within a sentence with fill-in-the-blank sentences.

    “StereoSet evaluates intrasentence bias within a sentence with fill-in-the-blank sentences, where the options describe a social group in the sentence context”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nadeem, Bethke, and Reddy 2021

  • StereoSet measures intersentence bias between sentences in a discourse with three continuation options, where the first sentence mentions a social group.

    “It measures intersentence bias between sentences in a discourse with three continuation options, where the first sentence mentions a social group.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Nadeem, Bethke, and Reddy 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Intersentence CAT counts do not add up to the reported total

    Major

    The four domain counts sum to 8,858, not 8,497. This affects the reported size of the intersentence dataset.

    Reported 8497 · recomputed 8858

  • Intrasentence CAT counts do not add up to the reported total

    Major

    The four domain counts sum to 8,853, not 8,498. This affects the reported size of the intrasentence dataset.

    Reported 8498 · recomputed 8853

  • Religion target-term count differs from the listed terms

    Minor

    The statistics table reports 12 religion target terms, but the religion section of the target-term list contains 24.

  • The number of validators differs between sections

    Minor

    The validation procedure says each example was assessed by five validators, while the data statement says it was assessed by four other crowdworkers.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the Association for Computational Linguistics, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.

Cited by

Sorted by citations

Related

Authors
Nadeem, M., Bethke, A., & Reddy, S.
Affiliations
  • Massachusetts Institute of Technology (US)
  • University of California, Santa Barbara (US)
  • University of Cambridge (GB)
ORCID
Reddy, Siva
Venue
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2021
Pages
5356-5371
DOI
10.18653/v1/2021.acl-long.416
OpenAlex
W3019416653
Semantic Scholar
babeda48b10a4d638252118f2238d05a06f4ec55
Cited by
643
References
15
Citations by year
2026: 70 · 2025: 131 · 2024: 98 · 2023: 191 · 2022: 85 · 2021: 49 · 2020: 19
FWCI
95.35
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
stereotypical bias, pre-trained language models, gender stereotypes, language model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review