The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Open accessAbstract
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluating the expanding capabilities of LLMs is a complex research area28,29, with group-specific bias benchmarks shown to be time-consuming to develop and utilize30–33, and the overall field lacking measurement validity and theoretical grounding30,34.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many approaches, for instance, assume some implicitly desirable criterion ( e.g. , a model output should be independent of any social group in the input), but do not explicitly acknowledge or state the normative social values that justify their framework.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Blodgett et al. 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In nearly half of all instances, these datasets contain ambiguities about what stereotypes they capture, raising questions whether they are valid indicators of real-world stereotypes at all.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Blodgett et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“several widely-used evaluation datasets suffer from reliability and validity issues, including ambiguities about whether instances accurately reflect real-world stereotypes, inconsistent treatment of social groups, assumptions of near-perfect understanding of language, and lack of syntactic and semantic diversity (Blodgett et al. 2021; Gupta et al. 2023; Selvam et al. 2023).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Blodgett et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Blodgett et al. (2021) highlight several severe shortcomings of several counterfactual input datasets, namely Winogender, WinoBias, StereoSet, and CrowS-Pairs.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Blodgett et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“At the core of these issues are (1) unclear articulation of the power imbalances, harm, or other stereotype content present in each instance; and (2) inconsistent, invalid, or unrelated perturbations of social groups.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Blodgett et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“These datasets also suffer from some data reliability issues, but to a lesser extent than those discussed in Blodgett et al. (2021) (Liang et al. 2022).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Blodgett et al. (2021)
In the citing paper
“In addition, prompt-based learning with GPT-3 has achieved very strong performances in zero-shot and one-shot settings (i.e. with only zero or one labelled example), without any fine-tuning [7].”
From GPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills (2024) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Austin, J. L. (1975) · doi:10.1093/acprof:oso/9780198245537.001.0001 · Cited by 21,496
Black Feminist Thought: Knowledge, Consciousness, and the Politics of Empowerment.
Brewer, R. M., & Collins, P. H. (1992) · Contemporary Sociology · doi:10.2307/2074808 · Cited by 19,445
A General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858 · cited 2×
“This inductive approach aligns with our exploratory goal of surfacing a range of pitfalls, including many that are not on the community radar and thus likely to be overlooked by a more deductive approach requiring a predefined codebook grounded in existing assumptions (Thomas, 2006).”
MethodClark, H. H. (1996) · doi:10.1017/cbo9780511620539 · Cited by 5,123
Glick, P., & Fiske, S. T. (2001) · American Psychologist · doi:10.1037/0003-066x.56.2.109 · Cited by 2,224
Measurement Validity: A Shared Standard for Qualitative and Quantitative Research
Adcock, R., & Collier, D. (2001) · American Political Science Review · doi:10.1017/s0003055401003100 · Cited by 1,629
Measurement Validity: A Shared Standard for Qualitative and Quantitative Research
Collier, D., & Adcock, R. (2001) · eScholarship (California Digital Library) · Cited by 1,603
Bucholtz, M., & Hall, K. (2005) · Wiley eBooks · doi:10.1002/9780470996522.ch16 · Cited by 1,153
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 2×
“Yet even as such benchmarks are added to popular NLP leaderboards like Super-Glue (Wang et al., 2019), whether these they actually help measure the extent to which NLP systems produce computational harms remains unknown.”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1×
“, 2020), and a lack of consensus about how to conceptualize or operationalize such harms (Blodgett et al., 2020; Jacobs and Wallach, 2021).”
BackgroundSelf-citation
Research works printed but not linked: 1
- David J. Schneider. 2005. The Psychology of Stereotyp- ing. Guilford Press.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“…et al., 2021b), there is a lack of standardization of fairness benchmarks, an understanding of what harms different bias measures in NLP relate to (Blodgett et al., 2020, 2021; Jacobs & Wallach, 2021), and coverage of identities in fluid, comprehensive ways (Cao & Daumé III, 2020; Dev et al.,…”
BackgroundGemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
“Others lack consistency in their definitions of bias, or do not seriously engage with the relevant power dynamics that perpetuate the underlying harm (Blodgett et al., 2021).”
BackgroundEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Metadata
- Authors
- Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H.
- Affiliations
- Microsoft (United States) (US)
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 1004-1015
- OpenAlex
- W3172415559
- Cited by
- 147
- References
- 29
- Citations by year
- 2026: 13 · 2025: 17 · 2024: 21 · 2023: 55 · 2022: 34 · 2021: 7
- FWCI
- 26.22
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- stereotyping, Norwegian language, bias in language models, dataset evaluation, fairness metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
