CoChat

French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English

Open access
Névéol, Aurélie, Dupont, Yoann, Bezançon, Julien, Fort, Karën · 2022 · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Cited by 38

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting Much work on biases in natural language processing has addressed biases linked to the social and cultural experience of English speaking individuals in the United States.We seek to widen the scope of bias studies by creating material to measure social bias in language models (LMs) against specific demographic groups in France.We build on the US-centered CrowS-pairs dataset to create a multilingual stereotypes dataset that allows for comparability across languages while also characterizing biases that are specific to each country and language.We introduce 1,677 sentence pairs in French that cover stereotypes in ten types of bias like gender and age.1,467 sentence pairs are translated from CrowS-pairs and 210 are newly crowdsourced and translated back into English.The sentence pairs contrast stereotypes concerning underadvantaged groups with the same sentence concerning advantaged groups.We find that four widely used language models (three French, one multilingual) favor sentences that express stereotypes in most bias categories.We report on the translation process, which led to a characterization of stereotypes in CrowS-pairs including the identification of US-centric cultural traits.We offer guidelines to further extend the dataset to other languages and cultural environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.acl-long.583

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Cited by

See all 38

Metadata

Authors
Névéol, A., Dupont, Y., Bezançon, J., & Fort, K.
Venue
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Published
2022
Pages
8521-8531
OpenAlex
W4281621415
Cited by
38
References
8
FWCI
8.73
Type
conference paper
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. https://app.cochat.ai/papers/10.18653/v1/2022.acl-long.583
The paper
Névéol, A., Dupont, Y., Bezançon, J., & Fort, K. (2022). French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2022.acl-long.583

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.