Cited by 132Ethics and Social Impacts of AINatural Language Processing TechniquesTopic Modeling

Societal Biases in Language Generation: Progress and Challenges

Open access
Sheng, Emily, Chang, Kai-Wei, Natarajan, Prem, Peng, Nanyun·2021·Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

Abstract

Emily Sheng, Kai-Wei Chang, Prem Natarajan, Nanyun Peng. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.acl-long.330

Cites

107 of 108 references verified0 citation errors · 1 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs internalize, spread, and potentially magnify harmful information and social biases existing in crawled training corpora.

    “LLMs have been found to internalize, spread, and potentially magnify harmful information existing in the crawled training corpora, usually, toxic languages, like offensiveness, hate speech, and insults [53], as well as social biases like stereotypes towards people with a particular demographic identity ( e.g. , gender, race, religion, occupation, and ideology) [175].”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [175]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. Saunders and Byrne (2020);
  2. Saunders et al. (2020); Kocmi et al. (2020); Costa-jussà and de Jorge (2020); Costa-jussà et al. (2020);
  3. Choubey et al. (2021); Saunders et al. (2021); Tomalin et al. (2021) Re-writing References
  4. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Al- locative versus representational harms in machine learning. In 9th Annual Conference of the Special Interest Group for Computing, Information and So- ciety.
  5. Kate Crawford. 2017. The trouble with bias. Keynote at NeurIPS.
  6. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi- erarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898.
  7. Anna Farkas and Renáta Németh. 2020. How to mea- sure gender bias in machine translation: Optimal translators, multiple reference points. arXiv preprint arXiv:2011.06445. · arXiv
  8. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  9. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI

Cited by

Sorted by citations

Authors
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N.
Affiliations
  • University of Southern California (US)
  • University of California, Los Angeles (US)
Venue
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Published
2021
Pages
4275-4293
OpenAlex
W3174685870
Cited by
131
References
99
Citations by year
2026: 18 · 2025: 28 · 2024: 18 · 2023: 43 · 2022: 16 · 2021: 8
FWCI
19.25
Type
conference paper
Language
en
Open access
Licence
cc-by
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
language generation, algorithmic fairness, algorithmic bias, natural language generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review