CoChat
Sign Up
Cited by 121Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionNatural Language Processing Techniques

RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models

Open access
Barikeri, Soumya, Lauscher, Anne, Vulić, Ivan, Glavaš, Goran·2021·Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
DOI 10.18653/v1/2021.acl-long.151
Open this paper in CoChat — free accountOpen PDF

Abstract

Soumya Barikeri, Anne Lauscher, Ivan Vulić, Goran Glavaš. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.acl-long.151

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RedditBias introduces a conversational dataset generated from Reddit conversations to assess stereotypes between dominant and minoritized groups along gender, race, religion, and queerness.

    “RedditBias (Barikeri et al. 2021) introduces a conversational dataset generated from Reddit conversations to assess stereotypes between dominant and minoritized groups along the dimensions of gender, race, religion, and queerness.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Barikeri et al. 2021

  • RedditBias contains 11,873 sentences constructed by querying Reddit for comments with pre-specified demographic and descriptor words, with human annotation for negative stereotypes.

    “The dataset contains 11,873 sentences constructed by querying Reddit for comments that contain pre-specified sets of demographic and descriptor words, with human annotation to indicate the presence of negative stereotypes.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Barikeri et al. 2021

  • For RedditBias evaluation, counterfactual sentence pairs are formed by replacing demographic terms with alternative groups.

    “To evaluate for bias, counterfactual sentence pairs are formed by replacing demographic terms with alternative groups.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Barikeri et al. 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in con- textualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33-39, Florence, Italy. Associa- tion for Computational Linguistics. · DOI · Paper page
  2. Emily M. Bender, Dirk Hovy, and Alexandra Schofield. 2020. Integrating ethics into the NLP curriculum. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 6-9, Online. Association for Com- putational Linguistics. · DOI
  3. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5454- 5476, Online. Association for Computational Lin- guistics. · DOI · Paper page
  4. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Pro- ceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 4356-4364, Red Hook, NY, USA.
  5. Shikha Bordia and Samuel R. Bowman. 2019. Identify- ing and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Work- shop, pages 7-15, Minneapolis, Minnesota. Associ- ation for Computational Linguistics. · DOI · Paper page
  6. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašić. 2018. MultiWOZ -a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 5016-5026, Brus- sels, Belgium. Association for Computational Lin- guistics. · DOI · Paper page
  7. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
  8. Stefania Degaetano-Ortlieb. 2018. Stylistic variation over 200 years of court proceedings according to gender and social class. In Proceedings of the Sec- ond Workshop on Stylistic Variation, pages 1-10, New Orleans. Association for Computational Lin- guistics. · DOI
  9. Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Sriku- mar. 2020. On measuring and mitigating biased in- ferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 7659-7666.
  10. Sunipa Dev and Jeff Phillips. 2019. Attenuating bias in word vectors. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 879- 887. PMLR.
  11. Catherine D'Ignazio and Lauren F Klein. 2020. The power chapter. In Data Feminism. The MIT Press. · DOI
  12. Emily Dinan, Angela Fan, Adina Williams, Jack Ur- banek, Douwe Kiela, and Jason Weston. 2020a. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 8173-8188, On- line. Association for Computational Linguistics. · DOI
  13. Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020b. Multi- dimensional gender bias classification. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 314-331, Online. Association for Computational Linguistics. · DOI
  14. Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding undesirable word embedding associations. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1696-1705, Florence, Italy. Associa- tion for Computational Linguistics. · DOI
  15. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 609-614, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI · Paper page
  16. Matthew Henderson, Blaise Thomson, and Jason D. Wiliams. 2014. The Second Dialog State Tracking Challenge. In Proceedings of SIGDIAL, pages 263- 272.
  17. Peter Henderson, Koustuv Sinha, Nicolas Angelard- Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123-129.
  18. Saket Karve, Lyle Ungar, and João Sedoc. 2019. Con- ceptor debiasing of word representations evaluated on WEAT. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 40-48, Florence, Italy. Association for Com- putational Linguistics. · DOI
  19. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR 2015.
  20. Anne Lauscher and Goran Glavaš. 2019. Are we con- sistently biased? multidimensional analysis of bi- ases in distributional word vectors. In Proceedings of the Eighth Joint Conference on Lexical and Com- putational Semantics (*SEM 2019), pages 85-91, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
  21. Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2020a. A general framework for im- plicit and explicit debiasing of distributional word vector spaces. volume 34, pages 8131-8138. Associ- ation for the Advancement of Artificial Intelligence (AAAI). · DOI
  22. Anne Lauscher, Rafik Takieddin, Simone Paolo Ponzetto, and Goran Glavaš. 2020b. AraWEAT: Multidimensional analysis of biases in Arabic word embeddings. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 192- 199, Barcelona, Spain (Online). Association for Computational Linguistics.
  23. Nayeon Lee, Andrea Madotto, and Pascale Fung. 2019. Exploring social bias in chatbots using stereotype knowledge. In Proceedings of the 2019 Workshop on Widening NLP, pages 177-180, Florence, Italy. Association for Computational Linguistics.
  24. Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. 2020a. Towards debiasing sen- tence representations. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 5502-5515, Online. Association for Computational Linguistics. · DOI
  25. Sheng Liang, Philipp Dufter, and Hinrich Schütze. 2020b. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5082-5093, Barcelona, Spain (Online). International Committee on Computational Linguistics. · DOI
  26. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020a. Does gender matter? towards fairness in dialogue systems. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 4403-4416, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics. · DOI
  27. Haochen Liu, Wentao Wang, Yiqi Wang, Hui Liu, Zi- tao Liu, and Jiliang Tang. 2020b. Mitigating gender bias for neural dialogue generation with adversarial learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 893-903, Online. Association for Computational Linguistics. · DOI
  28. Katherine McCurdy and Oguz Serbetci. 2020. Gram- matical gender associations outweigh topical gen- der bias in crosslinguistic word embeddings. arXiv preprint arXiv:2005.08864. · arXiv
  29. Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neu- ral belief tracker: Data-driven dialogue state track- ing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1777-1788.
  30. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
  31. Thomas V Pollet and Leander van der Meij. 2017. To remove or not to remove: the impact of outlier handling on significance testing in testosterone data. Adaptive Human Behavior and Physiology, 3(1):43- 60.
  32. Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. 2019. Reducing gender bias in word-level language models with a gender-equalizing loss func- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Stu- dent Research Workshop, pages 223-228, Florence, Italy. Association for Computational Linguistics. · DOI
  33. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  34. Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5248-5264, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  35. Ibrahim Seaga Shaw. 2012. Stereotypical representa- tions of muslims and islam following the 7/7 london terror attacks: Implications for intercultural com- munication and terrorism prevention. International Communication Gazette, 74(6):509-524.
  36. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3407- 3412, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
  37. Gabriel Stanovsky, Noah A. Smith, and Luke Zettle- moyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics. · DOI
  38. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 1630-1640, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  39. Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Ja- son Weston. 2019. Learning to speak and act in a fantasy text adventure game. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 673-683, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  40. Dina Utami, Timothy Bickmore, Asimina Nikolopoulou, and Michael Paasche-Orlow. 2017. Talk about death: End of life planning with a virtual agent. In International Conference on Intelligent Virtual Agents, pages 441-450. Springer.
  41. Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beu- tel, Emily Pitler, Ellie Pavlick, Jilin Chen, and Slav Petrov. 2020. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032. · arXiv
  42. Kelly Welch. 2007. Black criminal stereotypes and racial profiling. Journal of contemporary criminal justice, 23(3):276-288.
  43. Catherine Yeo and Alyssa Chen. 2020. Defining and evaluating fair natural language generation. In Pro- ceedings of the The Fourth Widening Natural Lan- guage Processing Workshop, pages 107-109, Seat- tle, USA. Association for Computational Linguis- tics. · DOI
  44. Koichiro Yoshino, Chiori Hori, Julien Perez, Luis Fer- nando D'Haro, Lazaros Polymenakos, Chulaka Gu- nasekara, Walter S Lasecki, Jonathan K Kummer- feld, Michel Galley, Chris Brockett, et al. 2019. Di- alog system technology challenge 7. arXiv preprint arXiv:1901.03461. · arXiv
  45. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270- 278, Online. Association for Computational Linguis- tics. · DOI · Paper page
  46. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cot- terell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629-634, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
  47. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15-20, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page

Cited by

Sorted by citations

Authors
Barikeri, S., Lauscher, A., Vulić, I., & Glavaš, G.
Affiliations
  • University of Mannheim (DE)
  • University of Cambridge (GB)
ORCID
Venue
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2021
Pages
1941-1955
DOI
10.18653/v1/2021.acl-long.151
OpenAlex
W3168901128
Cited by
121
References
5
Citations by year
2026: 9 · 2025: 28 · 2024: 27 · 2023: 32 · 2022: 21 · 2021: 4
FWCI
18.10
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
H2020 European Research Council (648909)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
bias assessment, bias mitigation, conversational AI, online communities, Reddit, algorithmic bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review