CoChat
Sign Up
Cited by 76Hate Speech and Cyberbullying DetectionEthics and Social Impacts of AISentiment Analysis and Opinion Mining

Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation

Open access
Goyal, Nitesh, Kivlichan, Ian D., Rosen, Rachel, Vasserman, Lucy·2022·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3555088
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning models are commonly used to detect toxicity in online conversations. These models are trained on datasets annotated by human raters. We explore how raters' self-described identities impact how they annotate toxicity in online comments. We first define the concept of Specialized Rater Pools: rater pools formed based on raters' self-described identities, rather than at random. We formed three such rater pools for this study - specialized rater pools of raters from the U.S. who identify as African American, LGBTQ, and those who identify as neither. Each of these rater pools annotated the same set of comments, which contains many references to these identity groups. We found that rater identity is a statistically significant factor in how raters will annotate toxicity for identity-related annotations. Using preliminary content analysis, we examined the comments with the most disagreement between rater pools and found nuanced differences in the toxicity annotations. Next, we trained models on the annotations from each of the different rater pools, and compared the scores of these models on comments from several test sets. Finally, we discuss how using raters that self-identify with the subjects of comments can create more inclusive machine learning models, and provide more nuanced ratings than those by random raters.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3555088

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The identity and views of human annotators become embedded in the data, influencing subsequent system behavior.

    “In expressing 'their' perspective through data annotations, it important to recognize how the identity and views of human annotators become embedded in the data, influencing subsequent system behavior [51, 52, 90].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [51]

  • Researchers have previously problematized anonymous crowdsourced data annotation approaches as being unspecific about who's worldviews and beliefs become manifest in the data.

    “Researchers have previously problematized anonymous crowdsourced data annotation approaches [31, 53, 65] as being unspecific about who's worldviews and beliefs become manifest in the data [11, 17, 51, 78, 90, 91].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [51]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020. Modeling Annotator Perspective and Polarized Opinions to Improve Hate Speech Detection. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 8, 1 (Oct. 2020), 151-154. https://ojs.aaai.org/index.php/HCOMP/article/view/7473
  2. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arXiv preprint arXiv:2106.15896 (2021). · arXiv
  3. Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and Measuring Annotator Bias Based on Annotators' Demographic Characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 184-190. https://doi.org/10.18653/v1/2020.alw-1.21 · DOI
  4. Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465 (2020). · arXiv
  5. Valerio Basile. 2021. It's the End of the Gold Standard as We Know It. In AIxIA 2020 -Advances in Artificial Intelligence, Matteo Baldoni and Stefania Bandini (Eds.). Springer International Publishing, Cham, 441-453.
  6. Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021. Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. arXiv preprint arXiv:2109.04270 (2021). · arXiv
  7. Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comp. Lab. L. & Pol'y J. 37 (2015), 543.
  8. Reuben Binns, Michael Veale, Max Van Kleek, and Nigel Shadbolt. 2017. Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation. In Social Informatics, Giovanni Luca Ciampaglia, Afra Mashhadi, and Taha Yasseri (Eds.). Springer International Publishing, Cham, 405-415.
  9. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW '19). Association for Computing Machinery, New York, NY, USA, 491-500. https://doi.org/10.1145/3308560.3317593 · DOI
  10. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial Bias in Hate Speech and Abusive Language Detection Datasets. In Proceedings of the Third Workshop on Abusive Language Online. Association for Computational Linguistics, Florence, Italy, 25-35. https://doi.org/10.18653/v1/W19-3504 · DOI · Paper page
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171-4186. https://doi.org/10.18653/v1/N19-1423 · DOI · Paper page
  12. Armend Duzha, Cristiano Casadei, Michael Tosi, and Fabio Celli. 2021. Hate versus politics: detection of hate against policy makers in Italian tweets. SN Social Sciences 1, 9 (2021), 1-15.
  13. Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 2021. Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Framework. In Equity and Access in Algorithms, Mechanisms, and Optimization (New York, NY, USA) (EAAMO '21). Association for Computing Machinery, New York, NY, USA, Article 7, 11 pages. https://doi.org/10.1145/3465416.3483299 · DOI
  14. Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J. Paul. 2020. Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 1440-1448. https://aclanthology. org/2020.lrec-1.180
  15. Gunay Kazimzade and Milagros Miceli. 2020. Biased Priorities, Biased Outcomes: Three Recommendations for Ethics- Oriented Data Annotation Practices. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA) (AIES '20). Association for Computing Machinery, New York, NY, USA, 71. https://doi.org/10.1145/3375627. 3375809 · DOI · Paper page
  16. Jana Kurrek, Haji Mohammad Saleem, and Derek Ruths. 2020. Towards a Comprehensive Taxonomy and Large-Scale Annotated Corpus for Online Slur Usage. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 138-149. https://doi.org/10.18653/v1/2020.alw-1.17 · DOI
  17. Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. 2021. Reconsidering Annotator Disagreement about Racist Language: Noise or Signal?. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, Online, 81-90. https://doi.org/10. 18653/v1/2021.socialnlp-1.7 · DOI
  18. Haochen Liu, Wei Jin, Hamid Karimi, Zitao Liu, and Jiliang Tang. 2021. The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021. Association for Computational Linguistics, Online, 74-85. https://doi.org/10.18653/v1/2021.findings-acl.7 · DOI
  19. Saif M Mohammad. 2021. Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis. arXiv preprint arXiv:2109.08256 (2021). · arXiv
  20. Edoardo Mosca, Maximilian Wich, and Georg Groh. 2021. Understanding and Interpreting the Impact of User Context in Hate Speech Detection. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, Online, 91-102. https://doi.org/10.18653/v1/2021.socialnlp-1.8 · DOI
  21. Lisa Posch, Arnim Bleier, Fabian Flöck, and Markus Strohmaier. 2018. Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. arXiv preprint arXiv:1812.05948 (2018). · arXiv
  22. Daryl Pregibon. 1981. Logistic regression diagnostics. The annals of statistics 9, 4 (1981), 705-724.
  23. Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional Tests for Hate Speech Detection Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 41-58. https://doi.org/10.18653/v1/2021.acl-long.4 · DOI
  24. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The Risk of Racial Bias in Hate Speech Detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 1668-1678. https://doi.org/10.18653/v1/P19-1163 · DOI · Paper page
  25. Shivashankar Subramanian, Xudong Han, Timothy Baldwin, Trevor Cohn, and Lea Frermann. 2021. Evaluating Debiasing Techniques for Intersectional Biases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2492- 2498. https://doi.org/10.18653/v1/2021.emnlp-main.193 · DOI
  26. Jialu Wang, Yang Liu, and Xin Eric Wang. 2021. Assessing Multilingual Fairness in Pre-trained Multimodal Representa- tions. arXiv preprint arXiv:2106.06683 (2021). · arXiv
  27. Zeerak Waseem. 2016. Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science. Association for Computational Linguistics, Austin, Texas, 138-142. https://doi.org/10.18653/v1/W16-5618 · DOI
  28. Maximilian Wich, Hala Al Kuwatly, and Georg Groh. 2020. Investigating Annotator Bias with a Graph-Based Approach. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 191-199. https://doi.org/10.18653/v1/2020.alw-1.22 · DOI
  29. Maximilian Wich, Jan Bauer, and Georg Groh. 2020. Impact of Politically Biased Data on Hate Speech Classification. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 54-64. https://doi.org/10.18653/v1/2020.alw-1.7 · DOI
  30. Maximilian Wich, Adrian Gorniak, Tobias Eder, Daniel Bartmann, Burak Enes Cakici, and Georg Groh. 2021. Introducing an Abusive Language Classification Framework for Telegram to Investigate the German Hater Community. arXiv preprint arXiv:2109.07346 (2021). · arXiv
  31. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web (Perth, Australia) (WWW '17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1391-1399. https://doi.org/10.1145/3038912. 3052591 · DOI · Paper page
  32. Michael Miller Yoder. 2021. Computational Models of Identity Presentation in Language. Ph.D. Dissertation. Carnegie Mellon University.
  33. Philine Zeinert, Nanna Inie, and Leon Derczynski. 2021. Annotating Online Misogyny. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 3181-3197. https: //doi.org/10.18653/v1/2021.acl-long.247 · DOI

Cited by

Sorted by citations

Authors
Goyal, N., Kivlichan, I. D., Rosen, R., & Vasserman, L.
Affiliations
Google (United States) (US)
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2022-11-07
Volume
6
Issue
CSCW2
Pages
1-28
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3555088
OpenAlex
W4309618884
Cited by
76
References
5
Citations by year
2026: 17 · 2025: 21 · 2024: 20 · 2023: 17 · 2022: 1
FWCI
4.85
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
online comments, annotation bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review