CoChat
Sign Up
PreprintCited by 28Hate Speech and Cyberbullying DetectionEthics and Social Impacts of AIMisinformation and Its Impacts

Designing Toxic Content Classification for a Diversity of Perspectives

Open access
Kumar, Deepak, Kelley, Patrick Gage, Consolvo, Sunny, Mason, Joshua, Bursztein, Elie, Durumeric, Zakir, Thomas, Kurt, Bailey, Michael·2021·arXiv
DOI 10.48550/arxiv.2106.04511arXiv 2106.04511
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work, we demonstrate how existing classifiers for identifying toxic comments online fail to generalize to the diverse concerns of Internet users. We survey 17,280 participants to understand how user expectations for what constitutes toxic content differ across demographics, beliefs, and personal experiences. We find that groups historically at-risk of harassment - such as people who identify as LGBTQ+ or young adults - are more likely to to flag a random comment drawn from Reddit, Twitter, or 4chan as toxic, as are people who have personally experienced harassment in the past. Based on our findings, we show how current one-size-fits-all toxicity classification algorithms, like the Perspective API from Jigsaw, can improve in accuracy by 86% on average through personalized model tuning. Ultimately, we highlight current pitfalls and new design directions that can improve the equity and efficacy of toxic content classifiers for all users.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.04511

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The toxicity threshold is set at 0.61, identified by Kumar et al. (2021) and Aleksandric et al. (2024).

    “We set the threshold at 0.61 which is identified by Kumar et al. (2021) by employing a survey on 17,280 participants and Aleksandric et al. (2024) who found that Perspective API gets the highest accuracy on this threshold on their independent test set.”

    From Cross-Partisan Interactions on Twitter (2025) · cites this paper as Kumar et al. (2021)

    Section 6.1 (p. 11) reports 0.61 as the median of thresholds optimized separately for individual participants, not as a recommended single toxicity threshold. It instead gives 0.18–0.49 as the optimal range for the aggregate dataset. The Conclusion (p. 12) confirms that the study surveyed 17,280 participants. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • The median optimal threshold for classifying toxic content across participants is 0.61.

    “They find that the median optimal threshold across participants 0.61.”

    From Cross-Partisan Interactions on Twitter (2025) · cites this paper as Kumar et al. (2021)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [2][2] J. Blackburn and H. Kwak. Stfu noob!: predicting crowdsourced decisions on toxic behavior in online games. In Proceedings of the 23rd international conference on World wide web . ACM, 2014. -
  2. [3][3] L. Blackwell, J. Dimond, S. Schoenebeck, and C. Lampe. Classification and its consequences for online harassment: Design insights from heartmob. In Proceedings of the ACM on Human-Computer Interaction , 2017. -
  3. [4][4] L. Blackwell, J. Hardy, T. Ammari, T. Veinot, C. Lampe, and S. Schoenebeck. Lgbt parents and social media: Advocacy, privacy, and disclosure during shifting social movements. In ACM CHI conference on human factors in computing systems , 2016. -
  4. [6][6] E. Chandrasekharan, M. Samory, A. Srinivasan, and E. Gilbert. The bag of communities: identifying abusive behavior online with preexisting internet data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , 2017. -
  5. [7][7] D. Chatzakou, N. Kourtellis, J. Blackburn, E. De Cristofaro, G. Stringhini, and A. Vakali. Mean birds: Detecting aggression and bullying on Twitter. In ACM Web Science Conference , 2017. -
  6. [8][8] J. Cheng, C. Danescu-Niculescu-Mizil, and J. Leskovec. Antisocial behavior in online discussion communities. In Ninth International AAAI Conference on Web and Social Media , 2015. -
  7. [9][9] D. K. Citron. Addressing cyber harassment: An overview of hate crimes in cyberspace. Journal of Law, Technology & the Internet , 2014. -
  8. [10][10] G. Cowan and J. Mettrick. The effects of target variables and settting on perceptions of hate speech. Journal of Applied Social Psychology , 2002. -
  9. [12][12] T. Davidson, D. Bhattacharya, and I. Weber. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online , 2019. -
  10. [13][13] T. Davidson, D. Warmsley, M. Macy, and I. Weber. Automated hate speech detection and the problem of offensive language. In AAAI International Conference On Web and Social Media , 2017. -
  11. [14][14] D. DiFranzo, S. H. Taylor, F. Kazerooni, O. D. Wherry, and N. N. Bazarova. Upstanding by design: Bystander intervention in cyberbullying. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , 2018. -
  12. [15][15] K. Dinakar, R. Reichart, and H. Lieberman. Modeling the detection of textual cyberbullying. In AAAI International Conference On Web and Social Media , 2011. -
  13. [16][16] L. Dixon, J. Li, J. Sorensen, N. Thain, and L. Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , 2018. -
  14. [17][17] N. Djuric, J. Zhou, R. Morris, M. Grbovic, V. Radosavljevic, and N. Bhamidipati. Hate speech detection with comment embeddings. In The Web Conference , 2015. -
  15. [18][18] M. ElSherief, S. Nilizadeh, D. Nguyen, G. Vigna, and E. Belding. Peer to peer hate: Hate speech instigators and their targets. In AAAI International Conference On Web and Social Media , 2018. -
  16. [19][19] J. Finkelstein, S. Zannettou, B. Bradlyn, and J. Blackburn. A quantitative approach to understanding online antisemitism. In Proceedings of the AAAI International Conference on Web and Social Media , 2020. -
  17. [20][20] A. M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, and N. Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In 12th International AAAI Conference on Web and Social Media , 2018. -
  18. [23][23] T. Gröndahl, L. Pajola, M. Juuti, M. Conti, and N. Asokan. All you need is 'love' evading hate speech detection. In Proceedings of the ACM Workshop on Artificial Intelligence and Security , 2018. -
  19. [27][27] H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran. Deceiving google's perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138 , 2017. - · arXiv
  20. [28][28] Y. Hua, M. Naaman, and T. Ristenpart. Characterizing twitter users who engage in adversarial interactions against political candidates. In ACM CHI Conference on Human Factors in Computing Systems , 2020. -
  21. [29][29] Y. Hua, T. Ristenpart, and M. Naaman. Towards measuring adversarial twitter interactions against candidates in the us midterm elections. In International Conference on Web and Social Media , 2020. -
  22. [32][32] E. Jain, S. Brown, J. Chen, E. Neaton, M. Baidas, Z. Dong, H. Gu, and N. S. Artan. Adversarial text generation for google's perspective api. In 2018 International Conference on Computational Science and Computational Intelligence (CSCI) , 2018. -
  23. [33][33] S. Jhaver, S. Ghoshal, A. Bruckman, and E. Gilbert. Online harassment and content moderation: The case of blocklists. In Proceedings of the ACM Transactions on Computer-Human Interaction , 2018. -
  24. [35][35] H. Kwak, J. Blackburn, and S. Han. Exploring cyberbullying and other toxic behavior in team competition online games. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems . ACM, 2015. -
  25. [39][39] K. Mahar, D. Karger, and A. X. Zhang. Squadbox: A tool to combat online harassment using friendsourced moderation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , 2018. -
  26. [40][40] B. Maher. Can a video game company tame toxic behaviour? https://www.nature.com/news/can-a-video-game-companytame-toxic-behaviour-1.19647. -
  27. [43][43] C. Nobata, J. Tetreault, A. Thomas, Y. Mehdad, and Y. Chang. Abusive language detection in online user content. In The Web Conference , 2016. -
  28. [44][44] J. A. Pater, M. K. Kim, E. D. Mynatt, and C. Fiesler. Characterizations of online harassment: Comparing policies across social media platforms. In Proceedings of the 19th International Conference on Supporting Group Work , 2016. -
  29. [46][46] D. Razo and S. Kübler. Investigating sampling bias in abusive language detection. In 4th Workshop on Online Abuse and Harms , 2020. -
  30. [47][47] E. M. Redmiles, J. Bodford, and L. Blackwell. 'i just want to feel safe': A diary study of safety perceptions on social media. In International AAAI Conference on Web and Social Media , 2019. -
  31. [48][48] E. M. Redmiles, S. Kross, and M. L. Mazurek. How well do my results generalize? comparing security and privacy survey results from mturk, web, and telephone samples. In Proceedings of the IEEE Symposium on Security and Privacy , 2019. -
  32. [49][49] E. M. Redmiles, Z. Zhu, S. Kross, D. Kuchhal, T. Dumitras, and M. L. Mazurek. Asking for a friend: Evaluating response biases in security user studies. In 25th ACM SIGSAC Conference on Computer and Communications Security , 2018. -
  33. [50][50] N. Sambasivan, A. Batool, N. Ahmed, T. Matthews, K. Thomas, L. S. Gaytán-Lugo, D. Nemer, E. Bursztein, E. Churchill, and S. Consolvo. 'they don't leave us alone anywhere we go': Gender and digital abuse in south asia. In Proceedings of the Conference on Human Factors in Computing Systems , 2019. -
  34. [51][51] A. Saravanaraj, J. Sheeba, and S. P. Devaneyan. Automatic detection of Cyberbullying from Twitter. International Journal of Computer Science and Information Technology & Security (IJCSITS) , 2016. -
  35. [52][52] K. Thomas, D. Akhawe, M. Bailey, D. Boneh, E. Bursztein, S. Consolvo, N. Dell, Z. Durumeric, P. G. Kelley, D. Kumar, D. McCoy, S. Meiklejohn, T. Ristenpart, and G. Stringhini. Sok: Hate, harassment, and the changing landscape of online abuse. In Proceedings of the IEEE Symposium on Security and Privacy , 2021. -
  36. [54][54] C. Van Hee, E. Lefever, B. Verhoeven, J. Mennes, B. Desmet, G. De Pauw, W. Daelemans, and V. Hoste. Automatic detection and prevention of cyberbullying. In International Conference on Human and Social Analytics , 2015. -
  37. [55][55] W. Warner and J. Hirschberg. Detecting hate speech on the World Wide Web. In Proceedings of the second workshop on language in social media , 2012. -
  38. [56][56] M. Wich, H. Al Kuwatly, and G. Groh. Investigating annotator bias with a graph-based approach. In 4th Workshop on Online Abuse and Harms , 2020. -
  39. [57][57] E. Wulczyn, N. Thain, and L. Dixon. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web , 2017. -
  40. [58][58] S. Zannettou, T. Caulfield, J. Blackburn, E. De Cristofaro, M. Sirivianos, G. Stringhini, and G. Suarez-Tangil. On the origins of memes by means of fringe web communities. In Proceedings of the Internet Measurement Conference 2018 . ACM, 2018.

Cited by

Sorted by citations

Authors
Kumar, D., Kelley, P. G., Consolvo, S., Mason, J., Bursztein, E., Durumeric, Z., Thomas, K., & Bailey, M.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
299-318
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.04511
arXiv
2106.04511
OpenAlex
W3168006205
Cited by
28
References
1
Citations by year
2025: 3 · 2024: 2 · 2023: 20 · 2022: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
online harassment, Perspective API, young adults, Reddit, Twitter, 4chan, algorithmic fairness, classifier generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review