CoChat
Sign Up
Cited by 260Hate Speech and Cyberbullying DetectionEthics and Social Impacts of AISentiment Analysis and Opinion Mining

Hate speech detection and racial bias mitigation in social media based on BERT model

Open access
Mozafari, Marzieh, Farahbakhsh, Reza, Crespi, Noël·2020·PLOS ONE
DOI 10.1371/journal.pone.0237861PMC PMC7451563
Open this paper in CoChat — free accountOpen PDF

Abstract

Disparate biases associated with datasets and trained classifiers in hateful and abusive content identification tasks have raised many concerns recently. Although the problem of biased datasets on abusive language detection has been addressed more frequently, biases arising from trained classifiers have not yet been a matter of concern. In this paper, we first introduce a transfer learning approach for hate speech detection based on an existing pre-trained language model called BERT (Bidirectional Encoder Representations from Transformers) and evaluate the proposed model on two publicly available datasets that have been annotated for racism, sexism, hate or offensive content on Twitter. Next, we introduce a bias alleviation mechanism to mitigate the effect of bias in training set during the fine-tuning of our pre-trained BERT-based model for hate speech detection. Toward that end, we use an existing regularization method to reweight input samples, thereby decreasing the effects of high correlated training set' s n-grams with class labels, and then fine-tune our pre-trained BERT-based model with the new re-weighted samples. To evaluate our bias alleviation mechanism, we employed a cross-domain approach in which we use the trained classifiers on the aforementioned datasets to predict the labels of two new datasets from Twitter, AAE-aligned and White-aligned groups, which indicate tweets written in African-American English (AAE) and Standard American English (SAE), respectively. The results show the existence of systematic racial bias in trained classifiers, as they tend to assign tweets written in AAE from AAE-aligned group to negative classes such as racism, sexism, hate, and offensive more often than tweets written in SAE from White-aligned group. However, the racial bias in our classifiers reduces significantly after our bias alleviation mechanism is incorporated. This work could institute the first step towards debiasing hate speech and abusive language detection systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0237861

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Toxicity detection models misclassify African-American English tweets as negative more often than those written in Standard American English.

    “Toxicity detection models misclassify African-American English tweets as negative more often than those written in Standard American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019).”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Mozafari, Farahbakhsh, and Crespi 2020

  • Toxicity classifiers may disproportionately flag African-American English.

    “For example, toxicity classifiers may disproportionately flag African-American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019)”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Mozafari, Farahbakhsh, and Crespi 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 30

  1. [1]Pete B, L WM. Cyber Hate Speech on Twitter: An Application of Machine Classification and Statistical Modeling for Policy and Decision Making. Policy and Internet. 2015; 7(2): 223– 242. 10.1002/poi3.85 · DOI
  2. [2]Waseem Z, Hovy D. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In: Proceedings of the NAACL Student Research Workshop. San Diego, California: Association for Computational Linguistics; 2016. p. 88–93.
  3. [3]Nobata C, Tetreault JR, Thomas AO, Mehdad Y, Chang Y. Abusive Language Detection in Online User Content. In: Proceedings of the 25th International Conference on World Wide Web. WWW’16. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee; 2016. p. 145–153.
  4. [4]Davidson T, Warmsley D, Macy MW, Weber I. Automated Hate Speech Detection and the Problem of Offensive Language. CoRR. 2017;abs/1703.04009. Available from: http://arxiv.org/abs/1703.04009. · arXiv
  5. [5]Mozafari M, Farahbakhsh R, Crespi N. Content Similarity Analysis of Written Comments under Posts in Social Media. In: SNAMS 2019: 6th International Conference on Social Networks Analysis, Management and Security. Grenade, Spain; Oct 2019. p. 158–165. Available from: 10.1109/SNAMS.2019.8931726. · DOI
  6. [6]Badjatiya P, Gupta S, Gupta M, Varma V. Deep Learning for Hate Speech Detection in Tweets. CoRR. 2017;abs/1706.00188. Available from: http://arxiv.org/abs/1706.00188. · arXiv
  7. [7]Vidgen B, Yasseri T. Detecting weak and strong Islamophobic hate speech on social media. Journal of Information Technology & Politics. 2020; 17(1): 66– 78. Available from: 10.1080/19331681.2019.1702607 · DOI
  8. [11]ALW3: 3rd Workshop on Abusive Language Online;. Available from: https://sites.google.com/view/alw3/home.
  9. [16]Gambäck B, Sikdar UK. Using Convolutional Neural Networks to Classify Hate-Speech. In: Proceedings of the First Workshop on Abusive Language Online. Vancouver, BC, Canada: Association for Computational Linguistics; 2017. p. 85–90.
  10. [17]Waseem Z, Thorne J, Bingel J. In: Golbeck J, editor. Bridging the Gaps: Multi Task Learning for Domain Transfer of Hate Speech Detection. Cham: Springer International Publishing; 2018 p. 29– 55.
  11. [18]Fortuna P, Nunes S. A Survey on Automatic Detection of Hate Speech in Text. ACM Comput Surv. 2018 7; 51(4): 85:1– 85:30.
  12. [19]Rizoiu M, Wang T, Ferraro G, Suominen H. Transfer Learning for Hate Speech Detection in Social Media. CoRR. 2019;abs/1906.03829. Available from: http://arxiv.org/abs/1906.03829. · arXiv
  13. [21]Davidson T, Bhattacharya D, Weber I. Racial Bias in Hate Speech and Abusive Language Detection Datasets. CoRR. 2019;abs/1905.12516. Available from: http://arxiv.org/abs/1905.12516. · arXiv
  14. [22]Wiegand M, Ruppenhofer J, Kleinbauer T. Detection of Abusive Language: the Problem of Biased Datasets. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics; 2019. p. 602–608.
  15. [24]Olteanu A, Castillo C, Boy J, Varshney KR. The Effect of Extremist Violence on Hateful Speech Online. CoRR. 2018;abs/1804.05704. Available from: http://arxiv.org/abs/1804.05704. · arXiv
  16. [25]Ottoni R, Cunha E, Magno G, Bernardina P, Jr WM, Almeida VAF. Analyzing Right-wing YouTube Channels: Hate, Violence and Discrimination. In: Proceedings of the 10th ACM Conference on Web Science. WebSci’18. New York, NY, USA: ACM; 2018. p. 323–332.
  17. [26]Mittos A, Zannettou S, Blackburn J, Cristofaro ED. “And We Will Fight For Our Race!” A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. CoRR. 2019;abs/1901.09735. Available from: http://arxiv.org/abs/1901.09735. · arXiv
  18. [27]Mehdad Y, Tetreault J. Do Characters Abuse More Than Words? In: Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Los Angeles: Association for Computational Linguistics; 2016. p. 299–303.
  19. [28]Malmasi S, Zampieri M. Challenges in Discriminating Profanity from Hate Speech. CoRR. 2018;abs/1803.05495. Available from: http://arxiv.org/abs/1803.05495. · arXiv
  20. [29]Zhang Z, Robinson D, Tepper J. Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network In: The Semantic Web. Cham: Springer International Publishing; 2018 p. 745– 760.
  21. [31]Pennington J, Socher R, Manning C. Glove: Global Vectors for Word Representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics; 2014. p. 1532–1543.
  22. [33]Djuric N, Zhou J, Morris R, Grbovic M, Radosavljevic V, Bhamidipati N. Hate Speech Detection with Comment Embeddings. In: Proceedings of the 24th International Conference on World Wide Web. WWW’15 Companion. New York, NY, USA: ACM; 2015. p. 29–30.
  23. [34]Founta A, Chatzakou D, Kourtellis N, Blackburn J, Vakali A, Leontiadis I. A Unified Deep Learning Architecture for Abuse Detection. In: Proceedings of the 10th ACM Conference on Web Science. WebSci’19. New York, NY, USA: ACM; 2019. p. 105–114.
  24. [35]Howard J, Ruder S. Fine-tuned Language Models for Text Classification. CoRR. 2018;abs/1801.06146. Available from: http://arxiv.org/abs/1801.06146. · arXiv
  25. [38]Badjatiya P, Gupta M, Varma V. Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizations. In: WWW; 2019. p. 49–59. Available from: 10.1145/3308558.3313504. · DOI
  26. [39]Waseem Z. Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. In: Proceedings of the First Workshop on NLP and Computational Social Science. Austin, Texas: Association for Computational Linguistics; 2016. p. 138–142.
  27. [43]Park JH, Fung P. One-step and Two-step Classification for Abusive Language Detection on Twitter. CoRR. 2017;abs/1706.01206. Available from: http://arxiv.org/abs/1706.01206. · arXiv
  28. [44]Kim Y. Convolutional Neural Networks for Sentence Classification. CoRR. 2014;abs/1408.5882. Available from: http://arxiv.org/abs/1408.5882. · arXiv
  29. [46]Schuster T, Shah DJ, Yeo YJS, Filizzola D, Santus E, Barzilay R. Towards Debiasing Fact Verification Models. EMNLP/IJCNLP; 2019. p. 3417–3423.
  30. [48]Bishop CM. Pattern recognition and machine learning. Springer; 2006.

Cited by

Sorted by citations

Authors
Mozafari, M., Farahbakhsh, R., & Crespi, N.
Affiliations
Centre National de la Recherche Scientifique (FR)
Corresponding author
Mozafari, Marzieh
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2020-08-27
Volume
15
Issue
8
Pages
e0237861
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0237861
PMC
PMC7451563
OpenAlex
W3049565363
Cited by
260
References
10
Citations by year
2026: 43 · 2025: 47 · 2024: 56 · 2023: 49 · 2022: 32 · 2021: 32 · 2020: 1
FWCI
11.02
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
hate speech detection, BERT, transfer learning, bias mitigation, sample reweighting, regularization, African American English, Standard American English, Twitter, offensive language detection, racism, sexism, cross-domain evaluation
MeSH terms
Deep Learning, Algorithms, Female, Hate, Humans, Male, Models, Theoretical, Speech, Databases as Topic, Social Media, Racism, Sexism

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review