Racial Bias in Hate Speech and Abusive Language Detection Datasets
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Technologies for abusive language detection are being developed and applied with little consideration of their potential biases. We examine racial bias in five different sets of Twitter data annotated for hate speech and abusive language. We train classifiers on these datasets and compare the predictions of these classifiers on tweets written in African-American English with those written in Standard American English. The results show evidence of systematic racial bias in all datasets, as classifiers trained on them tend to predict that tweets written in African-American English are abusive at substantially higher rates. If these abusive language detection systems are used in the field they will therefore have a disproportionate negative impact on African-American social media users. Consequently, these systems may discriminate against the groups who are often the targets of the abuse we are trying to detect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“and processes of creating resources and technologies may further entrench such disparities [25, 38, 144].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efron, B., & Tibshirani, R. (1986) · Statistical Science · doi:10.1214/ss/1177013815 · Cited by 6,033
(2018) · Cited by 3,510
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 2×
“For example, in unsupervised learning, word embed-dings often contain biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2018) which persist even after attempts to remove them (Go-nen and Goldberg, 2019).”
BackgroundAutomated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Self-citationHateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655 · cited 12×
“We see similar results for Waseem and Hovy (2016) and Waseem (2016).”
BackgroundMethodResultInfluentialMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 1×
“For example, in unsupervised learning, word embed-dings often contain biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2018) which persist even after attempts to remove them (Go-nen and Goldberg, 2019).”
BackgroundA Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media · doi:10.18653/v1/w17-1101 · Cited by 1,395 · cited 1×
“While hate speech and abusive language detection has become an important area for natural language processing research (Schmidt and Wiegand, 2017; Waseem et al., 2017; Fortuna and Nunes, 2018), there has been little work addressing the potential for these systems to be biased.”
BackgroundA Survey on Automatic Detection of Hate Speech in Text
Fortuna, P., & Nunes, S. (2019) · ACM Computing Surveys · doi:10.1145/3232676 · Cited by 1,241 · cited 1×
“While hate speech and abusive language detection has become an important area for natural language processing research (Schmidt and Wiegand, 2017; Waseem et al., 2017; Fortuna and Nunes, 2018), there has been little work addressing the potential for these systems to be biased.”
BackgroundGender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 1×
“…2017), numerous gender and racial biases exist in sentiment classification systems (Kir-itchenko and Mohammad , 2018), and errors in both co-reference resolution systems and occupational classification models reflect gendered occupational patterns (Zhao et al., 2018; De-Arteaga et al., 2019).”
BackgroundMeasuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · cited 2×
“Dixon et al. (2018) directly measured biases in the Google Perspective API classifier, 1 trained on data from Wikipedia talk comments, finding that it tended to give high toxicity scores to innocuous statements like “I am a gay man”.”
BackgroundBurnap, P., & Williams, M. L. (2015) · Policy & Internet · doi:10.1002/poi3.85 · Cited by 616 · cited 1×
“A number of studies have shown that false positive cases of hate speech are associated with the presence of terms related to race, gender, and sexuality (Kwok and Wang, 2013; Burnap and Williams , 2015; Davidson et al., 2017).”
BackgroundLarge Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
Founta, A., Djouvas, C., Chatzakou, D., Leontiadis, I., Blackburn, J., Stringhini, G., Vakali, A., Sirivianos, M., & Kourtellis, N. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14991 · Cited by 596
Beyond prediction: using big data for policy problems
Athey, S. (2017) · Science · doi:10.1126/science.aal4321 · Cited by 575 · cited 1×
“Scholars and practitioners have recently been de-voting more attention to bias in machine learning models, particularly as these models are becoming involved in more and more consequential decisions ( Athey, 2017).”
BackgroundAre You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter
Waseem, Z. (2016) · Proceedings of the First Workshop on NLP and Computational Social Science · doi:10.18653/v1/w16-5618 · Cited by 569 · cited 11×
“We see similar results for Waseem and Hovy (2016) and Waseem (2016).”
BackgroundMethodResultInfluentialUnderstanding Abuse: A Typology of Abusive Language Detection Subtasks
Waseem, Z., Davidson, T., Warmsley, D., & Weber, I. (2017) · Proceedings of the First Workshop on Abusive Language Online · doi:10.18653/v1/w17-3012 · Cited by 474 · cited 2×
“The aim of these models is to identify abusive language that directly targets certain individuals or groups, particularly people belonging to protected categories (Waseem et al., 2017).”
BackgroundSelf-citationGonen, H., & Goldberg, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1061 · Cited by 454
Locate the Hate: Detecting Tweets against Blacks
Kwok, I., & Wang, Y. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8539 · Cited by 446 · cited 4×
“However the variant with the “-a” suffix is typically used innocuously by African-Americans (Kwok and Wang, 2013), indeed our results indicate that it is used far more frequently in black-aligned tweets (although it is still used by many white people).”
BackgroundInfluentialExamining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384
Gender and Dialect Bias in YouTube’s Automatic Captions
Tatman, R. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1606 · Cited by 338 · cited 2×
“There are many examples of bias in supervised learning contexts: YouTube’s captioning models make more errors when transcribing women (Tatman, 2017), AAE is more likely to be misclassified as non-English by widely used language classifiers (Blodgett and O’Connor, 2017), numerous gender and racial…”
BackgroundBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337 · cited 4×
“This includes both supervised (Blodgett and O’Connor, 2017; Tatman, 2017; Kiritchenko and Mohammad, 2018; De-Arteaga et al., 2019) and unsupervised natural language processing systems (Bolukbasi et al.”
BackgroundInfluentialReducing Gender Bias in Abusive Language Detection
Park, J. H., Shin, J., & Fung, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1302 · Cited by 323 · cited 3×
“Park et al. (2018) build upon this study, using templates to study gender differences in performance across two hate speech and abusive language detection datasets.”
BackgroundMethodDemographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · cited 8×
“We find evidence of systematic racial biases across all of the classifiers, with AAE tweets predicted as belonging to negative classes like hate speech or harassment significantly more frequently than SAE tweets.”
BackgroundMethodInfluentialA survey on Shari'ah governance practices in Malaysia, GCC countries and the UK: Critical appraisal
Hasan, Z. (2011) · International Journal of Islamic and Middle Eastern Finance and Management · doi:10.1108/17538391111122195 · Cited by 240
Computer‐Assisted Keyword and Document Set Discovery from Unstructured Text
King, G., Lam, P., & Roberts, M. E. (2017) · American Journal of Political Science · doi:10.1111/ajps.12291 · Cited by 229 · cited 1×
“Some studies sampled tweets using small, ad hoc sets of key-words created by the authors (Waseem and Hovy, 2016; Waseem, 2016; Golbeck et al., 2017), an approach demonstrated to produce poor results (King et al., 2017).”
MethodA Large Labeled Corpus for Online Harassment Research
Golbeck, J., Ashktorab, Z., Banjo, R. O., Berlinger, A., Bhagwan, S., Buntain, C., Cheakalos, P., Geller, A. A., Gergory, Q., Gnanasekaran, R. K., et al. (2017) · Proceedings of the 2017 ACM on Web Science Conference · doi:10.1145/3091478.3091509 · Cited by 201 · cited 6×
“The datasets considered here relied upon a range of different annotators, from the authors (Golbeck et al., 2017; Waseem and Hovy, 2016) and crowdwork-ers (Davidson et al., 2017; Founta et al., 2018) to activists (Waseem, 2016).”
BackgroundMethodResultInfluentialGonen, H., & Goldberg, Y. (2019) · arXiv (Cornell University) · Cited by 156
Blodgett, S. L., & O'Connor, B. (2017) · arXiv · doi:10.48550/arxiv.1707.00061 · Cited by 153 · cited 3×
“Following Blod-gett and O’Connor (2017), we call these datasets black-aligned and white-aligned tweets, reflecting 5 We use this threshold following Blodgett and O’Connor (2017) and after consulting with the lead author.”
BackgroundMethodBridging the Gaps: Multi Task Learning for Domain Transfer of Hate Speech Detection
Talat, Z., Thorne, J., & Bingel, J. (2018) · Human-computer interaction series · doi:10.1007/978-3-319-78583-7_3 · Cited by 139 · cited 4×
“First, the dataset contains a large number of cases where AAE is used ( Waseem et al., 2018).”
BackgroundMethodInfluentialAnalyzing the Targets of Hate in Online Social Media
da Silva, L. H. G., Mondal, M., Correa, D., Benevenuto, F., & Weber, I. G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07709 · Cited by 100 · cited 2×
“When labeling datasets and evaluating our models we must pay more attention to the baseline rates of usage of different types of abusive language and how they may vary across populations (Silva et al., 2016).”
BackgroundMethodSelf-citationNIGGA: The 21st-Century Theoretical Superhero
Neal, M. A. (2013) · Cultural Anthropology · doi:10.1111/cuan.12025 · Cited by 38 · cited 1×
“While the history of the word and its usages is too complex to be summarized here (Neal, 2013), when used with the “-er“ suffix it is generally considered to be a racist ephiphet, associated with white supremacy.”
BackgroundAlorainy, W. S., Burnap, P., Liu, H., Javed, A., & Williams, M. L. (2018) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc.2018.8527001 · Cited by 27 · cited 1×
“Despite this distinction, some studies have considered this variant to be hateful (Silva et al., 2016; Alorainy et al., 2018).”
Background
Research works printed but not linked: 2
- Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015. Hate Speech Detection with Comment Embeddings. In WWW2015Companion , pages 2930. ACM Press.
- Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences , 115(16):E3635-E3644.
Cited by
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
“…associated with AAE than on text without these features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection systems score tweets containing features associated with AAE as more offensive than tweets without them (Davidson et al., 2019; Sap et al., 2019).”
BackgroundInfluentialRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Eloundou Nekoul, F., Lee, T., Adler, S., Jiang, A., & Weng, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26752 · Cited by 129
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.522 · Cited by 148
“…of large-scale evaluation required for systematic comparison of meth-ods, an abundance of work shows that their accuracy is far from ideal (Dixon et al., 2018; Sap et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020) in part due to reliance on spurious features, which we discuss in §8.”
BackgroundInfluentialHateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., & Mukherjee, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17745 · Cited by 513
Metadata
- Authors
- Davidson, T., Bhattacharya, D., & Weber, I.
- Affiliations
- Cornell University (US)
- Venue
- Proceedings of the Third Workshop on Abusive Language Online
- Published
- 2019
- Pages
- 25-35
- OpenAlex
- W2947838315
- Cited by
- 414
- References
- 27
- Citations by year
- 2026: 11 · 2025: 46 · 2024: 38 · 2023: 79 · 2022: 73 · 2021: 107 · 2020: 55 · 2019: 5
- FWCI
- 23.06
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Abusive Language Online (ALW) (publishedVersion, cc-by)
- arXiv (Cornell University) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- racial bias, hate speech detection, offensive language detection, annotated dataset, African American English, Twitter, algorithmic discrimination, social media abuse
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.