The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
The Risk of Racial Bias in Hate Speech Detection
Open accessAbstract
We investigate how annotators' insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations.We first uncover unexpected correlations between surface markers of African American English (AAE) and ratings of toxicity in several widely-used hate speech datasets.Then, we show that models trained on these corpora acquire and propagate these biases, such that AAE tweets and tweets by self-identified African Americans are up to two times more likely to be labelled as offensive compared to others.Finally, we propose dialect and race priming as ways to reduce the racial bias in annotation, showing that when annotators are made explicitly aware of an AAE tweet's dialect they are significantly less likely to label the tweet as offensive.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Toxicity detection models misclassify African-American English tweets as negative more often than those written in Standard American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Sap et al. 2019
The cited source says what the claim attributes to it.
In the citing paper
“For example, toxicity classifiers may disproportionately flag African-American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019)”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Sap et al. 2019
In the citing paper
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. J., & Hovy, E. H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1174 · Cited by 5,022
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Attention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450
The Impact of Social Media on Children, Adolescents, and Families
O'Keeffe, G. S., Clarke-Pearson, K., & Council on Communications and Media (2011) · Pediatrics · doi:10.1542/peds.2011-0054 · Cited by 1,987 · cited 1×
“Toxic language (e.g., hate speech, abusive speech, or other offensive speech) primarily targets members of minority groups and can catalyze real-life violence towards them (O’Keeffe et al., 2011; Cleland, 2014; Mozur, 2018).”
BackgroundHateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Talat, Z., & Hovy, D. (2016) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n16-2013 · Cited by 1,919
Internal and external motivation to respond without prejudice.
Plant, E. A., & Devine, P. G. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.75.3.811 · Cited by 1,689
A Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · SocialNLP@EACL · doi:10.18653/v1/w17-1101 · Cited by 1,478
Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network
Zhang, Z., Robinson, D., & Tepper, J. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93417-4_48 · Cited by 676
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674
Ex Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636
Burnap, P., & Williams, M. L. (2015) · Policy & Internet · doi:10.1002/poi3.85 · Cited by 615
Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
Founta, A.-M., Djouvas, C., Chatzakou, D., Ilias Leontiadis, A., Blackburn, J., Stringhini, G., Vakali, A. I., Sirivianos, M., & Kourtellis, N. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14991 · Cited by 605
Locate the Hate: Detecting Tweets against Blacks
Kwok, I., & Wang, Y. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8539 · Cited by 452
Race as a Bundle of Sticks: Designs that Estimate Effects of Seemingly Immutable Characteristics
Sen, M., & Wasow, O. (2016) · Annual Review of Political Science · doi:10.1146/annurev-polisci-032015-010015 · Cited by 422
One-step and Two-step Classification for Abusive Language Detection on Twitter
Park, J. H., & Fung, P. (2017) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w17-3006 · Cited by 409
Reducing Gender Bias in Abusive Language Detection
Park, J. H., Shin, J., & Fung, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1302 · Cited by 323 · cited 3×
“Since previous research has exposed the potential for other identity-based biases in offensive language detection (e.g., gender bias; Park et al., 2018), here we investigate racial bias against speech by African Americans, focusing on Twitter as it is a particularly important space for Black activism (Williams and Dom…”
BackgroundMethodDemographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · cited 6×
“In the dialect priming condition, we explicitly include the tweet’s dialect as measured by Blodgett et al. (2016), as well as extra instructions priming workers to think of tweet dialect as a proxy for the author’s race.”
MethodInfluentialTweets, Tweeps, and Signifyin’: Communication and Cultural Performance on “Black Twitter”
Florini, S. (2014) · Television & New Media · doi:10.1177/1527476413480247 · Cited by 271
Characterizing and Detecting Hateful Users on Twitter
Ribeiro, M. H., Calais, P., Santos, Y., Almeida, V. A. F., & Meira, W. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15057 · Cited by 260
Ex Machina: Personal Attacks Seen at ScalePreprint
Wulczyn, E., Thain, N., & Dixon, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08914 · Cited by 260
Challenges for Toxic Comment Classification: An In-Depth Error Analysis
van Aken, B., Risch, J., Krestel, R., & Löser, A. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5105 · Cited by 253
A Large Labeled Corpus for Online Harassment Research
Golbeck, J., Ashktorab, Z., Banjo, R. O., Berlinger, A., Bhagwan, S., Buntain, C., Cheakalos, P., Geller, A. A., Gergory, Q., Gnanasekaran, R. K., et al. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3091509 · Cited by 203
Egalitarian goals trigger stereotype inhibition: A proactive form of stereotype control
Moskowitz, G. B., & Li, P. (2010) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2010.08.014 · Cited by 153
Activism in the Social Media Age
Anderson, M., Toor, S., Olmstead, K., Rainie, L., & Smith, A. W. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10134877 · Cited by 149
Racism, Football Fans, and Online Message Boards
Cleland, J. A. (2013) · Journal of Sport and Social Issues · doi:10.1177/0193723513499922 · Cited by 143
The N Word: Its History and Use in the African American Community
Rahman, J. (2012) · Journal of English Linguistics · doi:10.1177/0075424211414807 · Cited by 142
Bridging the Gaps: Multi Task Learning for Domain Transfer of Hate Speech Detection
Talat, Z., Thorne, J. P., & Bingel, J. (2018) · Human-computer interaction series · doi:10.1007/978-3-319-78583-7_3 · Cited by 108
Comparative Studies of Detecting Abusive Language on Twitter
Lee, Y., Yoon, S., & Jung, K. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5113 · Cited by 100
African-American language use: Ideology and so-called obscenity
Bailey, G., Baugh, J., Mufwene, S. S., & Rickford, J. R. (2013) · Routledge eBooks · doi:10.4324/9780203355596-18 · Cited by 67
User-Level Race and Ethnicity Predictors from Twitter Text
Preoțiuc-Pietro, D., & Ungar, L. (2018) · International Conference on Computational Linguistics · Cited by 50
African-American language use: Ideology and so-called obscenity
Spears, A. K. (2021) · Routledge eBooks · doi:10.4324/9781003165330-9 · Cited by 42
African American Vernacular English: phonology
Edwards, W. F. (2008) · doi:10.1515/9783110208405.1.181 · Cited by 24
Examining a hate speech corpus for hate speech detection and popularity predictionPreprint
Klubička, F., & Fernández, R. (2021) · arXiv (Cornell University) · doi:10.21427/17ra-2p75 · Cited by 23
Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
Kapoor, R., Kumar, Y., Rajput, K., Shah, R. R., Kumaraguru, P., & Zimmermann, R. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33019951 · Cited by 23
Cyber Hate Classification: 'Othering' Language And Paragraph EmbeddingPreprint
Alorainy, W. S., Burnap, P., Liu, H., & Williams, M. L. (2018) · arXiv (Cornell University) · Cited by 21
African American English: Linguistic Introduction
Alim, H. S. (2006) · Journal of Linguistic Anthropology · doi:10.1525/jlin.2006.16.1.135 · Cited by 10
Examining a hate speech corpus for hate speech detection and popularity predictionPreprint
Klubička, F., & Fernández, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04661 · Cited by 8
The Enemy Among Us: Detecting Hate Speech with Threats Based 'Othering' Language EmbeddingsPreprint
Alorainy, W. S., Burnap, P., Liu, H., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.07495 · Cited by 6
African American Vernacular English: phonology
(2004) · doi:10.1515/9783110175325.1.383 · Cited by 3
Challenges for Toxic Comment Classification: An In-Depth Error Analysis
van Aken, B., Risch, J., Krestel, R., & Löser, A. (2018) · arXiv · doi:10.48550/arxiv.1809.07572
Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
Kapoor, R., Kumar, Y., Rajput, K., Shah, R. R., Kumaraguru, P., & Zimmermann, R. (2018) · arXiv · doi:10.48550/arxiv.1809.08652
Comparative Studies of Detecting Abusive Language on Twitter
Lee, Y., Yoon, S., & Jung, K. (2018) · arXiv · doi:10.48550/arxiv.1808.10245
Research works printed but not linked: 2
- E. Ashby Plant and Patricia G. Devine. 1998. Inter- nal and external motivation to respond without prej- udice. J. Pers. Soc. Psychol., 75(3):811-832.
- Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hi- erarchical attention networks for document classifi- cation. In NAACL.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“However, this may have disproportionately excluded casual language, code-switching, or dialectal diversity (e.g., Sap et al. (2019)) and may limit PaLM’s capability to model the non-dominant dialects across the English-speaking regions globally.”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
“…associated with AAE than on text without these features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection systems score tweets containing features associated with AAE as more offensive than tweets without them (Davidson et al., 2019; Sap et al., 2019).”
BackgroundInfluentialDecolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765
Metadata
- Authors
- Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A.
- Affiliations
- University of Washington (US)
- Carnegie Mellon University (US)
- Allen Institute for Artificial Intelligence (US)
- Corresponding author
- Sap, Maarten
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 1668-1678
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2949678053
- Cited by
- 817
- References
- 40
- Citations by year
- 2026: 59 · 2025: 96 · 2024: 86 · 2023: 139 · 2022: 142 · 2021: 178 · 2020: 105 · 2019: 12
- FWCI
- 40.99
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1714566)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- hate speech detection, racial bias, African American English, dialects, annotation bias, hate speech datasets, priming, minority groups
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
