The Risk of Racial Bias in Hate Speech Detection
Open accessAbstract
We investigate how annotators' insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations.We first uncover unexpected correlations between surface markers of African American English (AAE) and ratings of toxicity in several widely-used hate speech datasets.Then, we show that models trained on these corpora acquire and propagate these biases, such that AAE tweets and tweets by self-identified African Americans are up to two times more likely to be labelled as offensive compared to others.Finally, we propose dialect and race priming as ways to reduce the racial bias in annotation, showing that when annotators are made explicitly aware of an AAE tweet's dialect they are significantly less likely to label the tweet as offensive.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Toxicity detection models misclassify African-American English tweets as negative more often than those written in Standard American English.
“Toxicity detection models misclassify African-American English tweets as negative more often than those written in Standard American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Sap et al. 2019
Toxicity classifiers may disproportionately flag African-American English.
“For example, toxicity classifiers may disproportionately flag African-American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019)”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Sap et al. 2019
Models trained with historical data can present issues such as gender bias and racial bias, potentially exacerbating structural inequalities.
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. J., & Hovy, E. H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1174 · Cited by 4,874 · see citing works
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Attention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450 · see citing works
The Impact of Social Media on Children, Adolescents, and Families
O'Keeffe, G. S., Clarke-Pearson, K., & Council on Communications and Media (2011) · Pediatrics · doi:10.1542/peds.2011-0054 · Cited by 1,987 · cited 1× · see citing works
“Toxic language (e.g., hate speech, abusive speech, or other offensive speech) primarily targets members of minority groups and can catalyze real-life violence towards them (O’Keeffe et al., 2011; Cleland, 2014; Mozur, 2018).”
BackgroundHateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Talat, Z., & Hovy, D. (2016) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n16-2013 · Cited by 1,919 · see citing works
Internal and external motivation to respond without prejudice.
Plant, E. A., & Devine, P. G. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.75.3.811 · Cited by 1,613 · see citing works
A Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A. G., & Wiegand, M. (2017) · International Workshop on Natural Language Processing for Social Media (SocialNLP) · doi:10.18653/v1/w17-1101 · Cited by 1,447 · see citing works
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · see citing works
Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network
Zhang, Z., Robinson, D., & Tepper, J. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93417-4_48 · Cited by 653 · see citing works
Ex Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636 · see citing works
Burnap, P., & Williams, M. L. (2015) · Policy & Internet · doi:10.1002/poi3.85 · Cited by 615 · see citing works
Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
Founta, A.-M., Djouvas, C., Chatzakou, D., Ilias Leontiadis, A., Blackburn, J., Stringhini, G., Vakali, A. I., Sirivianos, M., & Kourtellis, N. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14991 · Cited by 605 · see citing works
Locate the Hate: Detecting Tweets against Blacks
Kwok, I., & Wang, Y. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8539 · Cited by 452 · see citing works
Race as a Bundle of Sticks: Designs that Estimate Effects of Seemingly Immutable Characteristics
Sen, M., & Wasow, O. (2016) · Annual Review of Political Science · doi:10.1146/annurev-polisci-032015-010015 · Cited by 422 · see citing works
One-step and Two-step Classification for Abusive Language Detection on Twitter
Park, J. H., & Fung, P. (2017) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w17-3006 · Cited by 383 · see citing works
Reducing Gender Bias in Abusive Language Detection
Park, J. H., Shin, J., & Fung, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1302 · Cited by 323 · see citing works
Demographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · cited 6× · see citing works
“In the dialect priming condition, we explicitly include the tweet’s dialect as measured by Blodgett et al. (2016), as well as extra instructions priming workers to think of tweet dialect as a proxy for the author’s race.”
MethodInfluentialTweets, Tweeps, and Signifyin’: Communication and Cultural Performance on “Black Twitter”
Florini, S. (2014) · Television & New Media · doi:10.1177/1527476413480247 · Cited by 271 · see citing works
Characterizing and Detecting Hateful Users on Twitter
Ribeiro, M. H., Calais, P., Santos, Y., Almeida, V. A. F., & Meira, W. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15057 · Cited by 260 · see citing works
Ex Machina: Personal Attacks Seen at ScalePreprint
Wulczyn, E., Thain, N., & Dixon, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08914 · Cited by 260 · see citing works
Challenges for Toxic Comment Classification: An In-Depth Error Analysis
van Aken, B., Risch, J., Krestel, R., & Löser, A. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5105 · Cited by 253 · see citing works
A Large Labeled Corpus for Online Harassment Research
Golbeck, J., Ashktorab, Z., Banjo, R. O., Berlinger, A., Bhagwan, S., Buntain, C., Cheakalos, P., Geller, A. A., Gergory, Q., Gnanasekaran, R. K., et al. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3091509 · Cited by 203 · see citing works
Egalitarian goals trigger stereotype inhibition: A proactive form of stereotype control
Moskowitz, G. B., & Li, P. (2010) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2010.08.014 · Cited by 153 · see citing works
Activism in the Social Media Age
Anderson, M., Toor, S., Olmstead, K., Rainie, L., & Smith, A. W. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10134877 · Cited by 149 · see citing works
Racism, Football Fans, and Online Message Boards
Cleland, J. A. (2013) · Journal of Sport and Social Issues · doi:10.1177/0193723513499922 · Cited by 143 · see citing works
The N Word: Its History and Use in the African American Community
Rahman, J. (2012) · Journal of English Linguistics · doi:10.1177/0075424211414807 · Cited by 142 · see citing works
Bridging the Gaps: Multi Task Learning for Domain Transfer of Hate Speech Detection
Talat, Z., Thorne, J. P., & Bingel, J. (2018) · Human-computer interaction series · doi:10.1007/978-3-319-78583-7_3 · Cited by 104 · see citing works
Comparative Studies of Detecting Abusive Language on Twitter
Lee, Y., Yoon, S., & Jung, K. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5113 · Cited by 96 · see citing works
African-American language use: Ideology and so-called obscenity
Bailey, G., Baugh, J., Mufwene, S. S., & Rickford, J. R. (2013) · Routledge eBooks · doi:10.4324/9780203355596-18 · Cited by 67 · see citing works
User-Level Race and Ethnicity Predictors from Twitter Text
Preoțiuc-Pietro, D., & Ungar, L. (2018) · International Conference on Computational Linguistics · Cited by 46 · see citing works
African-American language use: Ideology and so-called obscenity
Spears, A. K. (2021) · Routledge eBooks · doi:10.4324/9781003165330-9 · Cited by 42 · see citing works
African American Vernacular English: phonology
Edwards, W. F. (2008) · doi:10.1515/9783110208405.1.181 · Cited by 24 · see citing works
Examining a hate speech corpus for hate speech detection and popularity predictionPreprint
Klubička, F., & Fernández, R. (2021) · arXiv (Cornell University) · doi:10.21427/17ra-2p75 · Cited by 23 · see citing works
Mind Your Language: Abuse and Offense Detection for Code-Switched Languages
Kapoor, R., Kumar, Y., Rajput, K., Shah, R. R., Kumaraguru, P., & Zimmermann, R. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33019951 · Cited by 23 · see citing works
Cyber Hate Classification: 'Othering' Language And Paragraph EmbeddingPreprint
Alorainy, W. S., Burnap, P., Liu, H., & Williams, M. L. (2018) · arXiv (Cornell University) · Cited by 21 · see citing works
African American English: Linguistic Introduction
Alim, H. S. (2006) · Journal of Linguistic Anthropology · doi:10.1525/jlin.2006.16.1.135 · Cited by 10 · see citing works
Examining a hate speech corpus for hate speech detection and popularity predictionPreprint
Klubička, F., & Fernández, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04661 · Cited by 8 · see citing works
African American Vernacular English: phonology
(2004) · doi:10.1515/9783110175325.1.383 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- E. Ashby Plant and Patricia G. Devine. 1998. Inter- nal and external motivation to respond without prej- udice. J. Pers. Soc. Psychol., 75(3):811-832.
- Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hi- erarchical attention networks for document classifi- cation. In NAACL.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“However, this may have disproportionately excluded casual language, code-switching, or dialectal diversity (e.g., Sap et al. (2019)) and may limit PaLM’s capability to model the non-dominant dialects across the English-speaking regions globally.”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · see citing works
“…associated with AAE than on text without these features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection systems score tweets containing features associated with AAE as more offensive than tweets without them (Davidson et al., 2019; Sap et al., 2019).”
BackgroundInfluentialDecolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912 · see citing works
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765 · see citing works
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“showing that more work is needed to address discrimination and hate speech risks in tandem [49, 103, 165, 194].”
BackgroundBias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
Self-citationTowards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · see citing works
“For example, in a natural language processing context, hate speech detection models use dialect markers as toxicity predictors, which can result in bias against minority groups [194].”
BackgroundHateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., & Mukherjee, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17745 · Cited by 585 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020) · Studies in Computational Intelligence · doi:10.1007/978-3-030-36687-2_77 · Cited by 396 · see citing works
Resources and benchmark corpora for hate speech detection: a systematic review
Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., & Patti, V. (2020) · Language Resources and Evaluation · doi:10.1007/s10579-020-09502-8 · Cited by 387 · see citing works
The hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352 · see citing works
Social Bias Frames: Reasoning about Social and Power Implications of Language
Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.486 · Cited by 375 · see citing works
Self-citationHateBERT: Retraining BERT for Abusive Language Detection in English
Caselli, T., Basile, V., Mitrović, J., & Granitzer, M. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.3 · Cited by 373 · see citing works
Directions in abusive language training data, a systematic review: Garbage in, garbage out
Vidgen, B., & Derczynski, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243300 · Cited by 335 · see citing works
Five sources of bias in natural language processing
Hovy, D., & Prabhumoye, S. (2021) · Language and Linguistics Compass · doi:10.1111/lnc3.12432 · Cited by 298 · see citing works
Developing an online hate classifier for multiple social media platforms
Salminen, J. O., Hopf, M., Chowdhury, S. A., Jung, S., Almerekhi, H., & Jansen, B. J. J. (2020) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-019-0205-6 · Cited by 295 · see citing works
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · arXiv · doi:10.48550/arxiv.2304.03738 · Cited by 282
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.234 · Cited by 279 · see citing works
Self-citationWILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
Social Biases in NLP Models as Barriers for Persons with Disabilities
Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.487 · Cited by 274 · see citing works
Timmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221134490 · Cited by 263 · see citing works
Should ChatGPT be biased? Challenges and risks of bias in large language models
Ferrara, E. (2023) · First Monday · doi:10.5210/fm.v28i11.13346 · Cited by 282 · see citing works
Hate speech detection and racial bias mitigation in social media based on BERT model
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237861 · Cited by 260 · see citing works
Bots and Misinformation Spread on Social Media: Implications for COVID-19
Himelein-Wachowiak, M., Giorgi, S., Devoto, A., Rahman, M. M., Ungar, L., Schwartz, H. A., Epstein, D. H., Leggio, L., & Curtis, B. L. (2021) · Journal of Medical Internet Research · doi:10.2196/26933 · Cited by 238 · see citing works
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
Davani, A. M., Díaz, M., & Prabhakaran, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00449 · Cited by 237 · see citing works
GeDi: Generative Discriminator Guided Sequence Generation
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.424 · Cited by 220 · see citing works
Re-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · see citing works
“The majority of fairness research looks at racial [46, 57, 121, 125, 183] and gender biases [42, 46, 197,”
BackgroundAI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233 · see citing works
Detecting and Monitoring Hate Speech in Twitter
Pereira-Kohatsu, J. C., Quijano-Sánchez, L., Liberatore, F., & Camacho‐Collados, M. (2019) · Sensors · doi:10.3390/s19214654 · Cited by 220 · see citing works
Towards generalisable hate speech detection: a review on obstacles and solutions
Yin, W., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.598 · Cited by 220 · see citing works
Oliva, T. D., Antonialli, D. M., & Gomes, A. (2020) · Sexuality & Culture · doi:10.1007/s12119-020-09790-w · Cited by 201 · see citing works
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., Choudhury, M. D., & Yang, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.29 · Cited by 201 · see citing works
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261 · see citing works
(2020) · MPG.PuRe (Max Planck Society) · Cited by 192 · see citing works
HateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H. Z., & Pierrehumbert, J. B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.4 · Cited by 235 · see citing works
Explanation in AI and law: Past, present and future
Atkinson, K. M., Bench-Capon, T. J. M., & Bollegala, D. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103387 · Cited by 179 · see citing works
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection
Vidgen, B., Thrush, T., Waseem, Z., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.132 · Cited by 174 · see citing works
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., & Smith, N. A. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2022.naacl-main.431 · Cited by 463 · see citing works
Self-citationDExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.522 · Cited by 199 · see citing works
Self-citationToxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178 · see citing works
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149 · see citing works
Algorithmic Censorship by Social Platforms: Power and Resistance
Cobbe, J. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00429-0 · Cited by 161 · see citing works
Evaluating the Effectiveness of Deplatforming as a Moderation Strategy on Twitter
Jhaver, S., Boylston, C., Yang, D., & Bruckman, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479525 · Cited by 169 · see citing works
Challenges of Hate Speech Detection in Social Media
Kovács, G., Alonso, P., & Saini, R. (2021) · SN Computer Science · doi:10.1007/s42979-021-00457-3 · Cited by 162 · see citing works
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145 · see citing works
The Evolution of the Manosphere across the Web
Horta Ribeiro, M., Blackburn, J., Bradlyn, B., De Cristofaro, E., Stringhini, G., Long, S., Greenberg, S., & Zannettou, S. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18053 · Cited by 10 · see citing works
A Survey on Stance Detection for Mis- and Disinformation Identification
Hardalov, M., Arora, A., Nakov, P., & Augenstein, I. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.94 · Cited by 147 · see citing works
The Importance of Modeling Social Factors of Language: Theory and Practice
Hovy, D., & Yang, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.49 · Cited by 136 · see citing works
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Nekoul, F. E., Lee, T. T. H., Adler, S., Jiang, A., & Weng, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26752 · Cited by 153 · see citing works
Contextualizing Hate Speech Classifiers with Post-hoc Explanation
Kennedy, B., Jin, X., Davani, A. M., Dehghani, M., & Ren, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.483 · Cited by 133 · see citing works
Ethical implications of text generation in the age of artificial intelligence
Illia, L., Colleoni, E., & Zyglidopoulos, S. (2023) · Business Ethics, the Environment & Responsibility · doi:10.1111/beer.12479 · Cited by 128 · see citing works
Persistent interaction patterns across social media platforms and over time
Avalle, M., Marco, N. D., Etta, G., Sangiorgio, E., Alipour, S., Bonetti, A., Alvisi, L., Antonio Scala, A., Baronchelli, A., Cinelli, M., et al. (2024) · Nature · doi:10.1038/s41586-024-07229-y · Cited by 126 · see citing works
Iandoli, L., Primario, S., & Zollo, G. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120924 · Cited by 122 · see citing works
The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation
Plank, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.731 · Cited by 121 · see citing works
Challenges in Automated Debiasing for Toxic Language Detection
Zhou, X., Sap, M., Swayamdipta, S., Choi, Y., & Smith, N. A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.274 · Cited by 116 · see citing works
Self-citationConcealed Data Poisoning Attacks on NLP Models
Wallace, E., Zhao, T., Feng, S., & Singh, S. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.13 · Cited by 115 · see citing works
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 118 · see citing works
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112 · see citing works
Sociological perspectives on artificial intelligence: A typological reading
Liu, Z. (2021) · Sociology Compass · doi:10.1111/soc4.12851 · Cited by 118 · see citing works
Li, L., Fan, L., Atreja, S., & Hemphill, L. (2024) · ACM Transactions on the Web · doi:10.1145/3643829 · Cited by 116 · see citing works
Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach
Kocoń, J., Figas, A., Gruza, M., Puchalska, D., Kajdanowicz, T., & Kazienko, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102643 · Cited by 103 · see citing works
Demoting Racial Bias in Hate Speech Detection
Xia, M., Field, A., & Tsvetkov, Y. (2020) · International Workshop on Natural Language Processing for Social Media (SocialNLP) · doi:10.18653/v1/2020.socialnlp-1.2 · Cited by 102 · see citing works
Systematic Inequalities in Language Technology Performance across the World’s Languages
Blasí, D. E., Anastasopoulos, A., & Neubig, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.376 · Cited by 140 · see citing works
Do Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incels
Horta Ribeiro, M., Jhaver, S., Zannettou, S., Blackburn, J., Stringhini, G., De Cristofaro, E., & West, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476057 · Cited by 124 · see citing works
Do Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incels
Ribeiro, M. H., Jhaver, S., Zannettou, S., Blackburn, J., De Cristofaro, E., Stringhini, G., & West, R. (2020) · arXiv · doi:10.48550/arxiv.2010.10397 · Cited by 102
Challenges and Strategies in Cross-Cultural NLP
Hershcovich, D., Frank, S., Lent, H., de Lhoneux, M., Abdou, M., Brandl, S., Bugliarello, E., Cabello Piqueras, L., Chalkidis, I., Cui, R., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.482 · Cited by 99 · see citing works
Geiger, R. S., Cope, D., Ip, J., Lotosh, M., Shah, A., Weng, J., & Tang, R. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00144 · Cited by 102 · see citing works
Recipes for Safety in Open-domain ChatbotsPreprint
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07079 · Cited by 130 · see citing works
Detecting East Asian Prejudice on Social Media
Vidgen, B., Hale, S. A., Guest, E., Margetts, H. Z., Broniatowski, D. A., Talat, Z., Botelho, A., Hall, M., & Tromble, R. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.19 · Cited by 95 · see citing works
The Limits of Social Media for Public Administration Research and Practice
Feeney, M. K., & Porumbescu, G. A. (2020) · Public Administration Review · doi:10.1111/puar.13276 · Cited by 93 · see citing works
How We Do Things With Words: Analyzing Text as Social and Cultural Data
Nguyen, D., Liakata, M., DeDeo, S., Eisenstein, J., Mimno, D., Tromble, R., Winters, J., Processing, S. N. L., & Processing, N. L. (2020) · Utrecht University Repository (Utrecht University) · Cited by 92 · see citing works
An Expert Annotated Dataset for the Detection of Online Misogyny
Guest, E., Vidgen, B., Mittos, A., Sastry, N., Gareth Tyson, A., & Margetts, H. Z. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.114 · Cited by 95 · see citing works
Kennedy, B., Atari, M., Davani, A. M., Yeh, L., Omrani, A., Kim, Y., Coombs, K., Havaldar, S., Portillo-Wightman, G., Gonzalez, E., et al. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-021-09569-x · Cited by 90 · see citing works
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Solaiman, I., & Dennison, C. (2021) · arXiv · doi:10.48550/arxiv.2106.10328 · Cited by 90 · see citing works
Multi-Dimensional Gender Bias Classification
Dinan, E., Fan, A., Wu, L., Weston, J., Kiela, D., & Williams, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.23 · Cited by 94 · see citing works
Rajadesingan, A., Resnick, P., & Budak, C. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7323 · Cited by 88 · see citing works
The Structure of Toxic Conversations on Twitter
Saveski, M., Roy, B., & Roy, D. K. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449861 · Cited by 88 · see citing works
Introducing CAD: the Contextual Abuse Dataset
Vidgen, B., Nguyen, D., Margetts, H. Z., Rossini, P., & Tromble, R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.182 · Cited by 88 · see citing works
Handling Bias in Toxic Speech Detection: A Survey
Garg, T., Masud, S., Suresh, T., & Chakraborty, T. (2023) · ACM Computing Surveys · doi:10.1145/3580494 · Cited by 86 · see citing works
Identifying and Measuring Annotator Bias Based on Annotators’ Demographic Characteristics
Kuwatly, H. A., Wich, M., & Groh, G. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.21 · Cited by 88 · see citing works
A Survey of Race, Racism, and Anti-Racism in NLP
Field, A., Blodgett, S. L., Talat, Z., & Tsvetkov, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.149 · Cited by 94 · see citing works
Markl, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533117 · Cited by 86 · see citing works
Hudley, A. H. C., Mallinson, C., & Bucholtz, M. (2024) · doi:10.1093/oso/9780197755303.001.0001 · Cited by 79 · see citing works
Lam, M. S., Gordon, M., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 78 · see citing works
Tran, K., & Nguyen, T. A. (2021) · Religions · doi:10.3390/rel12030208 · Cited by 99 · see citing works
García-Díaz, J. A., Jiménez-Zafra, S. M., Cumbreras, M. Á. G., & Valencia‐García, R. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-022-00693-x · Cited by 73 · see citing works
Czarnowska, P., Vyas, Y., & Shah, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00425 · Cited by 72 · see citing works
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 74 · see citing works
Addressing Unintended Bias in Toxicity Detection: An LSTM and Attention-Based Approach
Dai, W., Tao, J., Xu, Y., Feng, Z., & Chen, J. (2023) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica58456.2023.10405429 · Cited by 73 · see citing works
How We Do Things With Words: Analyzing Text as Social and Cultural Data
Nguyen, D., Liakata, M., DeDeo, S., Eisenstein, J., Mimno, D., Tromble, R., & Winters, J. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00062 · Cited by 118 · see citing works
A Survey of Offensive Language Detection for the Arabic Language
Husain, F., & Uzuner, Ö. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3421504 · Cited by 71 · see citing works
Competency Problems: On Finding and Removing Artifacts in Language Data
Gardner, M., Merrill, W., Dodge, J., Peters, M. E., Ross, A., Singh, S., & Smith, N. A. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.135 · Cited by 129 · see citing works
Self-citation“I Didn’t Know I Looked Angry”: Characterizing Observed Emotion and Reported Affect at Work
Kaur, H., McDuff, D., Williams, A. C., Teevan, J. B., & Iqbal, S. T. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517453 · Cited by 72 · see citing works
Arango, A., Pérez, J. E. P., & Poblete, B. (2020) · Information Systems · doi:10.1016/j.is.2020.101584 · Cited by 71 · see citing works
Content Removal as a Moderation Strategy
Srinivasan, K. B., Danescu-Niculescu-Mizil, C., Lee, L., & Tan, C. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359265 · Cited by 70 · see citing works
Designing Word Filter Tools for Creator-led Comment Moderation
Jhaver, S., Chen, Q. Z., Knauss, D., & Zhang, A. X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517505 · Cited by 72 · see citing works
Empirical Analysis of Multi-Task Learning for Reducing Identity Bias in Toxic Comment Detection
Vaidya, A., Mai, F., & Ning, Y. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7334 · Cited by 83 · see citing works
- Authors
- Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A.
- Affiliations
- University of Washington (US)
- Carnegie Mellon University (US)
- Allen Institute for Artificial Intelligence (US)
- Corresponding author
- Sap, Maarten
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 1668-1678
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2949678053
- Cited by
- 817
- References
- 40
- Citations by year
- 2026: 59 · 2025: 96 · 2024: 86 · 2023: 139 · 2022: 142 · 2021: 178 · 2020: 105 · 2019: 12
- FWCI
- 40.99
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1714566)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- hate speech detection, racial bias, African American English, dialects, annotation bias, hate speech datasets, priming, minority groups
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
