A Survey on Automatic Detection of Hate Speech in Text
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
The scientific study of hate speech, from a computer science point of view, is recent. This survey organizes and describes the current state of the field, providing a structured overview of previous approaches, including core algorithms, methods, and main features used. This work also discusses the complexity of the concept of hate speech, defined in many platforms and contexts, and provides a unifying definition. This area has an unquestionable potential for societal impact, particularly in online communities and digital media platforms. The development and systematization of shared resources, such as guidelines, annotated datasets in multiple languages, and algorithms, is a crucial step in advancing the automatic detection of hate speech.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recent studies(Jurgens etal. 2019; Fortuna and Nunes 2018) that have looked into tackling and proposing subtle hate detection suggest the consideration of making all subtle forms of discrimination, even jokes, as hate speech since they negatively affect some people psychological even though they are considered harmless(Douglass etal. 2016).”
From Misogynoir: challenges in detecting intersectional hate (2022) · cites this paper as Fortuna and Nunes 2018
Citations
Cites
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655 · cited 1×
“Nevertheless, detecting hate speech is a very important task, because it is connected with actual hate crimes [42, 65, 77], and automatic hate speech detection in text can also provide data about this phenomenon.”
BackgroundA Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media · doi:10.18653/v1/w17-1101 · Cited by 1,395 · cited 4×
“A problem that is referred in hate speech detection is that sentences must be classified and not words [66].”
BackgroundInfluentialAbusive Language Detection in Online User Content
Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., & Chang, Y. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883062 · Cited by 1,122 · cited 9×
“[10], abusive language [58], discrimination [69], profanity [23], toxicity [43], flaming [35], extremism [55, 62], and radicalization [2].”
BackgroundMethodInfluentialDetecting Hate Speech on the World Wide Web
Warner, W. B., & Hirschberg, J. (2012) · Cited by 964
Deep Learning for Hate Speech Detection in Tweets
Badjatiya, P., Gupta, S., Gupta, M., & Varma, V. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3054223 · Cited by 849 · cited 2×
“N-grams are one of the most used techniques in hate speech automatic detection and related tasks [4, 9, 19, 33, 52, 58, 77].”
MethodStanford typed dependencies manual
de Marnee, M.-C., & Manning, C. D. (2010) · Cited by 717
Hate Speech Detection with Comment Embeddings
Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., & Bhamidipati, N. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2740908.2742760 · Cited by 712 · cited 2×
“Some authors [25] use a paragraph2vec approach to classify language on user comments as abusive or clean and also to predict the central word in the message [25].”
Background
Research works printed but not linked: 18
- ACL., ALW1: 1st workshop on abusive language online.
- Burnap Peter, Proceedings of the Conference on the Internet, Policy 8 Politics. 1--18
- CHI2017. 2017. 2017 Workshop on online harassment. Retrieved from http://social.umd.edu/woh/. CHI2017. 2017. 2017 Workshop on online harassment. Retrieved from http://social.umd.edu/woh/.
- CONTACT., 2017, Interdisciplinary conference on Hate speech. Definitions, Interpretations and Practices.
- Davidson Thomas, 2017, Automated hate speech detection and the problem of offensive language. arXiv Preprint arXiv:1703.04009 · arXiv
- De Marneffe Marie-Catherine, 2008, Manning
- Pauw Guy De, 2016, First workshop on text analytics for cybersecurity and online safety (TA-COS
- Vigna Fabio Del, 2017, Proceedings of the 1st Italian Conference on Cybersecurity. 86--95
- Cambridge Dictionary. 2017. Profanity. Retrieved from https://dictionary.cambridge.org/dictionary/english/profanity. Cambridge Dictionary. 2017. Profanity. Retrieved from https://dictionary.cambridge.org/dictionary/english/profanity.
- Giblietto Fabio, 2015, Proceedings of the 4th Workshop on Making Sense of Microposts (#Microposts’14)
- Reporting Hate. 2017. Hate speech conference I.H.D.I.P. Retrieved from http://reportinghate.eu/contact2017/. Reporting Hate. 2017. Hate speech conference I.H.D.I.P. Retrieved from http://reportinghate.eu/contact2017/.
- Kwok Irene, 2013, Proceedings of the Association for the Advancement of Artificial Intelligence.
- Ross Bjorn, 2017, Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv Preprint arXiv:1701.08118 · arXiv
- Silva Leandro, 2016, Analyzing the targets of hate in online social media. arXiv Preprint arXiv:1603.07709 · arXiv
- Natalya Tarasova. 2016. Classification of Hate Tweets and Their Reasons using SVM. Master’s thesis. Uppsala Universitet. Natalya Tarasova. 2016. Classification of Hate Tweets and Their Reasons using SVM. Master’s thesis. Uppsala Universitet.
- Tulkens Stéphan, 2016, A dictionary-based approach to racism detection in dutch social media. arXiv Preprint arXiv:1608.08738 · arXiv
- Warner William, 2012, Proceedings of the Second Workshop on Language in Social Media. Association for Computational Linguistics, 19--26
- Yuan Shuhan, 2016, Proceedings of the International Conference on Extending Database Technology. 696--697
Cited by
Items 101–200 of 1,004
Racism is a Virus: Anti-Asian Hate and Counterhate in Social Media during the COVID-19 CrisisPreprint
Ziems, C., He, B., Soni, S., & Kumar, S. (2020) · arXiv (Cornell University) · Cited by 96
Wiedemann, G., Yimam, S. M., & Biemann, C. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.213 · Cited by 93
“these datasets. Hence, results of different datasets are hardly comparable, and training sets usually cannot be combined to obtain a more robust classification system. Schmidt and Wiegand (2017), and Fortuna and Nunes (2018) conducted insightful surveys on this rapidly growing field. Mandl et al. (2019), Stru et al. (20…”
BackgroundHate speech detection in social media: Techniques, recent trends, and future challenges
Rawat, A., Kumar, S., & Samant, S. S. (2024) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1648 · Cited by 113
García-Díaz, J. A., Jiménez-Zafra, S. M., Cumbreras, M. Á. G., & Valencia‐García, R. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-022-00693-x · Cited by 89
“In view of that, this study aims to contribute to the detection of hate-speech in Spanish, due to the growing need for research on this topic in languages other than English [15].”
BackgroundAddressing Unintended Bias in Toxicity Detection: An LSTM and Attention-Based Approach
Dai, W., Tao, J., Xu, Y., Feng, Z., & Chen, J. (2023) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica58456.2023.10405429 · Cited by 89
“Fortuna and Nunes [16] presented an all-encompassing survey on methodologies deployed in hate speech detection.”
BackgroundExposing the limits of Zero-shot Cross-lingual Hate Speech Detection
Nozza, D. (2021) · doi:10.18653/v1/2021.acl-short.114 · Cited by 75
“Second, available labeled corpora refer to different definitions of hate speech, collection strategies, and annotation frameworks (Fortuna and Nunes, 2018).”
BackgroundDetecting ethnicity-targeted hate speech in Russian social media texts
Pronoza, E., Panicheva, P., Koltsova, O. Y., & Rosso, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102674 · Cited by 84
Arango, A., Pérez, J. E. P., & Poblete, B. (2020) · Information Systems · doi:10.1016/j.is.2020.101584 · Cited by 97
Fortifying Toxic Speech Detectors Against Veiled Toxicity
Han, X., & Tsvetkov, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.622 · Cited by 60
Offline events and online hate
Lupu, Y., Sear, R. F., Velásquez, N., Leahy, R., Restrepo, N. J., Goldberg, B., & Johnson, N. F. (2023) · PLoS ONE · doi:10.1371/journal.pone.0278511 · Cited by 99
“A recent survey found that Twitter was by far the most studied source in articles using automated detection of online hate speech [26].”
BackgroundNikolov-Radivchev at SemEval-2019 Task 6: Offensive Tweet Classification with BERT and Ensembles
Nikolov, A., & Radivchev, V. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2123 · Cited by 72
“Survey papers describing key areas that have been explored for this task include (Schmidt and Wiegand, 2017), (Fortuna and Nunes, 2018) and (Malmasi and Zampieri, 2017).”
BackgroundOnline interventions for reducing hate speech and cyberhate: A systematic review
Windisch, S., Wiedlitzka, S., Olaghere, A., & Jenaway, E. M. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1243 · Cited by 85
“Another definition that also points out the ambiguity and challenges involved with identifying more subtle forms of hate speech, and also makes reference to the potential threat of hate speech escalating to offline violence, is put forward by Fortuna and Nunes (2018): “Hate speech is”
BackgroundAkhtar, S., Basile, V., & Patti, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15896 · Cited by 72
“Fortuna and Nunes [2018], in their recent survey on HS literature, provided a deep and thorough analysis of issue and challenges of automatic HS detection faced by research community.”
Democratic Transparency in the Platform Society
Gorwa, R., & Ash, T. G. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.013 · Cited by 64
On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark
Sun, H., Xu, G., Deng, J., Cheng, J., Zheng, C., Zhou, H., Peng, N., Zhu, X., & Huang, M. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.308 · Cited by 49
Sharif, O., & Hoque, M. M. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.12.022 · Cited by 84
A Large-Scale Semi-Supervised Dataset for Offensive Language IdentificationPreprint
Rosenthal, S., Atanasova, P., Karadzhov, G., Zampieri, M., & Nakov, P. (2020) · arXiv (Cornell University) · Cited by 83
Raman, N., Cao, M., Tsvetkov, Y., Kästner, C., & Vasilescu, B. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results · doi:10.1145/3377816.3381732 · Cited by 70
“Our work builds on prior research on detecting toxic language— from hate speech to microaggressions—in the Natural Language Processing (NLP) community [3, 12, 32], making two main contributions.”
BackgroundWhat’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A., & Viviano, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.24 · Cited by 69
“…sources such as Twitter (Founta et al., 2018; Basile et al., 2019; Waseem and Hovy, 2016) and Reddit (Tadesse et al., 2019; Farrell et al., 2019), with both ML-based (Badjatiya et al., 2017) and count-based approaches (Davidson et al., 2017) achieving comparable results (Fortuna and Nunes, 2018).”
ResultGongane, V. U., Munot, M. V., & Anuse, A. D. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00248-9 · Cited by 91
“[ 47 ] T ox i c l a ngu a g e T h e t ox i c l a ngu a g e i s u s e d i n t h e f o r m o f c o mm e n t s w h i c h i n c l ud e r ud e , d i s r e s p ec t f u l o r un r ea s on a b l e m e ss a g e s t h a t ca n m a k e o t h e r u s e r s t o e x it [ 15 ] *”
Influential“Short is the Road that Leads from Fear to Hate”: Fear Speech in Indian WhatsApp Groups
Saha, P., Mathew, B., Garimella, K., & Mukherjee, A. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450137 · Cited by 59
“styles, even in subtle forms or when humour is used [22]".”
BackgroundStappen, L., Brunn, F., & Schuller, B. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13850 · Cited by 77
“As hate speech detection is already very challenging to model in an uni-language setting (MacAvaney et al., 2019; Zhang and Luo, 2019; Fortuna and Nunes, 2018), most existing work in this and related fields (Benballa et al.”
HurtBERT: Incorporating Lexical Features with BERT for the Detection of Abusive Language
Koufakou, A., Pamungkas, E. W., Basile, V., & Patti, V. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.5 · Cited by 68
“The automatic classification of abusive and offensive language is a complex problem, that has raised a growing interest in the Natural Language Processing community in the last decade or so (Fortuna and Nunes, 2018; Vidgen et al., 2019; Poletto et al., 2020).”
BackgroundText Detoxification using Large Pre-trained Neural Models
Dale, D., Voronov, A. Y., Dementieva, D., Logacheva, V., Kozlova, O., Semenov, N., & Panchenko, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.629 · Cited by 117
“While annotators agree when labelling serious cases of toxicity such as hate speech (Fortuna and Nunes, 2018), the labelling of less severe toxicity is subjective and depends on the annotator’s background (Al Kuwatly et al.”
BackgroundEmotion Based Hate Speech Detection using Multimodal LearningPreprint
Rana, A., & Jha, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06218 · Cited by 89
“Although various definitions of hate speech exist, the common elements we considered are [42] :”
BackgroundAn Empirical Study of AI-Generated Text Detection Tools
Akram, A. (2023) · Advances in Machine Learning & Artificial Intelligence · doi:10.33140/amlai.04.02.03 · Cited by 77
Papcunová, J., Martončik, M., Fedáková, D., Kentoš, M., Bozogáňová, M., Srba, I., Moro, R., Pikuliak, M., Šimko, M., & Adamkovič, M. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00561-0 · Cited by 51
“First, Fortuna and Nunes [10] identified four dimensions (that can be considered analogous to indicators) that allowed them to compare definitions of hate speech: (1) hate speech has specific targets, (2) hate speech is to incite violence or hate; (3) hate speech is to attack or diminish; and (4) hate speech can be ex…”
BackgroundInfluentialCharacterizing network dynamics of online hate communities around the COVID-19 pandemic
Uyheng, J., & Carley, K. M. (2021) · Applied Network Science · doi:10.1007/s41109-021-00362-x · Cited by 70
ABMM: Arabic BERT-Mini Model for Hate-Speech Detection on Social Media
Almaliki, M., Almars, A. M., Gad, I., & Atlam, E.-S. (2023) · Electronics · doi:10.3390/electronics12041048 · Cited by 73
Anjum, & Katarya, R. (2023) · International Journal of Information Security · doi:10.1007/s10207-023-00755-2 · Cited by 65
“Similarly, the survey paper [4] explains the short, structured overview of hate speech using NLP.”
Misinformation and Its Correction
Wittenberg, C., & Berinsky, A. J. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.009 · Cited by 58
A Comparative Study of Using Pre-trained Language Models for Toxic Comment Classification
Zhao, Z., Zhang, Z., & Hopfgartner, F. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442442.3452313 · Cited by 80
“In earlier studies, the majority of research handle TCC as a binary classification problem that separates one particular type of toxic comment from all the others [13].”
BackgroundWhich politicians receive abuse? Four factors illuminated in the UK general election 2019
Gorrell, G., Bakir, M. E., Roberts, I., Greenwood, M. A., & Bontcheva, K. (2020) · EPJ Data Science · doi:10.1140/epjds/s13688-020-00236-9 · Cited by 63
del Valle-Cano, G., Quijano-Sánchez, L., Liberatore, F., & Gómez, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119446 · Cited by 76
Reconsidering Tweets: Intervening during Tweet Creation Decreases Offensive Content
Katsaros, M., Yang, K., & Fratamico, L. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19308 · Cited by 87
Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models
Röttger, P., Seelawi, H., Nozza, D., Talat, Z., & Vidgen, B. (2022) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2022.woah-1.15 · Cited by 118
A Web Interface for Analyzing Hate Speech
Vrysis, L., Vryzas, N., Kotsakis, R., Saridou, T. A., Matsiola, M., Veglis, A. A., Calderón, C. A., & Dimoulas, C. A. (2021) · Future Internet · doi:10.3390/fi13030080 · Cited by 68
“5): “Hate speech is language that attacks or diminishes, that incites violence or hate against groups, based on specific characteristics such as physical appearance, religion, descent, national or ethnic origin, sexual orientation, gender identity or other, and it can occur with different linguistic styles, even in su…”
BackgroundAngryBERT: Joint Learning Target and Emotion for Hate Speech Detection
Awal, R., Cao, R., Lee, R. K.-W., & Mitrović, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-75762-5_55 · Cited by 60
Torregrosa, J., Bello-Orgaz, G., Martínez‐Cámara, E., Ser, J. D., & Camacho, D. (2022) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03658-z · Cited by 77
“The first type has analyzed NLP contributions to areas conceptually related to extremism, such as hate speech (Fortuna and Nunes 2018) or law enforcement (Edwards et al. 2015).”
BackgroundInfluentialEvaluating GPT-3 Generated Explanations for Hateful Content Moderation
Wang, H., Hee, M. S., Awal, M. R., Choo, K. T. W., & Lee, R. K.-W. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/694 · Cited by 97
Grimminger, L., & Klinger, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01664 · Cited by 73
“(2021); Fortuna and Nunes (2018); Schmidt and Wiegand (2017).”
BackgroundLatent Hatred: A Benchmark for Understanding Implicit Hate SpeechPreprint
ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., Choudhury, M. D., & Yang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05322 · Cited by 83
“Inferiority language is also related to assaults on human dignity (Kennedy et al., 2018), dominance (Saha et al., 2018), and declarations of superiority of the in-group (Fortuna and Nunes, 2018).”
BackgroundInfluentialHateminers : Detecting Hate speech against WomenPreprint
Saha, P., Mathew, B., Goyal, P., & Mukherjee, A. (2018) · arXiv (Cornell University) · doi:10.13140/rg.2.2.33967.18081 · Cited by 70
Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58
“It goes by many names and subcate-gories in the literature, including hate speech (Fortuna and Nunes 2018), aggression (Kumar et al. 2018), toxicity (Wul-czyn, Thain, and Dixon 2017), cyberbullying (Hosseinmardi et al. 2015), harassment (Golbeck et al. 2017) and incivility (Anderson et al. 2016).”
BackgroundLu, J., Xu, B., Zhang, X., Min, C., Yang, L., & Lin, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.898 · Cited by 58
Using Twitter to track immigration sentiment during early stages of the COVID-19 pandemic
Rowe, F., Mahony, M., Graells-Garrido, E., Rango, M., & Sievers, N. (2021) · Data & Policy · doi:10.1017/dap.2021.38 · Cited by 65
“A related and growing area of research, in which Twitter has displayed great potential, is in the detection and characterisation of hate speech (Fortuna and Nunes, 2018; Ribeiro et al., 2018).”
BackgroundSoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice
Singhal, M., Ling, C., Paudel, P., Thota, P., Kumarswamy, N., Stringhini, G., & Nilizadeh, S. (2023) · 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp57164.2023.00056 · Cited by 57
“Some recent surveys [122], [148], [161], [265], [327], [339] have reviewed detection techniques with respect to either hate speech or misinformation.”
BackgroundHate Speech Detection on Indonesian Long Text Documents Using Machine Learning Approach
Aulia, N., & Budi, I. D. (2019) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1145/3330482.3330491 · Cited by 62
“The number of hate speech has been growing in recent years [1], both through offline and online communication.”
BackgroundInfluentialChatbots Facilitating Consensus-Building in Asynchronous Co-Design
Shin, J., Hedderich, M. A., Lucero, A., & Oulasvirta, A. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545671 · Cited by 68
“Work on NLP has studied detection of unsuitable or malicious user content extensively, covering use cases such as spam [42], fake reviews [43] and hate speech [11].”
Lu, C., & Jurgens, D. (2022) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2022.woah-1.8 · Cited by 64
“Multiple approaches to recognizing such abusive language have focused on identifying explicit forms of abuse, such as using trans-specific slurs (Waseem et al., 2017; Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018).”
BackgroundModeling Annotator Perspective and Polarized Opinions to Improve Hate Speech Detection
Akhtar, S., Basile, V., & Patti, V. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7473 · Cited by 50
“The majority of computational approaches to hate speech detection are based on supervised machine learning, including deep learning, but also Support Vector Machines, Random Forest, Logistic Regression and Decision Trees (Fortuna and Nunes 2018; Schmidt and Wiegand 2017).”
BackgroundMethodHateGAN: Adversarial Generative-Based Data Augmentation for Hate Speech Detection
Cao, R., & Lee, R. K.-W. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.557 · Cited by 54
“Researchers have also proposed many traditional and deep learning hate speech classification methods to detect hate speeches in online social media automatically (Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018).”
BackgroundHassan, S., Huenerfauth, M., & Alm, C. O. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.267 · Cited by 32
Hate Speech and Counter Speech Detection: Conversational Context Does Matter
Yu, X., Blanco, E., & Hong, L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.433 · Cited by 89
Pavlopoulos, J., Thain, N., Dixon, L., & Androutsopoulos, I. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2102 · Cited by 71
“For recent overviews of hate speech detection, consult Schmidt and Wiegand (2017) and Fortuna and Nunes (2018).”
BackgroundIn Data We Trust: A Critical Analysis of Hate Speech Detection Datasets
Madukwe, K. J., Gao, X., & Xue, B. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.18 · Cited by 83
“As highlighted in (Fortuna and Nunes, 2018), the majority of the studies in this area of hate speech detection collected and annotated their own datasets, however, some were not made publicly available.”
BackgroundInfluentialBots Amplify and Redirect Hate Speech in Online Discourse About Racism During the COVID-19 Pandemic
Uyheng, J., Bellutta, D., & Carley, K. M. (2022) · Social Media + Society · doi:10.1177/20563051221104749 · Cited by 61
Inequalities and content moderation
Gregorio, G. D., & Stremlau, N. (2023) · Global Policy · doi:10.1111/1758-5899.13243 · Cited by 84
Improving hate speech detection using Cross-Lingual Learning
Firmino, A. A., Baptista, C. D. S., & de Paiva, A. C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121115 · Cited by 58
Diaz-Garcia, J. A., & Carvalho, J. P. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70029 · Cited by 57
“In recent years, several surveys have explored hate speech detection (Fortuna and Nunes 2018); however, there is a noticeable gap in covering the latest advancements, particularly the emergence of cutting-edge models like LLMs that have been released in the past year.”
Proceedings of the Third Workshop on Abusive Language Online
(2019) · doi:10.18653/v1/w19-35 · Cited by 54
Implicitly Abusive Language – What does it actually look like and why are we not getting there?
Wiegand, M., Ruppenhofer, J., & Eder, E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.48 · Cited by 112
“A more detailed summary of the methods explored can be found in Schmidt and Wiegand (2017) and Fortuna and Nunes (2018).”
MethodAbusive language detection in youtube comments leveraging replies as conversational context
Ashraf, N., Zubiaga, A., & Gelbukh, A. F. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.742 · Cited by 61
Althobaiti, M. J. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.01305109 · Cited by 66
“Among them, [23] classifies online content into hate speech, offensive, neither offensive nor hate-speech, while [24] classifies online content into abusive, hateful, normal, or spam.”
BackgroundCombating hate speech using an adaptive ensemble learning model with a case study on COVID-19
Agarwal, S., & Chowdary, C. R. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115632 · Cited by 58
Detecting Suspicious Texts Using Machine Learning Techniques
Sharif, O., Hoque, M. M., Kayes, A. S. M., Nowrozy, R., & Sarker, I. H. (2020) · Applied Sciences · doi:10.3390/app10186527 · Cited by 62
Evaluating text classification: A benchmark study
Reusens, M., Stevens, A., Tonglet, J., Smedt, J. D., Verbeke, W., Broucke, S. V., & Baesens, B. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124302 · Cited by 58
Nascimento, F. R. S., Cavalcanti, G. D. C., & Costa‐Abreu, M. D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117032 · Cited by 58
Uncovering hidden opinions: social norms and the expression of xenophobic attitudes
Álvarez-Benjumea, A. (2022) · European Sociological Review · doi:10.1093/esr/jcac056 · Cited by 51
Respectful or Toxic? Using Zero-Shot Learning with Language Models to Detect Hate Speech
Plaza-del-Arco, F. M., Nozza, D., & Hovy, D. (2023) · doi:10.18653/v1/2023.woah-1.6 · Cited by 114
“…face significant challenges, like the limited availability of labeled data, especially in languages other than English (Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018), and the subjective nature of hate speech, which varies based on cultural background, personal experiences, and individual…”
BackgroundHARE: Explainable Hate Speech Detection with Step-by-Step Reasoning
Yang, Y., Kim, J., Kim, Y., Ho, N., Thorne, J., & Yun, S.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.365 · Cited by 92
Detecting cyberbullying using deep learning techniques: a pre-trained glove and focal loss technique
Koshiry, A. M. E., Eliwa, E. H. I., El‐Hafeez, T. A., & Khairy, M. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1961 · Cited by 52
“Fortuna & Nunes (2018) have provided a search method to choose papers on detecting hate speech from the internet.”
BackgroundRodríguez, A., Chen, Y., & Argueta, C. (2022) · IEEE Access · doi:10.1109/access.2022.3151098 · Cited by 56
A Multimodal Memes Classification: A Survey and Open Research Issues
Afridi, T. H., Alam, A., Khan, M. N., Khan, J., & Lee, Y.-K. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-66840-2_109 · Cited by 62
Hate is the New Infodemic: A Topic-aware Modeling of Hate Speech Diffusion on Twitter
Masud, S., Dutta, S., Makkar, S., Jain, C., Goyal, V., Das, A., & Chakraborty, T. (2021) · 2021 IEEE 37th International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00050 · Cited by 38
Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model
Fraser, K., Nejadgholi, I., & Kiritchenko, S. (2021) · doi:10.18653/v1/2021.acl-long.50 · Cited by 86
“…including hate speech, cyber-bullying, on-line harassment, and other types of offensive and toxic behaviors, has been a focus of substantial research effort in the NLP community in the past decade (e.g. see surveys by Schmidt and Wiegand (2017); Fortuna and Nunes (2018); Vidgen et al. (2019)).”
BackgroundA Deep Learning Framework for the Detection of Malay Hate Speech
Maity, K., Bhattacharya, S., Saha, S., & Seera, M. (2023) · IEEE Access · doi:10.1109/access.2023.3298808 · Cited by 46
“…speech presents challenges in understanding contextual nuances, addressing data bias, handling multilingual and code-switching text, adapting to the evolving nature of hate speech, dealing with subjectivity and ambiguity, countering evasion techniques, and considering ethical considerations [16].”
BackgroundAlqarni, F., Sagheer, A., Alabbad, A., & Hamdoun, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-99515-6 · Cited by 34
Arif, M. (2021) · Journal of Information Security and Cybercrimes Research · doi:10.26735/gbtv9013 · Cited by 43
“Early detection of different social anomalies, including cyberbullying, hate speech [13, 14], trolling [15], fake news [16], rumors [17], counterfeit profile detection [18], misogyny [19], etc., is becoming a trend in recent social media-based research.”
BackgroundConversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations
Bao, J., Wu, J., Zhang, Y., Chandrasekharan, E., & Jurgens, D. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450122 · Cited by 73
“Multiple models have been proposed for handling different aspects of antisocial behavior [31] and that these models frequently have gaps in what they recognize [3, 43], can encode biases with respect to social groups [68], and be susceptible to adversarial attacks [37].”
BackgroundTowards multidomain and multilingual abusive language detection: a survey
Pamungkas, E. W., Basile, V., & Patti, V. (2021) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01609-1 · Cited by 60
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44
“Several surveys cover approaches to this problem (Schmidt & Wiegand, 2017; Fortuna & Nunes, 2018; Vidgen et al., 2019), and there exist reviews of offensive language datasets (Fortuna et al., 2020; Vidgen & Derczynski, 2020).”
BackgroundSaifullah, S., Dreżewski, R., Dwiyanto, F. A., Aribowo, A. S., Fauziah, Y., & Cahyana, N. H. (2024) · Applied Sciences · doi:10.3390/app14031078 · Cited by 44
Sequential targeting: A continual learning approach for data imbalance in text classification
Jang, J., Kim, Y., Choi, K.-H., & Suh, S. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115067 · Cited by 48
Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech
Wullach, T., Adler, A., & Minkov, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.402 · Cited by 44
Deep cepstrum-wavelet autoencoder: A novel intelligent sonar classifier
Jia, H., Khishe, M., Mohammadi, M., & Rashidi, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117295 · Cited by 47
Bots and Computational Propaganda: Automation for Communication and Control
Woolley, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.006 · Cited by 59
SoK: Content Moderation for End-to-End Encryption
Scheffler, S., & Mayer, J. R. (2023) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2023-0060 · Cited by 52
“particular kinds of content that could merit moderation, such as nudity [93, 184, 194, 317] and hate speech [10, 115, 235, 354, 371].”
BackgroundNothing Micro About It: Examining Ableist Microaggressions on Social Media
Heung, S., Phutane, M., Azenkot, S., Marathe, M., & Vashistha, A. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544801 · Cited by 49
“Existing models are mostly trained to detect hate speech [19, 20, 24, 44]; however, some researchers are developing machine learning techniques to identify microaggressions (based on gender, race, and sexuality) [10].”
BackgroundMultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili
Wang, H., Yang, T. R., Naseem, U., & Lee, R. K.-W. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681521 · Cited by 73
“Current research on detecting hateful content has primarily concentrated on text-based analysis [20, 35], with recent advancements extending to multimodal forms, such as memes [10, 11, 24, 32, 36].”
BackgroundPlaza-del-Arco, F. M., Halat, S., Padó, S., & Klinger, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10255 · Cited by 42
“At the same time, social media posts can also contain harmful content like hate speech and offensive language (HOF), often eased by the quasi-anonymity on social media platforms [1].”
BackgroundIdentifying vulgarity in Bengali social media textual content
Sazzed, S. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.665 · Cited by 41
“To annotate a corpus for an NLP task ( e.g ., hate speech detection, sentiment classi fi cation, profanity detection), it is required to follow a set of guidelines ( Khan, Shahzad & Malik, 2021 ; Mehmood et al., 2019 ; Pradhan et al., 2020 ; Fortuna & Nunes, 2018 ; Sazzed, 2020a ).”
Kordyaka, B., Park, S., Krath, J., & Laato, S. (2023) · ACM Transactions on Social Computing · doi:10.1145/3580346 · Cited by 49
Fine-Grained Multilingual Hate Speech Detection Using Explainable AI and Transformers
Siddiqui, J. A., Yuhaniz, S. S., Mujtaba, G., Soomro, S. A., & Mahar, Z. A. (2024) · IEEE Access · doi:10.1109/access.2024.3470901 · Cited by 46
“The complexity of hate speech detection lies in the linguistic nuances and cultural contexts that influence the interpretation of text [8].”
BackgroundFacts and Where to Find Them: Empirical Research on Internet Platforms and Content Moderation
Keller, D., & Leerssen, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.011 · Cited by 61
Malik, P., Aggrawal, A., & Vishwakarma, D. K. (2021) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc51019.2021.9418395 · Cited by 50
Human-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor
Kazienko, P., Bielaniewicz, J., Gruza, M., Kanclerz, K., Karanowski, K., Miłkowski, P., & Kocoń, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.010 · Cited by 38
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers
Laugier, L., Pavlopoulos, J., Sorensen, J. S., & Dixon, L. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.124 · Cited by 66
Ruddit: Norms of Offensiveness for English Reddit Comments
Hada, R., Sudhir, S., Mishra, P., Yannakoudakis, H., Mohammad, S. M., & Shutova, E. (2021) · doi:10.18653/v1/2021.acl-long.210 · Cited by 57
“Surveys by Schmidt and Wiegand (2017); Fortuna and Nunes (2018); Mishra et al. (2019); Vidgen and Derczynski (2020) discuss various existing datasets and their compositions in detail.”
BackgroundBalayn, A., Yang, J., Szlávik, Z., & Bozzon, A. (2021) · ACM Transactions on Social Computing · doi:10.1145/3479158 · Cited by 49
“Machine learning approaches are considered the best solutions [101], due to their promise to achieve reasonable detection performance at scale.”
BackgroundMethodInfluential
Metadata
- Authors
- Fortuna, P., & Nunes, S.
- Affiliations
- INESC TEC (PT)
- Universidade do Porto (PT)
- Venue
- ACM Computing Surveys
- Published
- 2018-07-31
- Volume
- 51
- Issue
- 4
- Pages
- 1-30
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2887782043
- Cited by
- 1,241
- References
- 52
- Citations by year
- 2026: 139 · 2025: 195 · 2024: 141 · 2023: 185 · 2022: 158 · 2021: 199 · 2020: 144 · 2019: 71 · 2018: 8
- FWCI
- 59.53
- Type
- journal article
- Language
- en
- Open access
- Locations
- ACM Computing Surveys (publishedVersion)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, other-oa)
- Funders
- FourEyes, a research line within project TEC4Growth—Pervasive Intelligence
- North Portugal Regional Operational Programme
- European Regional Development Fund
- Enhancers and Proofs of Concept with Industrial (Impact/NORTE-01- 0145-FEDER-000020)
- PORTUGAL 2020 Partnership Agreement
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- hate speech detection, text classification, definition of hate speech, annotated dataset, multilingual dataset, online communities, digital media platforms, social impact
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.