A Survey on Automatic Detection of Hate Speech in Text
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
The scientific study of hate speech, from a computer science point of view, is recent. This survey organizes and describes the current state of the field, providing a structured overview of previous approaches, including core algorithms, methods, and main features used. This work also discusses the complexity of the concept of hate speech, defined in many platforms and contexts, and provides a unifying definition. This area has an unquestionable potential for societal impact, particularly in online communities and digital media platforms. The development and systematization of shared resources, such as guidelines, annotated datasets in multiple languages, and algorithms, is a crucial step in advancing the automatic detection of hate speech.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recent studies(Jurgens etal. 2019; Fortuna and Nunes 2018) that have looked into tackling and proposing subtle hate detection suggest the consideration of making all subtle forms of discrimination, even jokes, as hate speech since they negatively affect some people psychological even though they are considered harmless(Douglass etal. 2016).”
From Misogynoir: challenges in detecting intersectional hate (2022) · cites this paper as Fortuna and Nunes 2018
Citations
Cites
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655 · cited 1×
“Nevertheless, detecting hate speech is a very important task, because it is connected with actual hate crimes [42, 65, 77], and automatic hate speech detection in text can also provide data about this phenomenon.”
BackgroundA Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media · doi:10.18653/v1/w17-1101 · Cited by 1,395 · cited 4×
“A problem that is referred in hate speech detection is that sentences must be classified and not words [66].”
BackgroundInfluentialAbusive Language Detection in Online User Content
Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., & Chang, Y. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883062 · Cited by 1,122 · cited 9×
“[10], abusive language [58], discrimination [69], profanity [23], toxicity [43], flaming [35], extremism [55, 62], and radicalization [2].”
BackgroundMethodInfluentialDetecting Hate Speech on the World Wide Web
Warner, W. B., & Hirschberg, J. (2012) · Cited by 964
Deep Learning for Hate Speech Detection in Tweets
Badjatiya, P., Gupta, S., Gupta, M., & Varma, V. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3054223 · Cited by 849 · cited 2×
“N-grams are one of the most used techniques in hate speech automatic detection and related tasks [4, 9, 19, 33, 52, 58, 77].”
MethodStanford typed dependencies manual
de Marnee, M.-C., & Manning, C. D. (2010) · Cited by 717
Hate Speech Detection with Comment Embeddings
Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., & Bhamidipati, N. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2740908.2742760 · Cited by 712 · cited 2×
“Some authors [25] use a paragraph2vec approach to classify language on user comments as abusive or clean and also to predict the central word in the message [25].”
Background
Research works printed but not linked: 18
- ACL., ALW1: 1st workshop on abusive language online.
- Burnap Peter, Proceedings of the Conference on the Internet, Policy 8 Politics. 1--18
- CHI2017. 2017. 2017 Workshop on online harassment. Retrieved from http://social.umd.edu/woh/. CHI2017. 2017. 2017 Workshop on online harassment. Retrieved from http://social.umd.edu/woh/.
- CONTACT., 2017, Interdisciplinary conference on Hate speech. Definitions, Interpretations and Practices.
- Davidson Thomas, 2017, Automated hate speech detection and the problem of offensive language. arXiv Preprint arXiv:1703.04009 · arXiv
- De Marneffe Marie-Catherine, 2008, Manning
- Pauw Guy De, 2016, First workshop on text analytics for cybersecurity and online safety (TA-COS
- Vigna Fabio Del, 2017, Proceedings of the 1st Italian Conference on Cybersecurity. 86--95
- Cambridge Dictionary. 2017. Profanity. Retrieved from https://dictionary.cambridge.org/dictionary/english/profanity. Cambridge Dictionary. 2017. Profanity. Retrieved from https://dictionary.cambridge.org/dictionary/english/profanity.
- Giblietto Fabio, 2015, Proceedings of the 4th Workshop on Making Sense of Microposts (#Microposts’14)
- Reporting Hate. 2017. Hate speech conference I.H.D.I.P. Retrieved from http://reportinghate.eu/contact2017/. Reporting Hate. 2017. Hate speech conference I.H.D.I.P. Retrieved from http://reportinghate.eu/contact2017/.
- Kwok Irene, 2013, Proceedings of the Association for the Advancement of Artificial Intelligence.
- Ross Bjorn, 2017, Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv Preprint arXiv:1701.08118 · arXiv
- Silva Leandro, 2016, Analyzing the targets of hate in online social media. arXiv Preprint arXiv:1603.07709 · arXiv
- Natalya Tarasova. 2016. Classification of Hate Tweets and Their Reasons using SVM. Master’s thesis. Uppsala Universitet. Natalya Tarasova. 2016. Classification of Hate Tweets and Their Reasons using SVM. Master’s thesis. Uppsala Universitet.
- Tulkens Stéphan, 2016, A dictionary-based approach to racism detection in dutch social media. arXiv Preprint arXiv:1608.08738 · arXiv
- Warner William, 2012, Proceedings of the Second Workshop on Language in Social Media. Association for Computational Linguistics, 19--26
- Yuan Shuhan, 2016, Proceedings of the International Conference on Extending Database Technology. 696--697
Cited by
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
“…create a range of harms, such as promoting social stereotypes that perpetuate the derogatory representation or unfair treat-ment of marginalised groups [22], inciting hate or violence [57], causing profound offence [199], or reinforcing social norms that exclude or marginalise identities [15, 58].”
BackgroundHate speech detection: Challenges and solutions
MacAvaney, S., Yao, H.-R., Yang, E., Russell, K., Goharian, N., & Frieder, O. (2019) · PLOS ONE · doi:10.1371/journal.pone.0221152 · Cited by 566
“’s definition is based on an analysis of the following characteristics from other definitions [8]:”
BackgroundHateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., & Mukherjee, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17745 · Cited by 513
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Racial Bias in Hate Speech and Abusive Language Detection Datasets
Davidson, T., Bhattacharya, D., & Weber, I. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3504 · Cited by 414
“While hate speech and abusive language detection has become an important area for natural language processing research (Schmidt and Wiegand, 2017; Waseem et al., 2017; Fortuna and Nunes, 2018), there has been little work addressing the potential for these systems to be biased.”
BackgroundA BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020) · Studies in Computational Intelligence · doi:10.1007/978-3-030-36687-2_77 · Cited by 396
Resources and benchmark corpora for hate speech detection: a systematic review
Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., & Patti, V. (2021) · Language Resources and Evaluation · doi:10.1007/s10579-020-09502-8 · Cited by 385
“We carefully read each work and labeled it with a set of specifically-designed labels, sorting our list by research field (e.g., field-socialsciences , field-NLP , etc.), main focus (e.g., content-resource , content-system , etc.), methodology (e.g., method-nn for neural nets, method-ml for machine learning, etc.), specifi…”
BackgroundMethodInfluentialSocial Media, Echo Chambers, and Political Polarization
Barberá, P. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.004 · Cited by 373
The hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352
“Hate speech There has been a lot of work in recent years on detecting hate speech in network science [61] and natural language processing [81, 65, 19].”
BackgroundDirections in abusive language training data, a systematic review: Garbage in, garbage out
Vidgen, B., & Derczynski, L. (2020) · PLOS ONE · doi:10.1371/journal.pone.0243300 · Cited by 318
“• Fortuna and Nunes [26] provide an end-to-end review of hate speech research, including the motivations for studying online hate, definitional challenges, dataset creation/sharing, and technical advances, both in terms of feature selection and algorithmic architecture ([26], 2018).”
BackgroundMethodInfluentialHateBERT: Retraining BERT for Abusive Language Detection in English
Caselli, T., Basile, V., Mitrović, J., & Granitzer, M. (2021) · Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021) · doi:10.18653/v1/2021.woah-1.3 · Cited by 307
Social Media and Democracy: The State of the Field, Prospects for Reform
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960 · Cited by 295
HateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H., & Pierrehumbert, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.4 · Cited by 187
“tified linguistic challenges in hate speech detection from review articles and commentaries (e.g. Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018; Vidgen et al., 2019).”
BackgroundDeveloping an online hate classifier for multiple social media platforms
Salminen, J., Hopf, M., Chowdhury, S. A., Jung, S.-G., Almerekhi, H., & Jansen, B. J. (2020) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-019-0205-6 · Cited by 286
Misinformation, Disinformation, and Online Propaganda
Guess, A. M., & Lyons, B. A. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.003 · Cited by 284
A systematic review of hate speech automatic detection using natural language processing
Jahan, M. S., & Oussalah, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126232 · Cited by 279
Mandl, T., Modha, S., Majumder, P., Patel, D., Dave, M., Mandlia, C., & Patel, A. (2019) · Proceedings of the 11th Forum for Information Retrieval Evaluation · doi:10.1145/3368567.3368584 · Cited by 279
“Recent overview papers provide a good introduction to the scientific issues that are involved in Hate Speech identification [12,36].”
BackgroundSentiment Analysis for Fake News Detection
Alonso, M. A., Vilares, D., Gómez-Rodríguez, C., & Vilares, J. (2021) · Electronics · doi:10.3390/electronics10111348 · Cited by 239
Exploring Misogyny across the Manosphere in Reddit
Farrell, T., Fernandez, M., Novotny, J., & Alani, H. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326045 · Cited by 265
“While the problem of online hate speech has been the focus of a wide body of research during the last few years [15], computational approaches targeting the problem of misogyny in particular are scarce and very recent.”
BackgroundA survey of Twitter research: Data model, graph structure, sentiment analysis and attacks
Antonakaki, D., Fragopoulou, P., & Ioannidis, S. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114006 · Cited by 265
Hate speech detection and racial bias mitigation in social media based on BERT model
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237861 · Cited by 260
“The majority of contributions in classic supervised machine learning-based methods, for hate speech detection, rely on different text mining-based features or user-based and platform-based metadata [4, 17,18], which require them to define an applicable feature extraction method and prevent them to generalize their app…”
BackgroundTowards generalisable hate speech detection: a review on obstacles and solutions
Yin, W., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.598 · Cited by 207
“Among these, Fortuna and Nunes (2018) provided an in-depth review of definitions, Vidgen et al. (2019) concisely summarised various challenges for the detection of abusive language in general, Poletto et al. (2020) and Vidgen and Derczynski (2020) created extensive lists of dataset and corpora…”
BackgroundMethodLatent Hatred: A Benchmark for Understanding Implicit Hate Speech
ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., De Choudhury, M., & Yang, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.29 · Cited by 190
“Inferiority language is also related to assaults on human dignity (Kennedy et al., 2018), dominance (Saha et al., 2018), and declarations of superiority of the in-group (Fortuna and Nunes, 2018).”
BackgroundInfluentialChallenges and frontiers in abusive content detection
Vidgen, B., Harris, A., Nguyen, D., Tromble, R., Hale, S., & Margetts, H. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3509 · Cited by 199
“Advances in machine learning and NLP have led to marked improvements in abusive content detection systems’ performance (Fortuna & Nunes, 2018; Schmidt & Wiegand, 2017).”
BackgroundDetecting and Monitoring Hate Speech in Twitter
Pereira-Kohatsu, J. C., Quijano-Sánchez, L., Liberatore, F., & Camacho-Collados, M. (2019) · Sensors · doi:10.3390/s19214654 · Cited by 214
“In fact, previous works [12] have focused on the message and not on the actors involved.”
BackgroundInfluentialDeep Learning Models for Multilingual Hate Speech DetectionPreprint
Aluru, S. S., Mathew, B., Saha, P., & Mukherjee, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.06465 · Cited by 201
“[14] conducted a comprehensive survey on this subject.”
BackgroundDETECTION OF HATE SPEECH IN SOCIAL NETWORKS: A SURVEY ON MULTILINGUAL CORPUS
Al-Hassan, A., & Al-Dossari, H. (2019) · Computer Science & Information Technology(CS & IT) · doi:10.5121/csit.2019.90208 · Cited by 179
Thirty years of research into hate speech: topics of interest and their evolution
Tontodimamma, A., Nissi, E., Sarra, A., & Fontanella, L. (2020) · Scientometrics · doi:10.1007/s11192-020-03737-6 · Cited by 195
“Previous surveys highlighted the state of the art and the evolution of research on hate speech (Schmidt and Wiegand 2017; Fortuna and Nunes 2018; MacAvaney et al. 2019; Waqas et al. 2019).”
BackgroundVulnerable community identification using hate speech detection on social media
Mossie, Z., & Wang, J.-H. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.102087 · Cited by 187
Advances in Machine Learning Algorithms for Hate Speech Detection in Social Media: A Review
Mullah, N. S., & Zainon, W. M. N. W. (2021) · IEEE Access · doi:10.1109/access.2021.3089515 · Cited by 162
“Different calls have been made to SM providers to filter each comment before allowing it into the public domain [4], [5].”
BackgroundDetection of Hate Speech using BERT and Hate Speech Word Embedding with Deep Model
Saleh, H., Alhothali, A., & Moria, K. M. (2023) · Applied Artificial Intelligence · doi:10.1080/08839514.2023.2166719 · Cited by 172
“However, According to Fortuna and Nunes (2018), hate speech could use offensive language but not necessary, which we agree with because frequent use of hate words does not mean that it should be socially acceptable, and no need to detect them and for this reason, we combined hate and offensive…”
BackgroundPersistent interaction patterns across social media platforms and over time
Avalle, M., Marco, N. D., Etta, G., Sangiorgio, E., Alipour, S., Bonetti, A., Alvisi, L., Scala, A., Baronchelli, A., Cinelli, M., et al. (2024) · Nature · doi:10.1038/s41586-024-07229-y · Cited by 219
“The growing interest in analysing online toxicity through massive data analysis coincides with advancements in machine learning capable of detecting toxic language 27 .”
BackgroundArabic Offensive Language on Twitter: Analysis and ExperimentsPreprint
Mubarak, H., Rashed, A. A.-A., Darwish, K. M., Samih, Y., & Abdelalí, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02192 · Cited by 170
“Further; Schmidt and Wie-gand (2017) surveyed major works on hate speech detection; Fortuna and Nunes (2018) provided a comprehensive survey for techniques and works done in the area between 2004 and 2017.”
A deep neural network based multi-task learning approach to hate speech detection
Kapil, P., & Ekbal, A. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106458 · Cited by 184
Hate Speech Detection is Not as Easy as You May Think: A Closer Look at Model Validation
Arango, A., Pérez, J. E. P., & Poblete, B. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331262 · Cited by 165
“It is used in several hate-speech detection studies [1, 2, 6, 13, 18, 23, 32] and it is consistently mentioned as an important source in the surveys about the subject [11, 26, 27].”
BackgroundMethodInfluentialComparing pre-trained language models for Spanish hate speech detection
Plaza-del-Arco, F. M., Molina-González, M. D., López, L. A. U., & Valdivia, M. T. M. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114120 · Cited by 189
A Hierarchically-Labeled Portuguese Hate Speech Dataset
Fortuna, P., Rocha da Silva, J., Soler-Company, J., Wanner, L., & Nunes, S. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3510 · Cited by 161
“2.1 Hate Speech Concepts Fortuna and Nunes (2018) analyze and compare several aggression-related concepts.”
BackgroundInfluentialSelf-citationGongane, V. U., Munot, M. V., & Anuse, A. D. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00951-3 · Cited by 153
Fortuna, P., & Wanner, L. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102524 · Cited by 153
Self-citationAngry by design: toxic communication and technical architectures
Munn, L. (2020) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00550-7 · Cited by 184
Automatic Hate Speech Detection using Machine Learning: A Comparative Study
Abro, S., Shaikh, S., Hussain, Z., Ali, Z., Khan, S. U., & Mujtaba, G. (2020) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2020.0110861 · Cited by 148
A Deep Learning Approach for Automatic Hate Speech Detection in the Saudi Twittersphere
Alshalan, R., & Al-Khalifa, H. S. (2020) · Applied Sciences · doi:10.3390/app10238614 · Cited by 147
“Most of the existing studies in the literature have primarily modeled the problem as a supervised classification task, whether using classical machine learning approaches or deep neural network approaches [2,6].”
BackgroundA Just and Comprehensive Strategy for Using NLP to Address Online Abuse
Jurgens, D., Hemphill, L., & Chandrasekharan, E. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1357 · Cited by 178
“However, NLP has largely considered a far narrower scope of what constitutes abuse through its selection of which types of behavior to recognize (Waseem et al., 2017; Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018).”
BackgroundLi, L., Fan, L., Atreja, S., & Hemphill, L. (2024) · ACM Transactions on the Web · doi:10.1145/3643829 · Cited by 179
Kennedy, C. J., Bacon, G., Sahn, A., & von Vacano, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10277 · Cited by 150
“Prior work on hate speech has often focused on a single platform, commonly Twitter, but our goal was to study hate speech in a variety of settings and to ultimately develop a model to measure hate speech without reliance on platform-specific language patterns (Fortuna and Nunes, 2018).”
A Survey on Multimodal Disinformation DetectionPreprint
Alam, F., Cresci, S., Chakraborty, T., Silvestri, F., Dimitrov, D., Martino, G. D. S., Shaar, S., Firooz, H., & Nakov, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12541 · Cited by 185
“…include detecting harmful contents in network science [Ribeiro et al., 2018], natural language processing [Waseem et al., 2017; Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018] and computer vision [Yang et al., 2019a; Vijayaraghavan et al., 2019; Gomez et al., 2020; Dimitrov et al., 2021].”
BackgroundAl-Makhadmeh, Z., & Tolba, A. K. R. (2019) · Computing · doi:10.1007/s00607-019-00745-0 · Cited by 136
“1 3 for identifying textual hate speech in different languages [13].”
BackgroundFinding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts
Breitfeller, L., Ahn, E. P., Jurgens, D., & Tsvetkov, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1176 · Cited by 133
“Instead, much of the recent work has focused on explicitly toxic language (e.g., Waseem et al., 2017), with surveys of the area also overlooking this important and challenging task of recognizing this subtle toxicity (van Aken et al., 2018; Salmi-nen et al., 2018; Fortuna and Nunes, 2018).”
BackgroundRacism is a virus: anti-asian hate and counterspeech in social media during the COVID-19 crisis
He, B., Ziems, C., Soni, S., Ramakrishnan, N., Yang, D., & Kumar, S. (2021) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3487351.3488324 · Cited by 131
“These features have previously been very effective in identifying hate speech and cyberbullying [19].”
BackgroundHate begets Hate: A Temporal Study of Hate Speech
Mathew, B., Illendula, A., Saha, P., Sarkar, S., Goyal, P., & Mukherjee, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415163 · Cited by 130
“Hate speech is a complex phenomenon, intrinsically associated to relationships among groups, and also relying on linguistic nuances [29].”
SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification
Rosenthal, S., Atanasova, P., Karadzhov, G., Zampieri, M., & Nakov, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.80 · Cited by 130
Detecting Harmful Content on Online Platforms: What Platforms Need vs. Where Research Efforts Go
Arora, A., Nakov, P., Hardalov, M., Sarwar, S. M., Nayak, V., Dinkov, Y., Zlatkova, D., Dent, K., Bhatawdekar, A., Bouchard, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3603399 · Cited by 128
“Schmidt and Wiegand [86] and Fortuna and Nunes [28] surveyed automated hate speech detection methods but focused primarily on the features shown to be most effective in classification systems.”
BackgroundIn Conversation with Artificial Intelligence: Aligning language Models with Human Values
Kasirzadeh, A., & Gabriel, I. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00606-x · Cited by 127
“38For a recent survey of attempts to reduce ‘toxic speech’, see Fortuna and Nunes [34]. consideration.”
BackgroundConfronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 172
Detecting and visualizing hate speech in social media: A cyber Watchdog for surveillance
Modha, S., Majumder, P., Mandl, T., & Mandalia, C. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113725 · Cited by 138
Siegel, A. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.005 · Cited by 162
“Automated hate speech detection tends to rely on natural language processing or text mining strategies (Fortuna and Nunes 2018).”
BackgroundMethodInfluentialDetection of hate speech in Arabic tweets using deep learning
Alhassan, A., & Al-Dossari, H. Z. (2021) · Multimedia Systems · doi:10.1007/s00530-020-00742-w · Cited by 123
Deep learning for hate speech detection: a comparative study
Malik, J. S., Qiao, H., Pang, G., & van den Hengel, A. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00650-6 · Cited by 116
“Many methods have been introduced for hate speech detection [4, 20, 44, 47, 60].”
MethodResultDeepHate: Hate Speech Detection via Multi-Faceted Text Representations
Cao, R., Lee, R. K.-W., & Hoang, T.-A. (2020) · ACM Web Science Conference (WebSci) · doi:10.1145/3394231.3397890 · Cited by 115
“In recent years, researchers from data mining, information retrieval, and Natural Language Processing (NLP) fields have proposed several such methods [14, 35].”
BackgroundQuarantining online hate speech: technical and ethical perspectives
Ullmann, S., & Tomalin, M. (2019) · Ethics and Information Technology · doi:10.1007/s10676-019-09516-z · Cited by 113
“In recent years, the automatic detection of online Hate Speech (HS), and offensive language more generally, has become an active research topic in machine learning (David-son et al. 2017; Schmidt and Wiegand 2017a, b; Fortuna and Nunes 2018).”
BackgroundMethodDetecting and Understanding Harmful Memes: A Survey
Sharma, S. K., Alam, F., Akhtar, M. S., Dimitrov, D., Martino, G. D. S., Firooz, H., Halevy, A. Y., Silvestri, F., Nakov, P., & Chakraborty, T. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/781 · Cited by 113
Demoting Racial Bias in Hate Speech Detection
Xia, M., Field, A., & Tsvetkov, Y. (2020) · International Workshop on Natural Language Processing for Social Media (SocialNLP) · doi:10.18653/v1/2020.socialnlp-1.2 · Cited by 110
Karim, M. R., Chakravarthi, B. R., McCrae, J. P., & Cochez, M. (2020) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa49011.2020.00053 · Cited by 108
“, carefully designed to extract useful features from a simple input feature representation [25] automatically.”
BackgroundA Multi-Task Learning Approach to Hate Speech Detection Leveraging Sentiment Analysis
Plaza-del-Arco, F. M., Molina-González, M. D., López, L. A. U., & Valdivia, M. T. M. (2021) · IEEE Access · doi:10.1109/access.2021.3103697 · Cited by 129
“However, there is a huge demand for research into languages other than English [16].”
BackgroundDynamics of online hate and misinformation
Cinelli, M., Pelicon, A., Mozetič, I., Quattrociocchi, W., Novak, P. K., & Zollo, F. (2021) · Scientific Reports · doi:10.1038/s41598-021-01487-w · Cited by 107
An Expert Annotated Dataset for the Detection of Online Misogyny
Guest, E., Vidgen, B., Mittos, A., Sastry, N., Gareth Tyson, A., & Margetts, H. Z. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.114 · Cited by 181
“These issues are common in abusive speech detection (Fortuna and Nunes, 2018; Fortuna et al., 2020).”
BackgroundThe Structure of Toxic Conversations on Twitter
Saveski, M., Roy, B., & Roy, D. K. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449861 · Cited by 106
“These approaches typically analyze the content in isolation and after the fact [21].”
BackgroundHuman social sensing is an untapped resource for computational social science
Galešić, M., de Bruin, W. B., Dalege, J., Feld, S. L., Kreuter, F., Olsson, H., Prelec, D., Stein, D. L., & van der Does, T. (2021) · Nature · doi:10.1038/s41586-021-03649-2 · Cited by 120
Uyheng, J., & Carley, K. M. (2020) · Journal of Computational Social Science · doi:10.1007/s42001-020-00087-4 · Cited by 113
“natural language processing standpoint have focused on the task of automatically classifying hate speech [25, 35, 85].”
BackgroundOffensive, aggressive, and hate speech analysis: From data-centric to human-centered approach
Kocoń, J., Figas, A., Gruza, M., Puchalska, D., Kajdanowicz, T., & Kazienko, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102643 · Cited by 103
Transfer learning for hate speech detection in social media
Yuan, L., Wang, T., Ferraro, G., Suominen, H., & Rizoiu, M.-A. (2023) · Journal of Computational Social Science · doi:10.1007/s42001-023-00224-9 · Cited by 103
“Despite its recency as a research field [14], online abuse has received considerable attention; its subproblems range from quantifying bias and stereotypes in detection models [15], to building dedicated web applications for actively reporting hate speech [16], to organizing shared tasks for aggression identification…”
BackgroundGenerating Counter Narratives against Online Hate Speech: Data and Strategies
Tekiroğlu, S. S., Chung, Y.-L., & Guerini, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.110 · Cited by 103
Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media
Florio, K., Basile, V., Polignano, M., Basile, P., & Patti, V. (2020) · Applied Sciences · doi:10.3390/app10124180 · Cited by 103
Emotionally Informed Hate Speech Detection: A Multi-target Perspective
Chiril, P., Pamungkas, E. W., Benamara, F., Moriceau, V., & Patti, V. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09862-5 · Cited by 121
“not abusive/not hateful) [43, 64, 115] relying on several manually annotated datasets that can be grouped into one of these categories:”
BackgroundHandling Bias in Toxic Speech Detection: A Survey
Garg, T., Masud, S., Suresh, T., & Chakraborty, T. (2023) · ACM Computing Surveys · doi:10.1145/3580494 · Cited by 162
“Additionally, several surveys have already examined the existing literature in the area of modelling the task of toxicity detection [28, 76].”
BackgroundIntroducing CAD: the Contextual Abuse Dataset
Vidgen, B., Nguyen, D., Margetts, H. Z., Rossini, P., & Tromble, R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.182 · Cited by 166
“Wulczyn et al. (2017) report an Alpha of 0.45, Sanguinetti et al. (2018) Kappas from k=0.37 for offence to k=0.54 for hate, Gomez et al. (2020) report Kappa of 0.15 in the “MMH150” dataset of hateful memes, and Fortuna and Nunes (2018) report a Kappa of 0.17 for a text-only task.”
BackgroundResultInfluentialDisentangling Hate in Online Memes
Lee, R. K.-W., Cao, R., Fan, Z., Jiang, J., & Chong, W.-H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475625 · Cited by 155
“…may improve an individual’s sense of connectedness with real and virtual communities, these platforms are increasingly exploited for the propagation of hateful content that attacks or uses discriminatory languages targeting a person or a group based on their race, religion, gender, etc. [17, 43].”
BackgroundMethodHate and offensive speech detection on Arabic social media
Alsafari, S., Sadaoui, S., & Mouhoub, M. (2020) · Online Social Networks and Media · doi:10.1016/j.osnem.2020.100096 · Cited by 99
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97
SWSR: A Chinese dataset and lexicon for online sexism detection
Jiang, A., Yang, X., Liu, Y., & Zubiaga, A. (2021) · Online Social Networks and Media · doi:10.1016/j.osnem.2021.100182 · Cited by 151
A literature survey on multimodal and multilingual automatic hate speech identification
Chhabra, A., & Vishwakarma, D. K. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01051-8 · Cited by 126
“…a selected bunch [13] Hate Speech is a purposeful attack on a specific social occasion of people motivated by the pieces of the group's character [14] Hate Speech assails or prompts malignance against gatherings in light of explicit qualities like actual looks, religion, ethnicity, sexism, and…”
BackgroundMethodTackling Online Abuse: A Survey of Automated Abuse Detection MethodsPreprint
Mishra, P., Yannakoudakis, H., & Shutova, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06024 · Cited by 94
“We differ from previous surveys (Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018; Salminen et al., 2018; Castelle, 2018) in the following respects: 1) we discuss the cate-gorizations of abuse based on coarse-grained vs. fine-grained taxonomies; 2) we present a detailed overview of datasets…”
ResultToxic language in online incel communities
Pelzer, B., Kaati, L., Cohen, K., & Fernquist, J. (2021) · SN Social Sciences · doi:10.1007/s43545-021-00220-8 · Cited by 98
Hate Speech in Pixels: Detection of Offensive Memes towards Automatic ModerationPreprint
Sàbat, B. O., Canton-Ferrer, C., & Giró-i-Nieto, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02334 · Cited by 92
“Social networks have already introduced many tools to detect offensive or misleading content, both for visual and textual content, ranging from nudity and pornography [7, 9] to hate speech [6] and misinformation [2].”
BackgroundHate Speech Detection in Roman Urdu
Khan, M. M., Shahzad, K., & Malik, M. K. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3414524 · Cited by 91
“The study has concluded that several attempts have been made to generate hate speech datasets; however, these datasets are not publicly available, which has thwarted the development of the hate speech detection techniques [13].”
BackgroundInfluentialAssessing the Impact of Contextual Information in Hate Speech Detection
Pérez, J. M., Luque, F. M., Zayat, D., Kondratzky, M., Moro, A., Serrati, P. S., Zajac, J., Miguel, P., Debandi, N., Gravano, A., et al. (2023) · IEEE Access · doi:10.1109/access.2023.3258973 · Cited by 90
“Another limitation for hate speech detection is that most of resources are built in English, restricting the research and its applicability in other languages [14], [15].”
BackgroundBERT-based ensemble learning for multi-aspect hate speech detection
Mazari, A. C., Boudoukhani, N., & Djeffal, A. (2024) · Cluster Computing · doi:10.1007/s10586-022-03956-x · Cited by 89
“…this behaviour comes in repeated toxic acts against powerless victims like cyberbullying [2], aggressive incidents between users like cyber aggression [3], or inciting hatred against minorities based on race, religion, ethnicity, sex or other protected characteristics like hate speech [4, 5].”
Un-Compromised Credibility: Social Media Based Multi-Class Hate Speech Classification for Text
Qureshi, K. A., & Sabih, M. (2021) · IEEE Access · doi:10.1109/access.2021.3101977 · Cited by 88
“…studies that using Text Mining, Information Retrieval, or Natural Language Processing for hate speech identification is considered much effective [9], [14] as compared to Keyword-based, Rule-based/Association Rule Mining [19], Source metadata/Social Network Analysis based approaches [12], [23].”
BackgroundMethodInfluentialIndurthi, V., Syed, B., Shrivastava, M., Chakravartula, N., Gupta, M., & Varma, V. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2009 · Cited by 87
“Papers published in the last two years include the surveys by (Schmidt and Wiegand, 2017) and (Fortuna and Nunes, 2018), the paper by (David-son et al., 2017) which presented the Hate Speech Detection dataset used in (Malmasi and Zampieri, 2017) and a few other recent papers such as (ElSh-erief et…”
MethodAlshalan, R., Al-Khalifa, H. S., AlSaeed, D., Al-Baity, H. H., & Alshalan, S. (2020) · Journal of Medical Internet Research · doi:10.2196/22609 · Cited by 87
“Hate speech on the internet is a complex concept with a wide spectrum of targets, forms, and other related concepts [2,3].”
BackgroundMeske, C., & Bunde, E. (2022) · Information Systems Frontiers · doi:10.1007/s10796-021-10234-5 · Cited by 103
“…automated hate speech detection systems and the involvement of end users to evaluate the design is underrepresented, is also confirmed in surveys about automated hate speech detection, since the before-mentioned aspects are not adequately represented (e.g., Ayo et al., 2020; Fortuna & Nunes, 2018).”
BackgroundMethodInfluentialWhen the Timeline Meets the Pipeline: A Survey on Automated Cyberbullying Detection
Elsafoury, F., Katsigiannis, S., Pervez, Z., & Ramzan, N. (2021) · IEEE Access · doi:10.1109/access.2021.3098979 · Cited by 105
“In a survey paper on the automated detection of hate speech in text, Fortuna and Nunes studied the definition of hate speech in the literature in relation to four dimensions: physical violence encouragement, targets, attack language, and humorous hate speech [34].”
BackgroundMansur, Z., Omar, N., & Tiun, S. B. (2023) · IEEE Access · doi:10.1109/access.2023.3239375 · Cited by 101
“Most of these efforts are still facing some challenges in reaching a feasible solution as the language in social media is developing rapidly [9].”
BackgroundResultInfluentialDetecting Misogyny and Xenophobia in Spanish Tweets Using Language Technologies
Plaza-del-Arco, F. M., Molina-González, M. D., López, L. A. U., & Valdivia, M. T. M. (2020) · ACM Transactions on Internet Technology · doi:10.1145/3369869 · Cited by 96
“Themajority of previous studies concerning HS detection in online communication are in English due to the availability of annotated datasets, and a few recent studies have also been conducted on other languages [21].”
BackgroundArabic Offensive and Hate Speech Detection Using a Cross-Corpora Multi-Task Learning Model
Aldjanabi, W., Dahou, A., Al‐qaness, M. A. A., Elaziz, M. A., Helmi, A. M., & Damaševičius, R. (2021) · Informatics · doi:10.3390/informatics8040069 · Cited by 85
“There have been some works on tasks related to hate speech and offensive language detection (OFF) [36,37], including in the Arabic language [38].”
BackgroundOmar, A., Mahmoud, T. M., & El‐Hafeez, T. A. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-44289-7_24 · Cited by 83
Gaikwad, M., Ahirrao, S. N., Phansalkar, S., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3068313 · Cited by 83
“[41] in 2018 elaborated the need to study the automatic detection of hate speech.”
BackgroundAlonso, M. Á., Vilares, D., Gómez‐Rodríguez, C., & Vilares, J. (2013) · Electronics · doi:10.3390/electronics · Cited by 83
Multimodal Learning For Hateful Memes Detection
Zhou, Y., Chen, Z., & Yang, H. (2021) · IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew53276.2021.9455994 · Cited by 112
A Survey of Offensive Language Detection for the Arabic Language
Husain, F., & Uzuner, Ö. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3421504 · Cited by 81
“Fortuna and Nunes [2018] defined a list of rules to identify hate speech, which consists of: using stereotypes to refer to individuals based on the groups to which they belong, using negative statements about minority groups, using racial and disparaging terms to cause harm, using racial and sexist…”
Background
Metadata
- Authors
- Fortuna, P., & Nunes, S.
- Affiliations
- INESC TEC (PT)
- Universidade do Porto (PT)
- Venue
- ACM Computing Surveys
- Published
- 2018-07-31
- Volume
- 51
- Issue
- 4
- Pages
- 1-30
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2887782043
- Cited by
- 1,241
- References
- 52
- Citations by year
- 2026: 139 · 2025: 195 · 2024: 141 · 2023: 185 · 2022: 158 · 2021: 199 · 2020: 144 · 2019: 71 · 2018: 8
- FWCI
- 59.53
- Type
- journal article
- Language
- en
- Open access
- Locations
- ACM Computing Surveys (publishedVersion)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, other-oa)
- Funders
- FourEyes, a research line within project TEC4Growth—Pervasive Intelligence
- North Portugal Regional Operational Programme
- European Regional Development Fund
- Enhancers and Proofs of Concept with Industrial (Impact/NORTE-01- 0145-FEDER-000020)
- PORTUGAL 2020 Partnership Agreement
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- hate speech detection, text classification, definition of hate speech, annotated dataset, multilingual dataset, online communities, digital media platforms, social impact
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.