A Survey on Automatic Detection of Hate Speech in Text
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
The scientific study of hate speech, from a computer science point of view, is recent. This survey organizes and describes the current state of the field, providing a structured overview of previous approaches, including core algorithms, methods, and main features used. This work also discusses the complexity of the concept of hate speech, defined in many platforms and contexts, and provides a unifying definition. This area has an unquestionable potential for societal impact, particularly in online communities and digital media platforms. The development and systematization of shared resources, such as guidelines, annotated datasets in multiple languages, and algorithms, is a crucial step in advancing the automatic detection of hate speech.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recent studies(Jurgens etal. 2019; Fortuna and Nunes 2018) that have looked into tackling and proposing subtle hate detection suggest the consideration of making all subtle forms of discrimination, even jokes, as hate speech since they negatively affect some people psychological even though they are considered harmless(Douglass etal. 2016).”
From Misogynoir: challenges in detecting intersectional hate (2022) · cites this paper as Fortuna and Nunes 2018
Citations
Cites
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655 · cited 1×
“Nevertheless, detecting hate speech is a very important task, because it is connected with actual hate crimes [42, 65, 77], and automatic hate speech detection in text can also provide data about this phenomenon.”
BackgroundA Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media · doi:10.18653/v1/w17-1101 · Cited by 1,395 · cited 4×
“A problem that is referred in hate speech detection is that sentences must be classified and not words [66].”
BackgroundInfluentialAbusive Language Detection in Online User Content
Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., & Chang, Y. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883062 · Cited by 1,122 · cited 9×
“[10], abusive language [58], discrimination [69], profanity [23], toxicity [43], flaming [35], extremism [55, 62], and radicalization [2].”
BackgroundMethodInfluentialDetecting Hate Speech on the World Wide Web
Warner, W. B., & Hirschberg, J. (2012) · Cited by 964
Deep Learning for Hate Speech Detection in Tweets
Badjatiya, P., Gupta, S., Gupta, M., & Varma, V. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3054223 · Cited by 849 · cited 2×
“N-grams are one of the most used techniques in hate speech automatic detection and related tasks [4, 9, 19, 33, 52, 58, 77].”
MethodStanford typed dependencies manual
de Marnee, M.-C., & Manning, C. D. (2010) · Cited by 717
Hate Speech Detection with Comment Embeddings
Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., & Bhamidipati, N. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2740908.2742760 · Cited by 712 · cited 2×
“Some authors [25] use a paragraph2vec approach to classify language on user comments as abusive or clean and also to predict the central word in the message [25].”
Background
Research works printed but not linked: 18
- ACL., ALW1: 1st workshop on abusive language online.
- Burnap Peter, Proceedings of the Conference on the Internet, Policy 8 Politics. 1--18
- CHI2017. 2017. 2017 Workshop on online harassment. Retrieved from http://social.umd.edu/woh/. CHI2017. 2017. 2017 Workshop on online harassment. Retrieved from http://social.umd.edu/woh/.
- CONTACT., 2017, Interdisciplinary conference on Hate speech. Definitions, Interpretations and Practices.
- Davidson Thomas, 2017, Automated hate speech detection and the problem of offensive language. arXiv Preprint arXiv:1703.04009 · arXiv
- De Marneffe Marie-Catherine, 2008, Manning
- Pauw Guy De, 2016, First workshop on text analytics for cybersecurity and online safety (TA-COS
- Vigna Fabio Del, 2017, Proceedings of the 1st Italian Conference on Cybersecurity. 86--95
- Cambridge Dictionary. 2017. Profanity. Retrieved from https://dictionary.cambridge.org/dictionary/english/profanity. Cambridge Dictionary. 2017. Profanity. Retrieved from https://dictionary.cambridge.org/dictionary/english/profanity.
- Giblietto Fabio, 2015, Proceedings of the 4th Workshop on Making Sense of Microposts (#Microposts’14)
- Reporting Hate. 2017. Hate speech conference I.H.D.I.P. Retrieved from http://reportinghate.eu/contact2017/. Reporting Hate. 2017. Hate speech conference I.H.D.I.P. Retrieved from http://reportinghate.eu/contact2017/.
- Kwok Irene, 2013, Proceedings of the Association for the Advancement of Artificial Intelligence.
- Ross Bjorn, 2017, Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv Preprint arXiv:1701.08118 · arXiv
- Silva Leandro, 2016, Analyzing the targets of hate in online social media. arXiv Preprint arXiv:1603.07709 · arXiv
- Natalya Tarasova. 2016. Classification of Hate Tweets and Their Reasons using SVM. Master’s thesis. Uppsala Universitet. Natalya Tarasova. 2016. Classification of Hate Tweets and Their Reasons using SVM. Master’s thesis. Uppsala Universitet.
- Tulkens Stéphan, 2016, A dictionary-based approach to racism detection in dutch social media. arXiv Preprint arXiv:1608.08738 · arXiv
- Warner William, 2012, Proceedings of the Second Workshop on Language in Social Media. Association for Computational Linguistics, 19--26
- Yuan Shuhan, 2016, Proceedings of the International Conference on Extending Database Technology. 696--697
Cited by
Items 201–300 of 1,004
Vietnamese hate and offensive detection using PhoBERT-CNN and social media streaming data
Tran, K. Q., Nguyen, A. T., Hoang, P. G., Luu, C. D., Do, T.-H., & Nguyen, K. V. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07745-w · Cited by 44
“They not only provide a structured overview of previous approaches [14] but also describe key sub-areas that have been explored to recognize these types of utterances automatically [8].”
BackgroundA Deep Dive into Multilingual Hate Speech Classification
Aluru, S. S., Mathew, B., Saha, P., & Mukherjee, A. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-67670-4_26 · Cited by 37
“[13] conducted a comprehensive survey on this subject.”
BackgroundWhy do volunteer content moderators quit? Burnout, conflict, and harmful behaviors
Schöpke-Gonzalez, A. M., Atreja, S., Shin, H. N., Ahmed, N., & Hemphill, L. (2024) · New Media & Society · doi:10.1177/14614448221138529 · Cited by 37
“…of computational research has focused on making the task of detecting and assessing toxic content easier or less distressing for moderation workers (Fortuna and Nunes, 2018; Schmidt and Wiegand, 2017), our study shows that researchers should consider building a different set of interventions for…”
Dewani, A., Memon, M. A., Bhatti, S., Sulaiman, A. A., Hamdi, M., Alshahrani, H. M., Alghamdi, A., & Shaikh, A. (2023) · Applied Sciences · doi:10.3390/app13042062 · Cited by 44
An Ensemble Method for Radicalization and Hate Speech Detection Online Empowered by Sentic Computing
Araque, Ó., & Iglesias, C. Á. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09845-6 · Cited by 43
AOMD: An analogy-aware approach to offensive meme detection on social media
Shang, L., Zhang, Y., Zha, Y., Chen, Y., Youn, C., & Wang, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102664 · Cited by 48
Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Framework
Halevy, M., Harris, C., Bruckman, A., Yang, D., & Howard, A. (2021) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3465416.3483299 · Cited by 53
“In recent years, interest in hate-speech detection has grown as a sub-topic of natu-ral language processing research and more datasets have become available for researchers [11, 12, 15–17, 21, 33].”
InfluentialZhong, J., Qiu, J., Sun, M.-P., Jin, X., Zhang, J., Guo, Y.-D., Qiu, X.-X., Xu, Y., Huang, J., & Zheng, Y.-X. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.861823 · Cited by 23
“, 2016) and may contain directly or indirectly offensive words (Fortuna and Nunes, 2018).”
BackgroundBunde, E. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.154 · Cited by 48
“Today, a large part of human communication takes place in the digital sphere, for instance via social media [1-2], and so does hate speech, which can be harmful for individuals and society as a whole [3].”
BackgroundInfluentialDetecting abusive comments at a fine-grained level in a low-resource language
Chakravarthi, B. R., Priyadharshini, R., Banerjee, S., Jagadeeshan, M. B., Kumaresan, P. K., Ponnusamy, R., Benhur, S., & McCrae, J. P. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100006 · Cited by 50
OLID-BR: offensive language identification dataset for Brazilian Portuguese
Trajano, D., Bordini, R. H., & Vieira, R. L. R. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09657-0 · Cited by 44
“Fortuna and Nunes (2018) analyzed several hate speech definitions from different sources and propose that hate speech is the language that attacks or diminishes, that incites violence or hate against groups, based on specific characteristics such as physical appearance, religion, descent, national…”
ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text
Hasanain, M., Alam, F., Mubarak, H., Abdaljalil, S., Zaghouani, W., Nakov, P., Martino, G. D. S., & Freihat, A. A. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.44 · Cited by 36
Ridha, M., Nurjanah, D., & Rakha, M. (2024) · International Conference on Intelligent Cybernetics Technology & Applications (ICICyTA) · doi:10.1109/icicyta64807.2024.10912874 · Cited by 38
“The approach with the subjectivity of human annotation also has shortcomings because it can have different interpretations for each person [14].”
Democratic Creative Destruction? The Effect of a Changing Media Landscape on Democracy
Nielsen, R. K., & Fletcher, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.008 · Cited by 62
“The Boundaries are Blurry…”: How Comment Moderators in Germany See and Respond to Hate Comments
Paasch‐Colberg, S., & Strippel, C. (2021) · Journalism Studies · doi:10.1080/1461670x.2021.2017793 · Cited by 35
“…moderation, but also has practical significance against the background of increasing efforts by both scholars and the industry to develop approaches for automated detection of “hate speech” (Fortuna and Nunes 2018), “incivility” (Su et al. 2018), or “toxicity” (Jigsaw 2021) in user comments.”
BackgroundDetection of homophobia and transphobia in YouTube comments
Chakravarthi, B. R. (2023) · International Journal of Data Science and Analytics · doi:10.1007/s41060-023-00400-0 · Cited by 58
“…from other types of speech on social media by the fact that it is directed toward a particular group of people; from this point of view, it is also distinct from offensive speech, which consists solely of the use of language that is considered to be vulgar or otherwise inappropriate [22–24].”
Leveraging Hate Speech Detection to Investigate Immigration-related Phenomena in Italy
Florio, K., Basile, V., Lai, M., & Patti, V. (2019) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw.2019.8925079 · Cited by 34
“The field has been recently surveyed in [4].”
BackgroundYasaswini, K., Puranik, K., Hande, A., Priyadharshini, R., Thavareesan, S., & Chakravarthi, B. R. (2021) · Cited by 73
Investigating Annotator Bias with a Graph-Based Approach
Wich, M., Al Kuwatly, H., & Groh, G. (2020) · Proceedings of the Fourth Workshop on Online Abuse and Harms · doi:10.18653/v1/2020.alw-1.22 · Cited by 31
“The advances in machine learning and deep learning have improved these systems tremen-dously, but there is still much space for enhancements because it is a challenging and complex task (Fortuna and Nunes, 2018; Schmidt and Wiegand, 2017).”
BackgroundUnderstanding social engagements: A comparative analysis of user and text features in Twitter
Toraman, Ç., Şahinuç, F., Yılmaz, E. H., & Akkaya, İ. B. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00872-1 · Cited by 34
“, preventing hate speech and misinformation (Fortuna and Nunes 2018), promoting public health awareness (Vraga et al.”
BackgroundExploring Automatic Hate Speech Detection on Social Media: A Focus on Content-Based Analysis
Nascimento, F. R. S., Cavalcanti, G. D. C., & Costa‐Abreu, M. D. (2023) · SAGE Open · doi:10.1177/21582440231181311 · Cited by 40
Accountability Issues, Online Covert Hate Speech, and the Efficacy of Counter‐Speech
Baider, F. (2023) · Politics and Governance · doi:10.17645/pag.v11i2.6465 · Cited by 33
Hossain, E., Sharif, O., Hoque, M. M., Dewan, M. A. A., Siddique, N. H., & Hossain, M. A. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.06.010 · Cited by 41
“…troll (Mojica, 2018; Mut Altin et al., 2020), aggression (Safi Samghabadi et al., 2020; Aroyehun et al., 2018), hate speech (Basile et al., 2019; Fortuna and Nunes, 2018) and abusive (Pamungkas et al., 2019; Vidgen et al., 2019) contents from a single modality (i.e. image, text), it is often…”
Multilingual Hate Speech Detection: A Semi-Supervised Generative Adversarial Approach
Mnassri, K., Farahbakhsh, R., & Crespi, N. (2024) · Entropy · doi:10.3390/e26040344 · Cited by 46
Hate Speech Detection in Twitter using Natural Language Processing
Pariyani, B., Shah, K., Shah, M., Vyas, T., & Degadwala, S. (2021) · International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV) · doi:10.1109/icicv50876.2021.9388496 · Cited by 33
“According to Paula Fortuna and Sergia Nunes “Hate speech is language that attacks or diminishes, that incites violence or hate against groups, based on specific characteristics such as physical appearance, religion, descent, national or ethnic origin, sexual orientation, gender identity or other, and it can occur with…”
BackgroundInfluentialA Survey on Online Aggression: Content Detection and Behavioral Analysis on Social Media
Mane, S., Kundu, S., & Sharma, R. R. (2025) · ACM Computing Surveys · doi:10.1145/3711125 · Cited by 34
“…causing harm [43], content encompassing aggression, credible threats, and misinformation [47], text glorifying violent actions and fostering hostility [51], communication undermining based on attributes [109], and the intent to cause harm through verbal, physical, or psychological means [19, 32].”
Four Types of Toxic People: Characterizing Online Users’ Toxicity over Time
Mall, R., Nagpal, M., Salminen, J. O., Almerekhi, H., Jung, S., & Jansen, B. J. J. (2020) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3419249.3420142 · Cited by 40
“General literature reviews on online toxicity are given in [6, 37].”
BackgroundBenchmarking Crisis in Social Media Analytics: A Solution for the Data-Sharing Problem
Assenmacher, D., Weber, D., Preuß, M., Valdez, A. S. C., Bradshaw, A., Roß, B., Cresci, S., Trautmann, H., Neumann, F., & Grimme, C. (2021) · Social Science Computer Review · doi:10.1177/08944393211012268 · Cited by 35
“…for sophisticated hate speech (abusive language) detection mechanisms has been observed, especially from news media sites confronted with the challenge of moderating toxic and hateful content in their websites’ comment sections (Fortuna & Nunes, 2018; Niemann et al., 2020; Riehle et al., 2020).”
BackgroundControversy and Conformity: from Generalized to Personalized Aggressiveness Detection
Kanclerz, K., Figas, A., Gruza, M., Kajdanowicz, T. J., Kocoń, J., Puchalska, D., & Kazienko, P. (2021) · doi:10.18653/v1/2021.acl-long.460 · Cited by 39
“This inconsistency is visible in many reviews related to automatic detection of hate speech (Fortuna and Nunes, 2018; Schmidt and Wiegand, 2017; Alrehili, 2019; Poletto et al., 2020) or more specifically on aggressiveness detection (Sadiq et al., 2021; Modha et al., 2020).”
BackgroundNecessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection
Balkır, E., Nejadgholi, I., Fraser, K., & Kiritchenko, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.192 · Cited by 43
HateCircle and Unsupervised Hate Speech Detection Incorporating Emotion and Contextual Semantics
Ghosal, S., & Jain, A. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3576913 · Cited by 32
“There are several challenges in existing cyberhate detection research [ 5 ].”
BackgroundPerera, S., Meedin, G. S. N., Caldera, M., Perera, I., & Ahangama, S. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e19097 · Cited by 32
Hate speech detection in the Bengali language: a comprehensive survey
Maruf, A. A., Abidin, A. J., Haque, M. M., Jiyad, Z. M., Golder, A., Alubady, R., & Aung, Z. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00956-z · Cited by 40
MIDAS at SemEval-2019 Task 6: Identifying Offensive Posts and Targeted Offense from Twitter
Mahata, D., Zhang, H., Uppal, K., Kumar, Y., Shah, R. R., Shahid, S., Mehnaz, L., & Anand, S. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2122 · Cited by 31
“Hate speech is an act of offending, insulting or threatening a person or a group of similar people on the basis of religion, race, caste, sexual orientation, gender or belongingness to a specific stereotyped community (Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018).”
A New Measure of Polarization in the Annotation of Hate Speech
Akhtar, S., Basile, V., & Patti, V. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-35166-3_41 · Cited by 31
Handling Imbalance Issue in Hate Speech Classification using Sampling-based Methods
Rathpisey, H., & Adji, T. B. (2019) · International Conference on Science in Information Technology · doi:10.1109/icsitech46713.2019.8987500 · Cited by 31
“Many datasets from those social networks were created by the researchers [3][4].”
BackgroundDetecting hate speech against politicians in Arabic community on social media
Guellil, I., Adeel, A., Azouaou, F., Chennoufi, S., Maafi, H., & Hamitouche, T. (2020) · International Journal of Web Information Systems · doi:10.1108/ijwis-08-2019-0036 · Cited by 31
Contextual-Lexicon Approach for Abusive Language Detection
Vargas, F., de Góes, F. R., Carvalho, I., Benevenuto, F., & Pardo, T. A. S. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_161 · Cited by 50
The design, construction and evaluation of annotated Arabic cyberbullying corpus
Shannag, F., HAMMO, B. H., & Faris, H. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11056-x · Cited by 36
“However, to better understand this complex phenomenon, we discuss the meaning of CB in the presence of other related concepts, such as hate speech (HS) (Fortuna & Nunes, 2018) offensive language (Chen et al., 2012; Fortuna & Nunes, 2018), and profanity.”
BackgroundZia, H. B., Castro, I., Zubiaga, A., & Tyson, G. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19402 · Cited by 42
“Still, much of this research focuses on a single language, generally English (Fortuna and Nunes 2018; Poletto et al. 2020; Vidgen and Derczynski 2020), and lesser-resourced languages have been rarely studied.”
BackgroundAccelerating automatic hate speech detection using parallelized ensemble learning models
Agarwal, S., Sonawane, A., & Chowdary, C. R. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120564 · Cited by 31
Countering Online Hate Speech: An NLP PerspectivePreprint
Chaudhary, M., Saxena, C., & Meng, H. M. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02941 · Cited by 34
“(Fortuna and Nunes, 2018) projected the definitions of hate speech from different sources into four dimensions – (i) hate speech is to incite violence or hate, (ii) hate speech is to attack or diminish, (iii) hate speech has spe-cific targets and (iv) humor has a specific status.”
BackgroundMisogynoir: challenges in detecting intersectional hate
Kwarteng, J., Perfumi, S. C., Farrell, T., Third, A., & Fernandez, M. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00993-7 · Cited by 30
“2019; Fortuna and Nunes 2018) that have looked into tackling and proposing subtle hate detection suggest the consideration of making all subtle forms of discrimination, even jokes, as hate speech since they negatively affect some people psychological even though they are considered…”
A Comprehensive Survey of Fake Text Detection on Misinformation and LM-Generated Texts
Kwon, S.-C., & Jang, B. (2025) · IEEE Access · doi:10.1109/access.2025.3538805 · Cited by 37
Li, S., Li, G., Law, R., & Paradies, Y. (2020) · Tourism Management · doi:10.1016/j.tourman.2020.104100 · Cited by 30
On Analyzing COVID-19-related Hate Speech Using BERT Attention
Vishwamitra, N., Hu, R., Luo, F., Cheng, L., Costello, M., & Yang, Y. (2020) · International Conference on Machine Learning and Applications · doi:10.1109/icmla51294.2020.00111 · Cited by 35
“Existing studies of hate speech from the social science literature [14], [25] have shown that hate speech is directed at an individual or group based on “an arbitrary or normatively irrelevant feature”, and that it casts the target as an “undesirable presence and a legitimate object of hostility.”
BackgroundHidden behind the obvious: Misleading keywords and implicitly abusive language on social media
Yin, W., & Zubiaga, A. (2022) · Online Social Networks and Media · doi:10.1016/j.osnem.2022.100210 · Cited by 39
Label modification and bootstrapping for zero-shot cross-lingual hate speech detection
Bigoulaeva, I., Hangya, V., Gurevych, I., & Fraser, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09637-4 · Cited by 38
“Fortuna and Nunes (2018) offer a comparison of nine such terms, such as “cyberbullying”, “discrimination”, “flaming”, “toxic language”, and “abusive language”, with explanations of how these concepts differ from the concept of hate speech itself.”
BackgroundCohen, S., Presil, D., Katz, O., Arbili, O., Messica, S., & Rokach, L. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101887 · Cited by 35
Kennedy, B., Golazizian, P., Trager, J. P., Atari, M., Hoover, J., Davani, A. M., & Dehghani, M. (2023) · PNAS Nexus · doi:10.1093/pnasnexus/pgad210 · Cited by 48
“While a wave of new research, both in academia and in industry, has focused on automatic detection (and removal) of hate speech (15), this task is very much unresolved.”
Arora, G. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02094 · Cited by 30
“Authors in [5] discuss the complexity of the concept of hate speech and its unquestionable potential for societal impact, particularly in online communities and digital media platforms.”
BackgroundFindings of the WOAH 5 Shared Task on Fine Grained Hateful Memes Detection
Mathias, L., Nie, S., Davani, A. M., Kiela, D., Prabhakaran, V., Vidgen, B., & Talat, Z. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.21 · Cited by 44
“This indicates ‘moderate’ to ‘strong’ agreement (Mchugh, 2012) and compares favourably with other abusive content datasets (Gomez et al., 2020; Fortuna and Nunes, 2018; Wulczyn et al., 2017), especially given that our labels contain five and seven levels respectively.”
ResultCyberbullying and Cyberviolence Detection: A Triangular User-Activity-Content View
Wang, S., Zhu, X., Ding, W., & Yengejeh, A. A. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105740 · Cited by 30
Plaza-del-Arco, F. M., Molina-González, M. D., López, L. A. U., & Valdivia, M. T. M. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109965 · Cited by 29
Detecting racial stereotypes: An Italian social media corpus where psychology meets NLP
Bosco, C., Patti, V., Frenda, S., Cignarella, A. T., Paciello, M., & D’Errico, F. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103118 · Cited by 29
Detection of Offensive Language and ITS Severity for Low Resource Language
Saeed, R., Afzal, H., Rauf, S. A., & Iltaf, N. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3580476 · Cited by 29
“Since the majority of the definitions of hate speech agree on inciting violence and attacking or diminishing individuals or groups based on protected characteristics, we adopt the definition of [21] as the base definition.”
BackgroundMethodInfluentialToKen: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection
AlKhamissi, B., Ladhak, F., Iyer, S. V., Stoyanov, V., Kozareva, Z., Li, X., Fung, P., Mathias, L., Çelikyılmaz, A., & Diab, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.136 · Cited by 41
Hate Speech and Offensive Language Detection Using an Emotion-Aware Shared Encoder
Mnassri, K., Rajapaksha, P., Farahbakhsh, R., & Crespi, N. (2023) · International Conference on Communications · doi:10.1109/icc45041.2023.10279690 · Cited by 39
“The majority of these attempts adopt supervised learning and deep learning methods that were trained using an annotated dataset [1].”
BackgroundAn archival perspective on pretraining data
Desai, M. A., Pasquetto, I. V., Jacobs, A. Z., & Card, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100966 · Cited by 35
“2,69 Beyond LLMs, toxic language detection has become a canonical task in natural language processing (NLP) (e.g., Waseem and Hovy, 70 Fortuna and Nunes, 71 Fortuna et al., 72 Liang et al. 2 ).”
BackgroundHate Speech Detection Research in South Asian Languages: A Survey of Tasks, Datasets and Methods
Sharma, D., Nath, T., Gupta, V., & Singh, V. K. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3711710 · Cited by 28
Hate speech in online social media
Das, M., Mathew, B., Saha, P., Goyal, P., & Mukherjee, A. (2020) · ACM SIGWEB Newsletter · doi:10.1145/3427478.3427482 · Cited by 28
Deep Learning for predicting neutralities in Offensive Language Identification Dataset
Sharma, M., Kandasamy, I., & Kandasamy, W. B. V. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115458 · Cited by 28
Comparative Media Regulation in the United States and Europe
Fukuyama, F., & Grotto, A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.010 · Cited by 42
Counterspeakers’ Perspectives: Unveiling Barriers and AI Needs in the Fight against Online Hate
Mun, J., Buerger, C., Liang, J. T., Garland, J., & Sap, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642025 · Cited by 45
“One aspect of hate speech most commonly shared by various definitions is that it has a specific target, a group of people distinguished by different identities such as religion, race, gender, and sexual identity [41].”
BackgroundThe language of bullying: Social issues on Chinese websites
Li, W. (2020) · Aggression and Violent Behavior · doi:10.1016/j.avb.2020.101453 · Cited by 32
COVID-HateBERT: a Pre-trained Language Model for COVID-19 related Hate Speech Detection
Li, M., Liao, S., Okpala, E., Tong, M., Costello, M., Cheng, L., Hu, H., & Luo, F. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00043 · Cited by 27
Tekiroğlu, S. S., Bonaldi, H., Fanton, M., & Guerini, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.245 · Cited by 62
Identification of offensive language in Urdu using semantic and embedding models
Hussain, S., Malik, M. S. I., & Masood, N. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1169 · Cited by 29
Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering
Bonaldi, H., Dellantonio, S., Tekiroğlu, S. S., & Guerini, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.549 · Cited by 74
Wu, F., Gao, B., Pan, X., Li, L., Ma, Y., Liu, S., & Liu, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103772 · Cited by 29
Hate Speech Detection of Arabic Shorttext
Aref, A., Mahmoud, R. H. A., Taha, K., & Al-Sharif, M. (2020) · doi:10.5121/csit.2020.100507 · Cited by 29
“Other researchers focused on algorithms for hate speech detection and used text mining and machine learning for hate speech classification [1] such as [3] and [7].”
BackgroundInfluentialExploring Stylometric and Emotion-Based Features for Multilingual Cross-Domain Hate Speech Detection
Markov, I., Ljubešić, N., Fišer, D., & Daelemans, W. M. P. (2021) · VU Research Portal · Cited by 26
Would Your Tweet Invoke Hate on the Fly? Forecasting Hate Intensity of Reply Threads on Twitter
Dahiya, S., Sharma, S., Sahnan, D., Goel, V., Chouzenoux, É., Elvira, V., Majumdar, A., Bandhakavi, A., & Chakraborty, T. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467150 · Cited by 33
“Recently, there has been a growing body of research in detection and characterizing online hate speech and modeling its spread [11].”
BackgroundUtilizing subjectivity level to mitigate identity term bias in toxic comments classification
Zhao, Z., Zhang, Z., & Hopfgartner, F. (2022) · Online Social Networks and Media · doi:10.1016/j.osnem.2022.100205 · Cited by 29
Improving Generalizability in Implicitly Abusive Language Detection with Concept Activation Vectors
Nejadgholi, I., Fraser, K., & Kiritchenko, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.378 · Cited by 35
Carvalho, P. C. Q. D. F., Caled, D., Silva, C., Batista, F., & Ribeiro, R. (2023) · Journal of Language Aggression and Conflict · doi:10.1075/jlac.00085.car · Cited by 32
Contextures of hate: Towards a systems theory of hate communication on social media platforms
Barth, N., Wagner, E., Raab, P., & Wiegärtner, B. (2023) · The Communication Review · doi:10.1080/10714421.2023.2208513 · Cited by 36
“…of computational social sciences is devoted to the question of detecting and combating hate speech by means of self-learning and automated programs (Fortuna & Nunes, 2019; Jigsaw, 2021; Jurgens, Hemphill, & Chandrasekharan, 2019; Schmidt & Wiegand, 2017; Van Aken, Risch, Krestel, & Löser, 2018;…”
BackgroundDeep Learning Algorithms for Cyber-Bulling Detection in Social Media Platforms
Obaida, M. H., Elkaffas, S. M., & Guirguis, S. K. (2024) · IEEE Access · doi:10.1109/access.2024.3406595 · Cited by 26
“Various types of abusive language can be observed on social networking platforms, including sexism, racism, cyberbullying, hate speech, and toxic remarks [1].”
BackgroundLuu, S. T., Nguyen, H., Nguyen, K. V., & Nguyen, N. L.-T. (2020) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf48685.2020.9140745 · Cited by 26
“Fortuna and Nunes (2018) has studied about the motivation of reseaching on Hatespeech dection problem and gave the definition of hate speech based on several sources such as European Union Commission, International minorities associations (ILGA), scientific papers and social networks.”
BackgroundOnline Political Advertising in the United States
Fowler, E. F., Franz, M. M., & Ridout, T. N. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.007 · Cited by 40
Hate Speech Detection In Indonesian Language Instagram
Putra, I. G. M., & Nurjanah, D. (2020) · International Conference on Advanced Computer Science and Information Systems · doi:10.1109/icacsis51025.2020.9263084 · Cited by 25
Hate Speech and Offensive Language Detection from Social Media
Mercan, V., Jamıl, A., Hameed, A. A., Magsi, I. A., Bazai, S., & Shah, S. A. (2021) · International Conference on Computing, Electronic and Electrical Engineering (ICE Cube) · doi:10.1109/icecube53880.2021.9628255 · Cited by 25
Detecting Hate Speech on Twitter Network using Ensemble Machine Learning
Mutanga, R. T., Naicker, N., & Olugbara, O. O. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.0130341 · Cited by 34
“Earlier studies have investigated the use of DT methods in hate speech detection tasks and recorded satisfactory performance [32].”
MethodDetecting Offensive Language Based on Graph Attention Networks and Fusion Features
Miao, Z., Chen, X., Wang, H., Tang, R., Yang, Z., Huang, T., & Tang, W. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3250502 · Cited by 25
“Among them, the number of offensive tweets is 1009, accounting for 7.90%. c) Annotation: The offensiveness of tweets includes racism, sexism, geography, and so on [2].”
BackgroundInfluentialPredicting hate intensity of twitter conversation threads
Meng, Q., Suresh, T., Lee, R. K.-W., & Chakraborty, T. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110644 · Cited by 34
Yin, W., Agarwal, V., Jiang, A., Zubiaga, A., & Sastry, N. R. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22198 · Cited by 37
Systematic keyword and bias analyses in hate speech detection
Sarracén, G. L. D. L. P., & Rosso, P. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103433 · Cited by 34
Detecting Unintended Social Bias in Toxic Language Datasets
Sahoo, N., Gupta, H., & Bhattacharyya, P. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.10 · Cited by 39
Yuan, L., & Rizoiu, M.-A. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101690 · Cited by 25
“Fortuna and Nunes (2018) surveyed definitions of hate speech and produced their own: “Hate speech is language that attacks or diminishes, that incites violence or hate against groups, based on specific characteristics such as physical appearance, religion, descent, national or ethnic origin, sexual…”
Say ‘YES’ to Positivity: Detecting Toxic Language in Workplace Communications
Bhat, M. M., Hosseini, S., Awadallah, A. H., Bennett, P. N., & Li, W. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.173 · Cited by 39
“While the research community has made enormous progress in detecting overly offensive language and hate speech (Schmidt and Wiegand, 2017; Waseem et al., 2018; Fortuna and Nunes, 2018; Qian et al., 2019), there has been less focus on computationally evaluating other subtle expressions of toxicity.”
BackgroundSavcı, P., & Daş, B. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e15670 · Cited by 24
“The solution to this problem can only be solved by classifying documents [2].”
BackgroundGhosh, S., Suri, M., Chiniya, P., Tyagi, U., Kumar, S., & Manocha, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.377 · Cited by 38
Conclusion: The Challenges and Opportunities for Social Media Research
Persily, N., & Tucker, J. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.014 · Cited by 24
Sharif, O., & Hoque, M. M. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-73696-5_2 · Cited by 24
HateMonitors: Language Agnostic Abuse Detection in Social MediaPreprint
Saha, P., Mathew, B., Goyal, P., & Mukherjee, A. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.20925.41445 · Cited by 24
The Economics of Content Moderation: Theory and Experimental Evidence from Hate Speech on TwitterPreprint
Jiménez-Durán, R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4590147 · Cited by 60
Opposition as “A Mould on the Fatherland”: Hate Speech and Grassroots Telegram Propaganda in Belarus
Sulzhytski, I. (2022) · The Journal of Belarusian Studies · doi:10.30965/20526512-12350018 · Cited by 23
Alhazmi, A. H., Mahmud, R. B., Idris, N. B., Abo, M. E., & Eke, C. I. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305657 · Cited by 23
Zufall, F., Hamacher, M., Kloppenborg, K., & Zesch, T. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.5 · Cited by 34
“Datasets encode all kinds of (partly incompatible) notions of hatefulness or offensiveness (Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018; Poletto et al., 2020; Fortuna et al., 2021) that make it difficult to decide which postings would justify restricting freedom of speech through deletion.”
BackgroundChakravarthi, B. R., Hande, A., Ponnusamy, R., Kumaresan, P. K., & Priyadharshini, R. (2022) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2022.100119 · Cited by 23
Metadata
- Authors
- Fortuna, P., & Nunes, S.
- Affiliations
- INESC TEC (PT)
- Universidade do Porto (PT)
- Venue
- ACM Computing Surveys
- Published
- 2018-07-31
- Volume
- 51
- Issue
- 4
- Pages
- 1-30
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2887782043
- Cited by
- 1,241
- References
- 52
- Citations by year
- 2026: 139 · 2025: 195 · 2024: 141 · 2023: 185 · 2022: 158 · 2021: 199 · 2020: 144 · 2019: 71 · 2018: 8
- FWCI
- 59.53
- Type
- journal article
- Language
- en
- Open access
- Locations
- ACM Computing Surveys (publishedVersion)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, other-oa)
- Funders
- FourEyes, a research line within project TEC4Growth—Pervasive Intelligence
- North Portugal Regional Operational Programme
- European Regional Development Fund
- Enhancers and Proofs of Concept with Industrial (Impact/NORTE-01- 0145-FEDER-000020)
- PORTUGAL 2020 Partnership Agreement
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- hate speech detection, text classification, definition of hate speech, annotated dataset, multilingual dataset, online communities, digital media platforms, social impact
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.