The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Hate speech detection: A solved problem? The challenging case of long tail on Twitter
Open accessAbstract
In recent years, the increasing propagation of hate speech on social media and the urgent need for effective counter-measures have drawn significant investment from governments, companies, and researchers. A large number of methods have been developed for automated hate speech detection online. Thi s aims to classify textual content into non-hate or hate speech, in which case the method may also identify the targeting characteristics (i.e., types of hate, such as race, and religion) in the hate speech. However, we notice significant difference between the performance of the two (i.e., non-hate vs. hate). In this work, we argue for a focus on the latter problem for practical reasons. We show that it is a much more challenging task, as our analysis of the language in the typical datasets shows that hate speech lacks unique, discriminative features and therefore is found in the ‘long tail’ in a dataset that is difficult to discover. We then propose Deep Neural Network structures serving as feature extractors that are particularly effective for capturing the semantics of hate speech. Our methods are evaluated on the largest collection of hate speech datasets based on Twitter, and are shown to be able to outperform the best performing method by up to 5 percentage points in macro-average F1, or 8 percentage points in the more challenging case of identifying hateful content.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Waseem, Z., & Hovy, D. (2016) · Proceedings of the NAACL Student Research Workshop · doi:10.18653/v1/n16-2013 · Cited by 1,655
A Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media · doi:10.18653/v1/w17-1101 · Cited by 1,395
Abusive Language Detection in Online User Content
Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., & Chang, Y. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883062 · Cited by 1,122
Detecting Hate Speech on the World Wide Web
Warner, W. B., & Hirschberg, J. (2012) · Cited by 964
Deep Learning for Hate Speech Detection in Tweets
Badjatiya, P., Gupta, S., Gupta, M., & Varma, V. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3054223 · Cited by 849
Hate Speech Detection with Comment Embeddings
Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., & Bhamidipati, N. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2740908.2742760 · Cited by 712
Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network
Zhang, Z., Robinson, D., & Tepper, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93417-4_48 · Cited by 638
Self-citationDetecting Offensive Language in Social Media to Protect Adolescent Online Safety
Chen, Y., Zhou, Y., Zhu, S., & Xu, H. (2012) · 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing · doi:10.1109/socialcom-passat.2012.55 · Cited by 637
Burnap, P., & Williams, M. L. (2015) · Policy & Internet · doi:10.1002/poi3.85 · Cited by 616
Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter
Waseem, Z. (2016) · Proceedings of the First Workshop on NLP and Computational Social Science · doi:10.18653/v1/w16-5618 · Cited by 569
Using Convolutional Neural Networks to Classify Hate-Speech
Gambäck, B., & Sikdar, U. K. (2017) · Proceedings of the First Workshop on Abusive Language Online · doi:10.18653/v1/w17-3013 · Cited by 532
A Lexicon-based Approach for Hate Speech Detection
Gitari, N. D., Zhang, Z., Damien, H., & Long, J. (2015) · International Journal of Multimedia and Ubiquitous Engineering · doi:10.14257/ijmue.2015.10.4.21 · Cited by 502
Self-citationA multidimensional approach for detecting irony in Twitter
Reyes, A., Rosso, P., & Veale, T. (2013) · Language Resources and Evaluation · doi:10.1007/s10579-012-9196-x · Cited by 454
Locate the Hate: Detecting Tweets against Blacks
Kwok, I., & Wang, Y. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8539 · Cited by 446
Common Sense Reasoning for Detection, Prevention, and Mitigation of Cyberbullying
Dinakar, K., Jones, B., Havasi, C., Lieberman, H., & Picard, R. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2362394.2362400 · Cited by 432
Us and them: identifying cyber hate on Twitter across multiple protected characteristics
Burnap, P., & Williams, M. L. (2016) · EPJ Data Science · doi:10.1140/epjds/s13688-016-0072-6 · Cited by 396
Improving Cyberbullying Detection with User Context
Dadvar, M., Trieschnigg, D., Ordelman, R., & de Jong, F. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-36973-5_62 · Cited by 376
Hate Me, Hate Me Not: Hate Speech Detection on Facebook.
Vigna, F. D., Cimino, A., Dell’Orletta⋄, F., Petrocchi, M., & Tesconi, M. (2017) · CNR ExploRA · Cited by 366
One-step and Two-step Classification for Abusive Language Detection on Twitter
Park, J. H., & Fung, P. (2017) · Proceedings of the First Workshop on Abusive Language Online · doi:10.18653/v1/w17-3006 · Cited by 353
Detecting offensive tweets via topical feature discovery over a large scale twitter corpus
Xiang, G., Fan, B., Wang, L., Hong, J., & Rose, C. (2012) · Proceedings of the 21st ACM international conference on Information and knowledge management · doi:10.1145/2396761.2398556 · Cited by 267
Characterizing and Detecting Hateful Users on Twitter
Ribeiro, M., Calais, P., Santos, Y., Almeida, V., & Meira Jr., W. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15057 · Cited by 221
Do Characters Abuse More Than Words?
Mehdad, Y., & Tetreault, J. (2016) · Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w16-3638 · Cited by 198
Classifying racist texts using a support vector machine
Greevy, E., & Smeaton, A. F. (2004) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1008992.1009074 · Cited by 194
Content-driven detection of cyberbullying on the instagram social network
Zhong, H., Li, H., Squicciarini, A., Rajtmajer, S. M., Griffin, C. H., Miller, D. J., & Caragea, C. (2016) · International Joint Conference on Artificial Intelligence · Cited by 133
Modeling Skip-Grams for Event Detection with Convolutional Neural Networks
Nguyen, T. H., & Grishman, R. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1085 · Cited by 127
Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
Chiu, B., Korhonen, A., & Pyysalo, S. (2016) · Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2501 · Cited by 95
A Two Phase Deep Learning Model for Identifying Discrimination from Tweets
Yuan, S., Wu, X., & Xiang, Y. (2016) · Movebank · doi:10.5441/002/edbt.2016.92 · Cited by 63
Hate Speech Detection on Twitter: Feature Engineering v.s. Feature Selection
Robinson, D., Zhang, Z., & Tepper, J. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98192-5_9 · Cited by 51
Self-citationData Sets: Word Embeddings Learned from Tweets and General Data
Li, Q., Shah, S., Liu, X., & Nourbakhsh, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14859 · Cited by 44
Research works printed but not linked: 1
- Nockleby, 2000, Hate Speech, 1277
Cited by
Internet, social media and online hate speech. Systematic review
Castaño-Pulgarín, S. A., Suárez-Betancur, N., Vega, L. M. T., & López, H. M. H. (2021) · Aggression and Violent Behavior · doi:10.1016/j.avb.2021.101608 · Cited by 395
A systematic review of hate speech automatic detection using natural language processing
Jahan, M. S., & Oussalah, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126232 · Cited by 279
Gongane, V. U., Munot, M. V., & Anuse, A. D. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00951-3 · Cited by 153
Combining FastText and Glove Word Embedding for Offensive and Hate speech Text Detection
Badri, N., Kboubi, F., & Chaibi, A. H. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.132 · Cited by 65
Ahmed, T., Ivan, S., Kabir, M., Mahmud, H., & Hasan, K. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00934-4 · Cited by 42
Metadata
- Authors
- Zhang, Z., & Luo, L.
- Venue
- Semantic Web
- Publisher
- SAGE Publications
- Published
- 2019-09-26
- Volume
- 10
- Issue
- 5
- Pages
- 925-945
- ISSN
- 1570-0844 (ISSN-L)
- 2210-4968
- OpenAlex
- W2963790884
- Cited by
- 325
- References
- 37
- FWCI
- 18.95
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.