“HOT” ChatGPT: The Promise of ChatGPT in Detecting and Discriminating Hateful, Offensive, and Toxic Comments on Social Media
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Harmful textual content is pervasive on social media, poisoning online communities and negatively impacting participation. A common approach to this issue is developing detection models that rely on human annotations. However, the tasks required to build such models expose annotators to harmful and offensive content and may require significant time and cost to complete. Generative AI models have the potential to understand and detect harmful textual content. We used ChatGPT to investigate this potential and compared its performance with MTurker annotations for three frequently discussed concepts related to harmful textual content on social media: Hateful, Offensive, and Toxic (HOT). We designed five prompts to interact with ChatGPT and conducted four experiments eliciting HOT classifications. Our results show that ChatGPT can achieve an accuracy of approximately 80% when compared to MTurker annotations. Specifically, the model displays a more consistent classification for non-HOT comments than HOT comments compared to human annotations. Our findings also suggest that ChatGPT classifications align with the provided HOT definitions. However, ChatGPT classifies “hateful” and “offensive” as subsets of “toxic.” Moreover, the choice of prompts used to interact with ChatGPT impacts its performance. Based on these insights, our study provides several meaningful implications for employing ChatGPT to detect HOT content, particularly regarding the reliability and consistency of its performance, its understanding and reasoning of the HOT concept, and the impact of prompts on its performance. Overall, our study provides guidance on the potential of using generative AI models for moderating large volumes of user-generated textual content on social media.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
A Survey on Automatic Detection of Hate Speech in Text
Fortuna, P., & Nunes, S. (2019) · ACM Computing Surveys · doi:10.1145/3232676 · Cited by 1,241
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · cited 3×
“This recommendation is also in line with Gilardi et al. (2023) [ 55 ] in that a lower temperature yields more accurate annotations.”
BackgroundResultQuestioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
The Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · cited 1×
“This is especially crucial for vulnerable communities who are disproportionately affected by these problems (Matamoros-Fernández & Farkas, 2021; Mossie & Wang, 2020; Sap et al., 2019).”
BackgroundMeasuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · cited 1×
“However, this approach has some limitations, principally because relying solely on keyword filtering or predetermined rules does not capture the context or intent of the message [ 42 ].”
MethodEx Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636
Hate speech detection: Challenges and solutions
MacAvaney, S., Yao, H.-R., Yang, E., Russell, K., Goharian, N., & Frieder, O. (2019) · PLOS ONE · doi:10.1371/journal.pone.0221152 · Cited by 566 · cited 1×
“Once the text is vectorized, ML classifiers, such as support vector machines (SVMs) or neural networks, can be trained to detect HOT content [ 52 , 53 ].”
MethodHow is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507
Cyberbullying Via Social Media
Whittaker, E., & Kowalski, R. M. (2015) · Journal of School Violence · doi:10.1080/15388220.2014.949377 · Cited by 498 · cited 1×
“These negative effects make it crucial to develop models that detect and moderate such textual content on social media [ 3 , 4 ].”
BackgroundLarge Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · cited 1×
“As a result, there is a growing trend in exploring the potential of generative AI for performing a variety of tasks, including toxicity detection [ 11 –13 ].”
BackgroundHuman-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator
Jhaver, S., Birman, I., Gilbert, E., & Bruckman, A. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3338243 · Cited by 289
Online Harassment and Content Moderation: The Case of Blocklists
Jhaver, S., Ghoshal, S., Bruckman, A., & Gilbert, E. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3185593 · Cited by 270
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209479 · Cited by 204 · cited 1×
“As a result, there is a growing trend in exploring the potential of generative AI for performing a variety of tasks, including toxicity detection (Sorensen et al., 2022; Strobelt et al., 2022; Zhou et al., 2023).”
BackgroundInteractive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204 · cited 1×
“As a result, there is a growing trend in exploring the potential of generative AI for performing a variety of tasks, including toxicity detection (Sorensen et al., 2022; Strobelt et al., 2022; Zhou et al., 2023).”
BackgroundSocial Support, Reciprocity, and Anonymity in Responses to Sexual Abuse Disclosures on Social Media
Andalibi, N., Haimson, O. L., Choudhury, M. D., & Forte, A. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3234942 · Cited by 200
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · cited 1×
“For instance, Riedl et al. (2020) [ 54 ] found that rating “uncivil” comments had a significant psychological toll on annotators and reduced their accuracy on subsequent tasks.”
BackgroundBiCHAT: BiLSTM with deep CNN and hierarchical attention for hate speech detection
Khan, S., Fazil, M., Sejwal, V. K., Alshara, M. A., Alotaibi, R. M., Kamal, A., & Baig, A. R. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.05.006 · Cited by 115 · cited 1×
“Kolhatkar et al. (2020) [ 36 ] Comments that use harsh, offensive, or abusive language, which include personal attacks or insults, or which are derogatory or demeaning.”
BackgroundAn Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., & Wingate, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.60 · Cited by 113 · cited 1×
“As a result, there is a growing trend in exploring the potential of generative AI for performing a variety of tasks, including toxicity detection (Sorensen et al., 2022; Strobelt et al., 2022; Zhou et al., 2023).”
BERT-based ensemble learning for multi-aspect hate speech detection
Mazari, A. C., Boudoukhani, N., & Djeffal, A. (2024) · Cluster Computing · doi:10.1007/s10586-022-03956-x · Cited by 89
Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
Combining FastText and Glove Word Embedding for Offensive and Hate speech Text Detection
Badri, N., Kboubi, F., & Chaibi, A. H. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.132 · Cited by 65
The downsides of digital labor: Exploring the toll incivility takes on online comment moderators
Riedl, M. J., Masullo, G. M., & Whipple, K. N. (2020) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106262 · Cited by 48
Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark
Reiss, M. V. (2023) · arXiv · doi:10.48550/arxiv.2304.11085 · Cited by 34
Cited by
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Sai, S., Yashvardhan, U., Chamola, V., & Sikdar, B. (2024) · IEEE Access · doi:10.1109/access.2024.3385107 · Cited by 125
Content moderation by LLM: from accuracy to legitimacy
Huang, T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11328-1 · Cited by 24
“Such practice generates substantial costs for subsidizing the labor ([108]; [70], 2) and also introduces biases into the process ([33]).”
Background
Metadata
- Authors
- Li, L., Fan, L., Atreja, S., & Hemphill, L.
- Venue
- ACM Transactions on the Web
- Published
- 2024-03-12
- Volume
- 18
- Issue
- 2
- Pages
- 1-36
- ISSN
- 1559-1131 (ISSN-L)
- 1559-114X
- OpenAlex
- W4391473457
- Cited by
- 102
- References
- 31
- FWCI
- 13.54
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.