SoK: Hate, Harassment, and the Changing Landscape of Online Abuse
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
We argue that existing security, privacy, and antiabuse protections fail to address the growing threat of online hate and harassment. In order for our community to understand and address this gap, we propose a taxonomy for reasoning about online hate and harassment. Our taxonomy draws on over 150 interdisciplinary research papers that cover disparate threats ranging from intimate partner violence to coordinated mobs. In the process, we identify seven classes of attacks—such as toxic content and surveillance—that each stem from different attacker capabilities and intents. We also provide longitudinal evidence from a three-year survey that hate and harassment is a pervasive, growing experience for online users, particularly for at-risk communities like young adults and people who identify as LGBTQ+. Responding to each class of hate and harassment requires a unique strategy and we highlight five such potential research directions that ultimately empower individuals, communities, and platforms to do so.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Cyberbullying: Another main type of bullying?
SLONJE, R., & SMITH, P. K. (2008) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.2007.00611.x · Cited by 1,844 · cited 1×
“While hate and harassment have a long history in the social sciences and the ethos of the Internet [92], [130]—”
BackgroundMeasuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · cited 1×
“Unlike a for-profit abuse context, bias in training data can result in classifiers incorrectly learning that terms for at-risk populations like “gay” or “black” are by default hate and harassment [5], [50].”
BackgroundEx Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636
Auditing radicalization pathways on YouTube
Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., & Meira, W. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372879 · Cited by 580
Cutting the Gordian Knot: A Look Under the Hood of Ransomware Attacks
Kharraz, A., Robertson, W., Balzarotti, D., Bilge, L., & Kirda, E. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-20550-2_1 · Cited by 458 · cited 1×
“Ransomware represents the closest equivalent in a for-profit cybercrime context [90].”
BackgroundYou Can't Stay Here: The Efficacy of Reddit's 2015 Ban Examined Through Hate Speech
Chandrasekharan, E., Pavalanathan, U., Srinivasan, A., Glynn, A., Eisenstein, J., & Gilbert, E. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134666 · Cited by 390
Starbird, K., Arif, A., & Wilson, T. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359229 · Cited by 370
Hate Lingo: A Target-Based Linguistic Analysis of Hate Speech in Social Media
ElSherief, M., Kulkarni, V., Nguyen, D., Yang Wang, W., & Belding, E. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15041 · Cited by 294
Chandrasekharan, E., Samory, M., Jhaver, S., Charvat, H., Bruckman, A., Lampe, C., Eisenstein, J., & Gilbert, E. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274301 · Cited by 282
Online Harassment and Content Moderation: The Case of Blocklists
Jhaver, S., Ghoshal, S., Bruckman, A., & Gilbert, E. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3185593 · Cited by 270
Classification and Its Consequences for Online Harassment: Design Insights from HeartMob
Blackwell, L., Dimond, J., Schoenebeck, S., & Lampe, C. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134659 · Cited by 245 · cited 2×
“HeartMob provided an interface to report hate and harassment to bystanders for confirmation and emotional support [13].”
Background"They Don't Leave Us Alone Anywhere We Go": Gender and Digital Abuse in South Asia
Sambasivan, N., Batool, A., Ahmed, N., Matthews, T., Thomas, K., Gaytán-Lugo, L. S., Nemer, D., Bursztein, E., Churchill, E., & Consolvo, S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300232 · Cited by 175
Self-citationThe Spyware Used in Intimate Partner Violence
Chatterjee, R., Doerfler, P., Orgad, H., Havron, S., Palmer, J., Freed, D., Levy, K., Dell, N., McCoy, D., & Ristenpart, T. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00061 · Cited by 142 · cited 2×
“attacks include intimate partner violence [27], [65], [66], [108], anonymous peers breaking into a target’s account to leak personal communication and photos [131], and coordinated bullying and sexual harassment campaigns that involve tens of thousands of attackers [1].”
BackgroundSelf-citationDetection of Cyberbullying Incidents on the Instagram Social Network
Hosseinmardi, H., Mattson, S. A., Rafiq, R. I., Han, R., Lv, Q., & Mishra, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03909 · Cited by 126
Mobilizing the Trump Train: Understanding Collective Action in a Political Trolling Community
Flores-Saviaga, C., Keegan, B., & Savage, S. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15024 · Cited by 111
Evaluating Login Challenges as aDefense Against Account Takeover
Doerfler, P., Thomas, K., Marincenko, M., Ranieri, J., Jiang, Y., Moscicki, A., & McCoy, D. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313481 · Cited by 39 · cited 1×
“For example, two-factor authentication and security checkups can stem the risk of unauthorized access—similar to a for-profit abuse context [52]—reducing the risk of surveillance, lockout and control, and content leakage.”
BackgroundSelf-citation
Cited by
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“Taxonomies have been developed related to online content [19, 193, 227], social media [157, 217, 218], users “at-risk" for experiencing online abuse [216, 234], and mali-”
BackgroundZhang, R., Li, H., Meng, H., Zhan, J., Gan, H., & Lee, Y.-C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713429 · Cited by 123
“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”
Social media affordances of LGBTQIA+ expression and community formation
Are, C., Talbot, C., & Briggs, P. (2025) · Convergence: The International Journal of Research into New Media Technologies · doi:10.1177/13548565241296628 · Cited by 17
Accountability Gaps in the Fake Follower Market
Kalameyets, M., Owens, R., Hammouchi, H., Sergeeva, A., Hoehn, S., Jalilzade, E., Wang, J., Patel, S., Aidinlis, S., Farrand, B., et al. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806758
“Overlaps with toxic content and abuse amplify risks: hate speech, extremism, and misinformation often intersect with fake social activity [56], and FAS can function as enabling infrastructure for them by supplying rented engagement, amplification, and visibility-on-demand.”
Metadata
- Authors
- Thomas, K., Akhawe, D., Bailey, M., Boneh, D., Bursztein, E., Consolvo, S., Dell, N., Durumeric, Z., Kelley, P. G., Kumar, D., McCoy, D., Meiklejohn, S., Ristenpart, T., & Stringhini, G.
- Venue
- 2021 IEEE Symposium on Security and Privacy (SP)
- Publisher
- IEEE
- Published
- 2021-05
- Pages
- 247-267
- ISSN
- 1081-6011 (ISSN-L)
- OpenAlex
- W3125759795
- Cited by
- 148
- References
- 17
- FWCI
- 10.74
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.