Classification and Its Consequences for Online Harassment: Design Insights from HeartMob
The DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index.
Abstract
Online harassment is a pervasive and pernicious problem. Techniques like natural language processing and machine learning are promising approaches for identifying abusive language, but they fail to address structural power imbalances perpetuated by automated labeling and classification. Similarly, platform policies and reporting tools are designed for a seemingly homogenous user base and do not account for individual experiences and systems of social oppression. This paper describes the design and evaluation of HeartMob, a platform built by and for people who are disproportionately affected by the most severe forms of online harassment. We conducted interviews with 18 HeartMob users, both targets and supporters, about their harassment experiences and their use of the site. We examine systems of classification enacted by technical systems, platform policies, and users to demonstrate how 1) labeling serves to validate (or invalidate) harassment experiences; 2) labeling motivates bystanders to provide support; and 3) labeling content as harassment is critical for surfacing community norms around appropriate user behavior. We discuss these results through the lens of Bowker and Star's classification theories and describe implications for labeling and classifying online abuse. Finally, informed by intersectional feminist theory, we argue that fully addressing online harassment requires the ongoing integration of vulnerable users' needs into the design and moderation of online platforms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [10]
Citations
Cites
Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color
Crenshaw, K. (1991) · Stanford Law Review · doi:10.2307/1229039 · Cited by 32,792 · cited 3×
“As a result, there are severe gaps in legal protections for women of color, which white women do not experience [16].”
BackgroundInfluentialA General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858 · cited 1×
“We used an inductive approach to develop codes [42].”
MethodStar, S. L., & Griesemer, J. R. (1989) · Social Studies of Science · doi:10.1177/030631289019003001 · Cited by 10,847
Sorting Things Out: Classification and Its Consequences
(2000) · The Journal of Academic Librarianship · doi:10.1016/s0099-1333(00)00155-5 · Cited by 5,210
Cyberbullying: Its nature and impact in secondary school pupils
Smith, P. K., Mahdavi, J., Carvalho, M., Fisher, S., Russell, S., & Tippett, N. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2007.01846.x · Cited by 3,639 · cited 1×
“One individual may also harass another, as is often the case in instances of cyberbullying [3,39] and non-consensual image sharing (also known as “revenge porn”), a form of doxing in which sexually explicit images or videos are distributed without their subject’s consent, often by a former romantic…”
BackgroundA Room with a Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels
Goldstein, N. J., Cialdini, R. B., & Griskevicius, V. (2008) · Journal of Consumer Research · doi:10.1086/586910 · Cited by 3,409
Ahmed, S. (2024) · Feminist Theory · doi:10.1007/978-3-031-55397-4_18 · Cited by 2,722
Ahmed, S. (2017) · doi:10.1215/9780822373377 · Cited by 2,722 · cited 2×
“Intersectional feminist theory [1,15,25] holds that various identities (such as race, gender, class, sexuality, religion, disability, and nationality) are inextricably bound in systems of entrenched structural oppression (such as racism, sexism, cissexism, classism, heterosexism, ableism, and colonialism).”
BackgroundRethinking Standpoint Epistemology: What Is ‘Strong Objectivity’?
Harding, S. (1996) · Feminism And Science · doi:10.1093/oso/9780198751458.003.0016 · Cited by 1,955
Unpacking "privacy" for a networked world
Palen, L., & Dourish, P. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642635 · Cited by 965 · cited 1×
“A digital audience is “large, unknown and distant” [34], and users must continually negotiate their own privacy and impression management practices to satisfy both their own expectations and those of a functionally invisible audience [31,34,44].”
Background
Research works printed but not linked: 13
- Becker Howard, 1982, The Free Press 9
- Geoffrey, 2000, Bowker and Susan Leigh Star
- Citron Danielle Keats, Criminalizing Revenge Porn. Social Science Research Network
- Donath Judith, Identity and Deception in the Virtual Community. Communities in cyberspace
- Kiesler Sara, Regulating behavior in online communities. Building Successful Online Communities: Evidence-Based Social Design
- Lampe Cliff, Gamification and social media. The Gameful world: Approaches, issues, applications
- Lenhart Amanda, Digital Abuse, and Cyberstalking in America
- Marx Karl, 1848, The communist manifesto
- Rainie Lee, Trolls, Anonymity and Fake News Online
- Sproull Lee, 1992, Kiesler
- Star Susan Leigh, 1989, Griesemer
- U. S. Department of Labor. 2015., Current Population Survey: Detailed occupation by sex and race
- Walther Joseph B., 2002, Parks
Cited by
Items 201–246 of 246
Cyber harassment of children with special reference to digital games
Baltezarević, V., Baltezarević, R., & Baltezarević, I. (2023) · Temida · doi:10.2298/tem2302261b · Cited by 6
Lackey, C., & Taylor, S. H. (2023) · AoIR Selected Papers of Internet Research · doi:10.5210/spir.v2023i0.13441 · Cited by 4
Understanding the mediator and moderator role of digital citizenship on cyber harassment
Macharia, M., & Dunaway, M. (2025) · Information Technology and People · doi:10.1108/itp-05-2022-0376 · Cited by 6
Guntrum, L. G., Nuñez, D. F., & Reuter, C. (2025) · i-com · doi:10.1515/icom-2025-0004 · Cited by 3
Liao, Z., Zhao, H., Kulkarni, A., Chattrath, S., & Zhang, A. X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757460 · Cited by 13
“In this study, we built proactive and instant-reactive strategies to combat embodied harassment in social VR. Proactive designs prevent harassment, and instant-reactive designs provide opportunities for players to intervene in ongoing harassment to minimize harm.”
Knittel, M. L., & Menking, A. (2024) · Journal of Online Trust and Safety · doi:10.54501/jots.v2i2.173 · Cited by 3
Automatic Moderation of User-Generated Content
Annamoradnejad, I., & Habibi, J. (2023) · IGI Global eBooks · doi:10.4018/978-1-7998-9220-5.ch079 · Cited by 2
Seeing the Politics of Decentralized Social Media Protocols
Oshinowo, T., Hwang, S., Zhang, A. X., & Monroy-Hernández, A. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816995 · Cited by 9
“…space, we encourage HCI and STS scholars—particularly those at CSCW—as well as builders and designers, to engage more deliberately with the values and politics encoded in decentralized infrastructures, asking not only how to decentralize, but for whom, to what ends, and at what costs [9, 112, 115].”
Evaluating Automated System Interventions Against Email Harassment
Chen, N., Kane, C., & Hang, E. Z. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451524 · Cited by 1
Creation of Fake Identities on Social Media for Pleasure Seeking: An Analysis of Facebook
Taj, S., & Khan, Z. (2024) · International Journal of Data Science and Analysis · doi:10.11648/j.ijdsa.20241006.12 · Cited by 1
Freeman, G., Frommel, J., Mandryk, R. L., Gugenheimer, J., Li, L., Zytko, D., Razi, A., & Lampe, C. A. C. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706708 · Cited by 1
“…problematic, and harmful interactions (e.g., harassment, verbal abuse, hate speech, and flaming on social media platforms and online forums) remains a severe and pervasive issue in today’s online social spaces, which seriously damages victims’ mental, emotional, and physical well-being [5, 17, 33].”
Self-citationVictim-Survivor Orientated Technology-Based Interventions to Address e-DFV: A Scoping Review
MacPhail, C. L., Barrington, M., Simpson, H. B., Rambaldini‐Gooding, D., Lewer, K., McLean, N., Win, K. T., & Klocker, N. (2025) · Journal of Family Violence · doi:10.1007/s10896-025-00881-6 · Cited by 3
“Indeed reports of online abuse are not always validated and actioned by moderators, which can cause further harm to victim-survivors (Blackwell et al., 2017).”
Wang, Y., Umashankar, M., Chandrasekharan, E., & Sundaram, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757668 · Cited by 6
Understanding Fanchuan in Livestreaming Platforms: A New Form of Online Antisocial Behavior
Wei, Y., He, J., & Tyson, G. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757690 · Cited by 3
Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation
Koshy, V., Choi, F., Chiang, Y.-S., Sundaram, H., Chandrasekharan, E., & Karahalios, K. G. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757699 · Cited by 15
Who Gets Written In? Gender, Identity, and Moderation in AO3’s Celebrity Fanfiction
Zhang, P., & Tyson, G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791973 · Cited by 1
Mandryk, R. L., Freeman, G., Frommel, J., Gugenheimer, J., Lampe, C. A. C., Li, L., Wijkstra, M., & Zytko, D. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778727 · Cited by 1
“…and harmful online interactions represent a severe and pervasive issue across diverse contemporary social spaces—from social media to online gaming to online dating to online virtual worlds—inflicting harm on the mental, emotional, and physical well-being of users and community members [4, 23, 52].”
Self-citationInclusive and Accessible Bystander Interventions for Technology Facilitated Sexual Violence
Chen, Y., McKay, D., Anderson, K. L. M., & Andrade, R. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3716879 · Cited by 1
Dym, B., & Fiesler, C. L. (2019) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/3323994.3369881 · Cited by 1
The Bully Pulpit:trolling the trolls with polysemous monstrosity
Snider, M. M. (2021) · Communication Culture and Critique · doi:10.1093/ccc/tcab004
Evaluating Automated System Interventions Against Email Harassment
Chen, N., Kane, C., & Hang, E. Z. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397482.3450724
“Products like HeartMob [1] and Squadbox [8] have also been developed to provide support to victims of email harassment through community-based strategies.”
BackgroundHow Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?Preprint
Sen, I., Samory, M., Floeck, F., Wagner, C., & Augenstein, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07022 · Cited by 5
“Constructs like sexism and hate speech detection are often depicted as neutral or objective but they are deeply contextual, subjective and ambiguous (Vidgen et al., 2019; Jurgens et al., 2019; Nakov et al., 2021), where misclassifications can cause harm (Blackwell et al., 2017).”
BackgroundIntegrating Signals of Online Harassment into Clinical Practices
Nova, F. F. (2022) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/3565967.3571761
“Online harassment or cyberbully is defined as when someone repeatedly and intentionally harasses, mistreats, or makes fun of another person aiming to scare, anger or shame them using cell phones or other electronic devices and online platforms [2, 22].”
BackgroundWomen survivors' experiences of and responses to technology-facilitated sexual violence
Sarkar, S. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.242462 · Cited by 5
Forestalling Cyber Bullying and Online Harassment
Kritika, E. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-3406-5.ch010 · Cited by 1
An X Study of the Evolution of COVID-19-Related Sentiments in the UK
Palomino, M. A., Allen, R., & Varma, A. P. (2025) · Artificial intelligence · doi:10.5772/intechopen.1008866
Harassment Detection in Online Discourse Using Modified Metaheuristic Natural Language Processing
Kozakijevic, S., Jovanović, L., Perišić, J., Bačanin, N., Mravik, M., Zivkovic, M. Z., & Antonijević, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86296-0_25
Threads and Needles: A Value-Sensitive Design Approach to Online Toxicity
Jenkins, R. (2025) · Science and Engineering Ethics · doi:10.1007/s11948-025-00533-4
“And, in fact, there is good evidence that the tenor of user interactions responds to platform-specific designs and policies (Blackwell et al., 2017; Buntain et al., 2021; Sun et al., 2021; Sude & Dvir-Gvirsman, 2023; though see Avalle et al., 2024 for an important counterpoint).”
Valencia, G., Serrano, M. D., & Gálvez‐Muñoz, L. (2025) · Information Communication & Society · doi:10.1080/1369118x.2025.2524576
Mapping Community Appeals Systems: Lessons for Community-led Moderation in Multi-Level Governance
Lee, J., Doan, B. N. (., Jee, J., & Seering, J. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757627 · Cited by 5
“For example, Squadbox allows users to manage harassment by requesting help from friend moderators [66].”
SnuggleSense: Empowering Online Harm Survivors Through a Structured Sensemaking Process
Xiao, S., Zou, H., Mathews, A., Rui, J., Cheshire, C., & Salehi, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757425
Oportunidades, riscos e danos nas plataformas adjacentes a games
Nunes, R. M., & Fortim, I. (2025) · Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames) · doi:10.5753/sbgames.2025.9558 · Cited by 2
“Entre as formas mais comuns de assédio online estão insultos, exposição de dados pessoais, cyberbullying e perseguição ( stalking ) (Blackwell, 2017), práticas frequentemente classificadas como comportamento tóxico.”
Multiple Perpetrator Bullying: An Exploration into Risk Factors and Consequences
Diaz, M., Encalada, T. M., & Flexon, J. L. (2025) · Journal of Interpersonal Violence · doi:10.1177/08862605251395658
Embracing Training Dataset Bias for Automated Harmful Detection
Schöpke-Gonzalez, A., Kim, N., & Hemphill, L. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1282
“…within which annotators work – volunteer content moderation, commercial content moderation, automated moderation, or some combination of the three (Blackwell et al., 2017; Roberts, 2019); platform-specific content policies (e.g., Reddit’s Content Policy (Reddit, 2024a), X’s Rules (X, 2024), or…”
Trolled for Teaching: A Mixed-Methods Study of Internet-Facilitated Harassment among Educators
Young, J., & Forte, A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-12773-0_18
Design for Hope: Cultivating Deliberate Hope in the Face of Complex Societal Challenges
Kim, J., Liu, J., Popowski, L., Pyle, C., Arif, A., Hayes, G. R., Hiniker, A., Ju, W. G., Mueller, F., Shen, H., et al. (2025) · Maryland Shared Open Access Repository (USMAI Consortium) · doi:10.13016/m2bqcn-8dh2 · Cited by 1
Taking the control back – An adventure in developing personalized content moderation
Luo, W., Pannuto, P., & Vaccaro, K. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791905 · Cited by 1
Starks, D. L., Thach, H., DeGuia, A., Dillahunt, T. R., & Haimson, O. L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791838
“Similar to prior findings about discrimination, the findings revealed a recurrent pattern where participants initially denied experiencing discrimination [11].”
InfluentialEmpathy Practices in Social Media Discourse: A Multidimensional and Relational Perspective
Chen, Y., Hogan, B., & Hale, S. A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790967
Leveraging AI for Direct Bystander Intervention Against Cyberbullying
Qin, P., Cheng, J., Lee, J., Zhang, J., Liu, Z., & Lee, Y. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816963
Harassment in social virtual reality: frequency, harmfulness, and governance responses
Chen, Q., Bellucci, A., Wu, Q., & Jacucci, G. (2026) · Virtual Reality · doi:10.1007/s10055-026-01388-1 · Cited by 1
Wu, Y., Li, Y., 韩平, Zhao, D., Wang, Q., & Jiang, M. (2026) · Behavioral Psychology/Psicología Conductual · doi:10.31083/bp46146 · Cited by 2
Hughes, A., Savage, A., Hunt, R., Taylor, S., Rackliffe, K., Page, X., & Hansen, D. L. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3817025
O'Brien, C. B., Ferreira-Alturas, I., Ayobi, A., & Warner, M. (2026) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3829807.3829889
TikTok on the Clock: Public Scholarship and CSCW in 2026
Gilbert, S., Fiesler, C. L., Doyle, D. T., Epps-Darling, A. C., Lenhart, A., Beatrys, B., & Starbird, K. (2026) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3785651.3818820
Technology-Based Prevention of Intimate Partner Violence: A Scoping Review
Wong, V., Lee, J., MacArthur, C., & Wallace, J. R. (2026) · Graphics Interface · doi:10.1145/3831423.3831448
“Some apply machine learning to detect harmful content at scale [71], while others critique technoso-lutionist approaches and emphasize survivor-centred design [18].”
Metadata
- Authors
- Blackwell, L., Dimond, J., Schoenebeck, S., & Lampe, C.
- Affiliations
- University of Michigan (US)
- Sassafras Tech Collective, Ann Arbor, MI, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2017-12-06
- Volume
- 1
- Issue
- CSCW
- Pages
- 1-19
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2775336929
- Cited by
- 245
- References
- 35
- Citations by year
- 2026: 15 · 2025: 41 · 2024: 34 · 2023: 32 · 2022: 31 · 2021: 34 · 2020: 22 · 2019: 22 · 2018: 14
- FWCI
- 9.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, public-domain)
- Funders
- Knight Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- online harassment, automated content moderation, platform policies, intersectionality, feminist theory, platform design, vulnerable users, community norms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.