Classification and Its Consequences for Online Harassment: Design Insights from HeartMob
The DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index.
Abstract
Online harassment is a pervasive and pernicious problem. Techniques like natural language processing and machine learning are promising approaches for identifying abusive language, but they fail to address structural power imbalances perpetuated by automated labeling and classification. Similarly, platform policies and reporting tools are designed for a seemingly homogenous user base and do not account for individual experiences and systems of social oppression. This paper describes the design and evaluation of HeartMob, a platform built by and for people who are disproportionately affected by the most severe forms of online harassment. We conducted interviews with 18 HeartMob users, both targets and supporters, about their harassment experiences and their use of the site. We examine systems of classification enacted by technical systems, platform policies, and users to demonstrate how 1) labeling serves to validate (or invalidate) harassment experiences; 2) labeling motivates bystanders to provide support; and 3) labeling content as harassment is critical for surfacing community norms around appropriate user behavior. We discuss these results through the lens of Bowker and Star's classification theories and describe implications for labeling and classifying online abuse. Finally, informed by intersectional feminist theory, we argue that fully addressing online harassment requires the ongoing integration of vulnerable users' needs into the design and moderation of online platforms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [10]
Citations
Cites
Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color
Crenshaw, K. (1991) · Stanford Law Review · doi:10.2307/1229039 · Cited by 32,792 · cited 3×
“As a result, there are severe gaps in legal protections for women of color, which white women do not experience [16].”
BackgroundInfluentialA General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858 · cited 1×
“We used an inductive approach to develop codes [42].”
MethodStar, S. L., & Griesemer, J. R. (1989) · Social Studies of Science · doi:10.1177/030631289019003001 · Cited by 10,847
Sorting Things Out: Classification and Its Consequences
(2000) · The Journal of Academic Librarianship · doi:10.1016/s0099-1333(00)00155-5 · Cited by 5,210
Cyberbullying: Its nature and impact in secondary school pupils
Smith, P. K., Mahdavi, J., Carvalho, M., Fisher, S., Russell, S., & Tippett, N. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2007.01846.x · Cited by 3,639 · cited 1×
“One individual may also harass another, as is often the case in instances of cyberbullying [3,39] and non-consensual image sharing (also known as “revenge porn”), a form of doxing in which sexually explicit images or videos are distributed without their subject’s consent, often by a former romantic…”
BackgroundA Room with a Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels
Goldstein, N. J., Cialdini, R. B., & Griskevicius, V. (2008) · Journal of Consumer Research · doi:10.1086/586910 · Cited by 3,409
Ahmed, S. (2024) · Feminist Theory · doi:10.1007/978-3-031-55397-4_18 · Cited by 2,722
Ahmed, S. (2017) · doi:10.1215/9780822373377 · Cited by 2,722 · cited 2×
“Intersectional feminist theory [1,15,25] holds that various identities (such as race, gender, class, sexuality, religion, disability, and nationality) are inextricably bound in systems of entrenched structural oppression (such as racism, sexism, cissexism, classism, heterosexism, ableism, and colonialism).”
BackgroundRethinking Standpoint Epistemology: What Is ‘Strong Objectivity’?
Harding, S. (1996) · Feminism And Science · doi:10.1093/oso/9780198751458.003.0016 · Cited by 1,955
Unpacking "privacy" for a networked world
Palen, L., & Dourish, P. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642635 · Cited by 965 · cited 1×
“A digital audience is “large, unknown and distant” [34], and users must continually negotiate their own privacy and impression management practices to satisfy both their own expectations and those of a functionally invisible audience [31,34,44].”
Background
Research works printed but not linked: 13
- Becker Howard, 1982, The Free Press 9
- Geoffrey, 2000, Bowker and Susan Leigh Star
- Citron Danielle Keats, Criminalizing Revenge Porn. Social Science Research Network
- Donath Judith, Identity and Deception in the Virtual Community. Communities in cyberspace
- Kiesler Sara, Regulating behavior in online communities. Building Successful Online Communities: Evidence-Based Social Design
- Lampe Cliff, Gamification and social media. The Gameful world: Approaches, issues, applications
- Lenhart Amanda, Digital Abuse, and Cyberstalking in America
- Marx Karl, 1848, The communist manifesto
- Rainie Lee, Trolls, Anonymity and Fake News Online
- Sproull Lee, 1992, Kiesler
- Star Susan Leigh, 1989, Griesemer
- U. S. Department of Labor. 2015., Current Population Survey: Detailed occupation by sex and race
- Walther Joseph B., 2002, Parks
Cited by
Starbird, K., Arif, A., & Wilson, T. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359229 · Cited by 370
Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator
Jhaver, S., Birman, I., Gilbert, E., & Bruckman, A. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3338243 · Cited by 289
“found that using automated approaches to identify abusive language can result in situations where the concerns of only the dominant groups are emphasized and existing structural inequalities are exacerbated [8].”
BackgroundOnline Harassment and Content Moderation: The Case of Blocklists
Jhaver, S., Ghoshal, S., Bruckman, A., & Gilbert, E. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3185593 · Cited by 270
“argued that public demonstrations of support not only provide validation for targets of harassment, but also create powerful descriptive norms that help other users determine what behaviors are and are not appropriate in an online community [5].”
BackgroundScheuerman, M. K., Paul, J. M., & Brubaker, J. R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359246 · Cited by 252
“For example, Blackwell et al. describe the marginalization of users whose experiences with harassment are invalidated when they meet rigid classifications [16].”
BackgroundScheuerman, M. K., Branham, S. M., & Hamidi, F. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274424 · Cited by 219
Metadata
- Authors
- Blackwell, L., Dimond, J., Schoenebeck, S., & Lampe, C.
- Affiliations
- University of Michigan (US)
- Sassafras Tech Collective, Ann Arbor, MI, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2017-12-06
- Volume
- 1
- Issue
- CSCW
- Pages
- 1-19
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2775336929
- Cited by
- 245
- References
- 35
- Citations by year
- 2026: 15 · 2025: 41 · 2024: 34 · 2023: 32 · 2022: 31 · 2021: 34 · 2020: 22 · 2019: 22 · 2018: 14
- FWCI
- 9.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, public-domain)
- Funders
- Knight Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- online harassment, automated content moderation, platform policies, intersectionality, feminist theory, platform design, vulnerable users, community norms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.