Classification and Its Consequences for Online Harassment: Design Insights from HeartMob
The DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index.
Abstract
Online harassment is a pervasive and pernicious problem. Techniques like natural language processing and machine learning are promising approaches for identifying abusive language, but they fail to address structural power imbalances perpetuated by automated labeling and classification. Similarly, platform policies and reporting tools are designed for a seemingly homogenous user base and do not account for individual experiences and systems of social oppression. This paper describes the design and evaluation of HeartMob, a platform built by and for people who are disproportionately affected by the most severe forms of online harassment. We conducted interviews with 18 HeartMob users, both targets and supporters, about their harassment experiences and their use of the site. We examine systems of classification enacted by technical systems, platform policies, and users to demonstrate how 1) labeling serves to validate (or invalidate) harassment experiences; 2) labeling motivates bystanders to provide support; and 3) labeling content as harassment is critical for surfacing community norms around appropriate user behavior. We discuss these results through the lens of Bowker and Star's classification theories and describe implications for labeling and classifying online abuse. Finally, informed by intersectional feminist theory, we argue that fully addressing online harassment requires the ongoing integration of vulnerable users' needs into the design and moderation of online platforms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [10]
Citations
Cites
Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color
Crenshaw, K. (1991) · Stanford Law Review · doi:10.2307/1229039 · Cited by 32,792 · cited 3×
“As a result, there are severe gaps in legal protections for women of color, which white women do not experience [16].”
BackgroundInfluentialA General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858 · cited 1×
“We used an inductive approach to develop codes [42].”
MethodStar, S. L., & Griesemer, J. R. (1989) · Social Studies of Science · doi:10.1177/030631289019003001 · Cited by 10,847
Sorting Things Out: Classification and Its Consequences
(2000) · The Journal of Academic Librarianship · doi:10.1016/s0099-1333(00)00155-5 · Cited by 5,210
Cyberbullying: Its nature and impact in secondary school pupils
Smith, P. K., Mahdavi, J., Carvalho, M., Fisher, S., Russell, S., & Tippett, N. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2007.01846.x · Cited by 3,639 · cited 1×
“One individual may also harass another, as is often the case in instances of cyberbullying [3,39] and non-consensual image sharing (also known as “revenge porn”), a form of doxing in which sexually explicit images or videos are distributed without their subject’s consent, often by a former romantic…”
BackgroundA Room with a Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels
Goldstein, N. J., Cialdini, R. B., & Griskevicius, V. (2008) · Journal of Consumer Research · doi:10.1086/586910 · Cited by 3,409
Ahmed, S. (2024) · Feminist Theory · doi:10.1007/978-3-031-55397-4_18 · Cited by 2,722
Ahmed, S. (2017) · doi:10.1215/9780822373377 · Cited by 2,722 · cited 2×
“Intersectional feminist theory [1,15,25] holds that various identities (such as race, gender, class, sexuality, religion, disability, and nationality) are inextricably bound in systems of entrenched structural oppression (such as racism, sexism, cissexism, classism, heterosexism, ableism, and colonialism).”
BackgroundRethinking Standpoint Epistemology: What Is ‘Strong Objectivity’?
Harding, S. (1996) · Feminism And Science · doi:10.1093/oso/9780198751458.003.0016 · Cited by 1,955
Unpacking "privacy" for a networked world
Palen, L., & Dourish, P. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642635 · Cited by 965 · cited 1×
“A digital audience is “large, unknown and distant” [34], and users must continually negotiate their own privacy and impression management practices to satisfy both their own expectations and those of a functionally invisible audience [31,34,44].”
BackgroundFeminism Is for Everybody: Passionate Politics
hooks, B. (2014) · Routledge eBooks · doi:10.4324/9781315743189 · Cited by 786 · cited 3×
“For those who have many identities that have been historically dominated, the effect of various oppressions is intensified [15,25].”
BackgroundEx Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636 · cited 2×
“, [9,50,51]); however, our findings show that, like in other cases where descriptive norms are being constantly negotiated, accounting for fluid social nuance may prove challenging for automated approaches.”
BackgroundResultDescriptive Social Norms as Underappreciated Sources of Social Control
Cialdini, R. B. (2007) · Psychometrika · doi:10.1007/s11336-006-1560-6 · Cited by 606 · cited 4×
“Crawford and Gillespie [11] argue that flagged content is not a proxy for non-normative or deviant behavior, but instead represents complex negotiations between platforms, individual users, and broader regulatory forces.”
BackgroundInfluentialTowards a feminist HCI methodology: social science, feminism, and HCI
Bardzell, S., & Bardzell, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979041 · Cited by 509 · cited 13×
“Start from this reference point, or you’ve already failed”—was removed, and resulted in a week-long account suspension.”
BackgroundMethodInfluentialAnyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions
Cheng, J., Bernstein, M., Danescu-Niculescu-Mizil, C., & Leskovec, J. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998213 · Cited by 480 · cited 1×
“Cheng et al. [10] found that perpetrators of online harassment were not, contrary to popular narratives, anti-social actors dedicated to violating norms of civility.”
Detection of Harassment on Web 2.0
Yin, D., Xue, Z., Hong, L., Davison, B. D., & Edwards, L. (2009) · Cited by 451
Slash(dot) and burn: distributed moderation in a large online conversation space
Lampe, C., & Resnick, P. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985761 · Cited by 400 · cited 2×
“In this work, we build on extensive prior research that has explored misbehavior in online communities [17,19,40] and mechanisms for moderating and mitigating it [8,27,29].”
BackgroundSelf-citationThe Imagined Audience on Social Network Sites
Litt, E., & Hargittai, E. (2016) · Social Media + Society · doi:10.1177/2056305116633482 · Cited by 395 · cited 1×
“A digital audience is “large, unknown and distant” [34], and users must continually negotiate their own privacy and impression management practices to satisfy both their own expectations and those of a functionally invisible audience [31,34,44].”
BackgroundThe Public Domain: Surveillance in Everyday Life
Marwick, A. (2012) · Surveillance & Society · doi:10.24908/ss.v9i4.4342 · Cited by 369
Connections: New Ways of Working in the Networked Organization.
Bailyn, L., Sproull, L., & Kiesler, S. (1992) · Administrative Science Quarterly · doi:10.2307/2393454 · Cited by 342
Hollaback!: the role of storytelling online in a social movement organization
Dimond, J. P., Dye, M., Larose, D., & Bruckman, A. S. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441831 · Cited by 248 · cited 1×
“Given their collective expertise in intersectional feminist practice, social movement framing, and bystander intervention to combat harassment in physical spaces [18] , the Hollaback! team sought to translate these practices online to support others with similar experiences.”
BackgroundSelf-citationThe future of free speech, trolls, anonymity and fake news online
Rainie, L., Anderson, J. Q., & Albright, J. (2017) · Cited by 221
Online Harassment, Digital Abuse, and Cyberstalking in America
Lenhart, A., Ybarra, M., Zickuhr, K., & Price-Feeney, M. (2016) · doi:10.69985/ippa8074 · Cited by 179 · cited 9×
“Roughly four in ten American internet users have personally experienced harassment online, including name-calling, embarrassment, physical threats, stalking, sexual harassment, and sustained harassment [20,21,30].”
BackgroundInfluentialAshktorab, Z., & Vitak, J. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858548 · Cited by 176 · cited 1×
“One individual may also harass another, as is often the case in instances of cyberbullying [3,39] and non-consensual image sharing (also known as “revenge porn”), a form of doxing in which sexually explicit images or videos are distributed without their subject’s consent, often by a former romantic…”
BackgroundCyber Stalking: An Analysis Of Online Harassment And Intimidation
Pittaro, M. (2007) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.18794 · Cited by 172 · cited 1×
“Indeed, some targets report distractions from personal responsibilities, work obligations, or sleep [23,35].”
BackgroundThe Bag of Communities: Identifying Abusive Behavior Online with Preexisting Internet Data
Chandrasekharan, E., Samory, M., Srinivasan, A., & Gilbert, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3026018 · Cited by 153
LGBT Parents and Social Media: Advocacy, Privacy, and Disclosure during Shifting Social Movements
Blackwell, L., Hardy, J., Ammari, T., Veinot, T., Lampe, C., & Schoenebeck, S. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858342 · Cited by 146 · cited 1×
“Computer-mediated communication can be cue-sparse and persistent, making it difficult for users to assess who has seen what and when [6,37].”
BackgroundSelf-citationBalancing Audience and Privacy Tensions on Social Network Sites
Vitak, J., Blasiola, S., Patil, S., & Litt, E. (2015) · International journal of communication · Cited by 137
Deceiving Google's Perspective API Built for Detecting Toxic Comments
Hosseini, H., Kannan, S., Zhang, B., & Poovendran, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08138 · Cited by 123 · cited 2×
“As Hosseini, Kannan, Zhang, and Poovendran [26] demonstrate with Google’s Perspective, automatic detection efforts are easily outmaneuvered by “subtly modifying” an otherwise highly toxic phrase in such a way that an automated system will assign it “a significantly lower toxicity score.””
BackgroundMethodSchoenebeck, S. Y., Ellison, N. B., Blackwell, L., Bayer, J., & Falk, E. B. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819923 · Cited by 80 · cited 1×
“Computer-mediated communication can be cue-sparse and persistent, making it difficult for users to assess who has seen what and when [6,37].”
BackgroundSelf-citationThere's methodology in the madness: toward critical HCI ethnography
Williams, A. M., & Irani, L. C. (2010) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1753846.1753857 · Cited by 54 · cited 1×
“Per feminist methodology, the authors recognize the importance of positioning the research team in relation to the current work and our analysis [4,48].”
MethodOccupational health issues concerning Internet use in the workplace
Griffiths, M. D. (2002) · Work & Stress · doi:10.1080/0267837031000071438 · Cited by 37 · cited 1×
“Indeed, some targets report distractions from personal responsibilities, work obligations, or sleep [23,35].”
BackgroundManaging deviant behavior in online communities
Bruckman, A. S., Danis, C. M., Lampe, C. A. C., Sternberg, J., & Waldron, C. (2006) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1125451.1125458 · Cited by 25
Self-citation
Research works printed but not linked: 13
- Becker Howard, 1982, The Free Press 9
- Geoffrey, 2000, Bowker and Susan Leigh Star
- Citron Danielle Keats, Criminalizing Revenge Porn. Social Science Research Network
- Donath Judith, Identity and Deception in the Virtual Community. Communities in cyberspace
- Kiesler Sara, Regulating behavior in online communities. Building Successful Online Communities: Evidence-Based Social Design
- Lampe Cliff, Gamification and social media. The Gameful world: Approaches, issues, applications
- Lenhart Amanda, Digital Abuse, and Cyberstalking in America
- Marx Karl, 1848, The communist manifesto
- Rainie Lee, Trolls, Anonymity and Fake News Online
- Sproull Lee, 1992, Kiesler
- Star Susan Leigh, 1989, Griesemer
- U. S. Department of Labor. 2015., Current Population Survey: Detailed occupation by sex and race
- Walther Joseph B., 2002, Parks
Cited by
Starbird, K., Arif, A., & Wilson, T. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359229 · Cited by 370
Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator
Jhaver, S., Birman, I., Gilbert, E., & Bruckman, A. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3338243 · Cited by 289
“found that using automated approaches to identify abusive language can result in situations where the concerns of only the dominant groups are emphasized and existing structural inequalities are exacerbated [8].”
BackgroundOnline Harassment and Content Moderation: The Case of Blocklists
Jhaver, S., Ghoshal, S., Bruckman, A., & Gilbert, E. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3185593 · Cited by 270
“argued that public demonstrations of support not only provide validation for targets of harassment, but also create powerful descriptive norms that help other users determine what behaviors are and are not appropriate in an online community [5].”
BackgroundScheuerman, M. K., Paul, J. M., & Brubaker, J. R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359246 · Cited by 252
“For example, Blackwell et al. describe the marginalization of users whose experiences with harassment are invalidated when they meet rigid classifications [16].”
BackgroundScheuerman, M. K., Branham, S. M., & Hamidi, F. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274424 · Cited by 219
Metadata
- Authors
- Blackwell, L., Dimond, J., Schoenebeck, S., & Lampe, C.
- Affiliations
- University of Michigan (US)
- Sassafras Tech Collective, Ann Arbor, MI, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2017-12-06
- Volume
- 1
- Issue
- CSCW
- Pages
- 1-19
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2775336929
- Cited by
- 245
- References
- 35
- Citations by year
- 2026: 15 · 2025: 41 · 2024: 34 · 2023: 32 · 2022: 31 · 2021: 34 · 2020: 22 · 2019: 22 · 2018: 14
- FWCI
- 9.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, public-domain)
- Funders
- Knight Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- online harassment, automated content moderation, platform policies, intersectionality, feminist theory, platform design, vulnerable users, community norms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.