Classification and Its Consequences for Online Harassment: Design Insights from HeartMob
The DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index.
Abstract
Online harassment is a pervasive and pernicious problem. Techniques like natural language processing and machine learning are promising approaches for identifying abusive language, but they fail to address structural power imbalances perpetuated by automated labeling and classification. Similarly, platform policies and reporting tools are designed for a seemingly homogenous user base and do not account for individual experiences and systems of social oppression. This paper describes the design and evaluation of HeartMob, a platform built by and for people who are disproportionately affected by the most severe forms of online harassment. We conducted interviews with 18 HeartMob users, both targets and supporters, about their harassment experiences and their use of the site. We examine systems of classification enacted by technical systems, platform policies, and users to demonstrate how 1) labeling serves to validate (or invalidate) harassment experiences; 2) labeling motivates bystanders to provide support; and 3) labeling content as harassment is critical for surfacing community norms around appropriate user behavior. We discuss these results through the lens of Bowker and Star's classification theories and describe implications for labeling and classifying online abuse. Finally, informed by intersectional feminist theory, we argue that fully addressing online harassment requires the ongoing integration of vulnerable users' needs into the design and moderation of online platforms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [10]
Citations
Cites
Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color
Crenshaw, K. (1991) · Stanford Law Review · doi:10.2307/1229039 · Cited by 32,792 · cited 3×
“As a result, there are severe gaps in legal protections for women of color, which white women do not experience [16].”
BackgroundInfluentialA General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858 · cited 1×
“We used an inductive approach to develop codes [42].”
MethodStar, S. L., & Griesemer, J. R. (1989) · Social Studies of Science · doi:10.1177/030631289019003001 · Cited by 10,847
Sorting Things Out: Classification and Its Consequences
(2000) · The Journal of Academic Librarianship · doi:10.1016/s0099-1333(00)00155-5 · Cited by 5,210
Cyberbullying: Its nature and impact in secondary school pupils
Smith, P. K., Mahdavi, J., Carvalho, M., Fisher, S., Russell, S., & Tippett, N. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2007.01846.x · Cited by 3,639 · cited 1×
“One individual may also harass another, as is often the case in instances of cyberbullying [3,39] and non-consensual image sharing (also known as “revenge porn”), a form of doxing in which sexually explicit images or videos are distributed without their subject’s consent, often by a former romantic…”
BackgroundA Room with a Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels
Goldstein, N. J., Cialdini, R. B., & Griskevicius, V. (2008) · Journal of Consumer Research · doi:10.1086/586910 · Cited by 3,409
Ahmed, S. (2024) · Feminist Theory · doi:10.1007/978-3-031-55397-4_18 · Cited by 2,722
Ahmed, S. (2017) · doi:10.1215/9780822373377 · Cited by 2,722 · cited 2×
“Intersectional feminist theory [1,15,25] holds that various identities (such as race, gender, class, sexuality, religion, disability, and nationality) are inextricably bound in systems of entrenched structural oppression (such as racism, sexism, cissexism, classism, heterosexism, ableism, and colonialism).”
BackgroundRethinking Standpoint Epistemology: What Is ‘Strong Objectivity’?
Harding, S. (1996) · Feminism And Science · doi:10.1093/oso/9780198751458.003.0016 · Cited by 1,955
Unpacking "privacy" for a networked world
Palen, L., & Dourish, P. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642635 · Cited by 965 · cited 1×
“A digital audience is “large, unknown and distant” [34], and users must continually negotiate their own privacy and impression management practices to satisfy both their own expectations and those of a functionally invisible audience [31,34,44].”
Background
Research works printed but not linked: 13
- Becker Howard, 1982, The Free Press 9
- Geoffrey, 2000, Bowker and Susan Leigh Star
- Citron Danielle Keats, Criminalizing Revenge Porn. Social Science Research Network
- Donath Judith, Identity and Deception in the Virtual Community. Communities in cyberspace
- Kiesler Sara, Regulating behavior in online communities. Building Successful Online Communities: Evidence-Based Social Design
- Lampe Cliff, Gamification and social media. The Gameful world: Approaches, issues, applications
- Lenhart Amanda, Digital Abuse, and Cyberstalking in America
- Marx Karl, 1848, The communist manifesto
- Rainie Lee, Trolls, Anonymity and Fake News Online
- Sproull Lee, 1992, Kiesler
- Star Susan Leigh, 1989, Griesemer
- U. S. Department of Labor. 2015., Current Population Survey: Detailed occupation by sex and race
- Walther Joseph B., 2002, Parks
Cited by
Starbird, K., Arif, A., & Wilson, T. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359229 · Cited by 370
Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator
Jhaver, S., Birman, I., Gilbert, E., & Bruckman, A. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3338243 · Cited by 289
“found that using automated approaches to identify abusive language can result in situations where the concerns of only the dominant groups are emphasized and existing structural inequalities are exacerbated [8].”
BackgroundOnline Harassment and Content Moderation: The Case of Blocklists
Jhaver, S., Ghoshal, S., Bruckman, A., & Gilbert, E. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3185593 · Cited by 270
“argued that public demonstrations of support not only provide validation for targets of harassment, but also create powerful descriptive norms that help other users determine what behaviors are and are not appropriate in an online community [5].”
BackgroundScheuerman, M. K., Paul, J. M., & Brubaker, J. R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359246 · Cited by 252
“For example, Blackwell et al. describe the marginalization of users whose experiences with harassment are invalidated when they meet rigid classifications [16].”
BackgroundScheuerman, M. K., Branham, S. M., & Hamidi, F. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274424 · Cited by 219
For You, or For"You"?: Everyday LGBTQ+ Encounters with TikTok
Simpson, E., & Semaan, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432951 · Cited by 230
“While scholars have shown that people use social media and online community spaces to reflect upon and negotiate their identities [44, 116, 117], as well as to seek out and provide social support [31, 96], people with marginalized identities continue to experience exclusion [110, 133] and harassment [14] in digital sp…”
BackgroundHarassment in Social Virtual Reality: Challenges for Platform Governance
Blackwell, L., Ellison, N., Elliott-Deflo, N., & Schwartz, R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359202 · Cited by 218
“Users’ subjective definitions for what constitutes online harassment in social VR make governing these spaces at the application- or platform-level challenging—a finding consistent with research on social media users’ definitions and perceptions of online harassment [6,19,20,35].”
BackgroundResultInfluentialSelf-citationWohn, D. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300390 · Cited by 191
Morally Motivated Networked Harassment as Normative Reinforcement
Marwick, A. E. (2021) · Social Media + Society · doi:10.1177/20563051211021378 · Cited by 185
“Previous research indicates that online harassment has significant negative impacts on people’s lives (Blackwell et al., 2017; Sobieraj, 2020), including emotional and psychological difficulties (Duggan, 2017).”
Background"They Don't Leave Us Alone Anywhere We Go": Gender and Digital Abuse in South Asia
Sambasivan, N., Batool, A., Ahmed, N., Matthews, T., Thomas, K., Gaytán-Lugo, L. S., Nemer, D., Bursztein, E., Churchill, E., & Consolvo, S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300232 · Cited by 175
Prevalence and Psychological Effects of Hateful Speech in Online College Communities
Saha, K., Chandrasekharan, E., & De Choudhury, M. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326032 · Cited by 170
“Other research has also studied moderation of online antisocial behaviors like undesirable posting [15, 21] and online abuse [11, 18, 43].”
BackgroundEvaluating the Effectiveness of Deplatforming as a Moderation Strategy on Twitter
Jhaver, S., Boylston, C., Yang, D., & Bruckman, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479525 · Cited by 169
“Online communities experience a wide variety of toxic behaviors, such as incivility [14], harassment [13, 24, 54], trolling [22] and cyberbullying [64].”
BackgroundDisturbing the Peace: Experiencing and Mitigating Emerging Harassment in Social Virtual Reality
Freeman, G., Zamanifard, S., Maloney, D., & Acena, D. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512932 · Cited by 162
“Examples include reporting mechanisms through which a user can describe another user as a harasser [2, 11, 26, 86, 88], algorithms that detect certain types of harassing language and text [25], Twitter’s block list [41], participatory design for cyberbullying mitigation and prevention solutions [3, 66], automatic fram…”
BackgroundInfluentialSoK: Hate, Harassment, and the Changing Landscape of Online Abuse
Thomas, K., Akhawe, D., Bailey, M., Boneh, D., Bursztein, E., Consolvo, S., Dell, N., Durumeric, Z., Kelley, P. G., Kumar, D., et al. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00028 · Cited by 148
“HeartMob provided an interface to report hate and harassment to bystanders for confirmation and emotional support [13].”
BackgroundCrossmod: A Cross-Community Learning-based System to Assist Reddit Moderators
Chandrasekharan, E., Gandhi, C., Mustelier, M. W., & Gilbert, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359276 · Cited by 151
Seering, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415178 · Cited by 147
“Blackwell et al. [7] detailed the consequences of this sort of classification – while it can validate users’ experiences by making norms clear, it can also invalidate the experiences of users whose experiences do not match the classification scheme.”
BackgroundToxic Behaviors in Team-Based Competitive Gaming: The Case of League of Legends
Kou, Y. (2020) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3410404.3414243 · Cited by 210
“Typical instances of rules include end user license agreements, codes of conduct, and classification of toxic behavior [10].”
BackgroundHuman-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
“3 However, researchers increasingly recognize the pitfalls of full automation: 1) models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68, 73], potentially exacerbating structural inequalities [10]; 2) there exist diverse rules and preferences of austerity an…”
BackgroundJiang, S., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274351 · Cited by 125
“In keeping with the norms established by prior work [12, 16], we decide not to moderate these words.”
A Just and Comprehensive Strategy for Using NLP to Address Online Abuse
Jurgens, D., Hemphill, L., & Chandrasekharan, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1357 · Cited by 121
“These categories of abuse and the importance of community norms have been noted elsewhere (Liu et al., 2018; Guberman and Hemphill, 2017; Salminen et al., 2018; Blackwell et al., 2017) but have not yet received the same level of attention in NLP.”
BackgroundDo Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incels
Horta Ribeiro, M., Jhaver, S., Zannettou, S., Blackburn, J., Stringhini, G., De Cristofaro, E., & West, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476057 · Cited by 124
“Internet platforms experience a myriad of toxic behaviors such as incivility [4], harassment [3, 23], trolling [9] and cyberbullying [26].”
BackgroundThe Politics of Privacy Theories: Moving from Norms to Vulnerabilities
McDonald, N., & Forte, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376167 · Cited by 118
“As HCI and adjacent research communities move towards greater inclusion of feminist intersectional approaches [11, 32, 58, 90, 94, 98, 111], we propose updating our conceptual and pragmatic approaches to studying privacy to integrate intersectional and queer-Marxist theories.”
BackgroundYes: Affirmative Consent as a Theoretical Framework for Understanding and Imagining Social Platforms
Im, J., Dimond, J., Berton, M., Lee, U., Mustelier, K., Ackerman, M. S., & Gilbert, E. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445778 · Cited by 119
Self-citationPrefigurative Design as a Method for Research Justice
Asad, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359302 · Cited by 115
A framework of severity for harmful content online
Scheuerman, M. K., Jiang, J. A., Fiesler, C., & Brubaker, J. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479512 · Cited by 114
Drawing from justice theories to support targets of online harassment
Schoenebeck, S., Haimson, O. L., & Nakamura, L. (2021) · New Media & Society · doi:10.1177/1461444820913122 · Cited by 84
“Social media sites evaluate whether content was in violation of community guidelines in isolation, which prevents them from recognizing sustained harassment over time (Blackwell et al., 2017; Duggan, 2017; Massanari, 2017).”
BackgroundSelf-citationModeration Challenges in Voice-based Online Communities on Discord
Jiang, J. A., Kiene, C., Middler, S., Brubaker, J. R., & Fiesler, C. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359157 · Cited by 106
Do Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incels
Ribeiro, M. H., Jhaver, S., Zannettou, S., Blackburn, J., De Cristofaro, E., Stringhini, G., & West, R. (2020) · arXiv · doi:10.48550/arxiv.2010.10397 · Cited by 102
“Internet platforms experience a myriad of toxic behaviors such as incivility [7], harassment [6, 21], trolling [9] and cyberbullying [24].”
BackgroundWong-Villacres, M., Kumar, A., Vishwanath, A., Karusala, N., DiSalvo, B., & Kumar, N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196794 · Cited by 98
“Also, the creation of a system of categories can be the outcome of participatory design processes that acknowledge individual experiences and systems of social oppression [9].”
BackgroundDigital Juries: A Civics-Oriented Approach to Platform Governance
Fan, J., & Zhang, A. X. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376293 · Cited by 89
Freed, D., Havron, S., Tseng, E., Gallardo, A., Chatterjee, R., Ristenpart, T., & Dell, N. L. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359304 · Cited by 123
Kou, Y., & Gui, X. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445279 · Cited by 119
Michie, L., Balaam, M., McCarthy, J., Osadchiy, T., & Morrissey, K. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173931 · Cited by 106
“However, the online platform, HeartMob, takes a community-based, human-led approach to both identifying and responding to online harassment [15].”
Background"'More gay' fits in better": Intracommunity Power Dynamics and Harms in Online LGBTQ+ Spaces
Walker, A. M., & DeVito, M. A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376497 · Cited by 80
“In cases such as outright harassment, this makes sense, as these conflicts are typically one-sided and specifically intended to harm a specific target [5, 68, 78].”
InfluentialGilbert, S. A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392822 · Cited by 80
When Online Harassment Is Perceived as Justified
Blackwell, L., Chen, T., Schoenebeck, S., & Lampe, C. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15036 · Cited by 79
Self-citationSquadbox: A Tool to Combat Email Harassment Using Friendsourced Moderation
Mahar, K., Zhang, A. X., & Karger, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174160 · Cited by 80
“A final population is volunteer moderators, much like the vetted community within HeartMob [4].”
BackgroundInfluentialConformity of Eating Disorders through Content Moderation
Feuston, J. L., Taylor, A. S., & Piper, A. M. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392845 · Cited by 79
A Trade-off-centered Framework of Content Moderation
Jiang, J. A., Nie, P., Brubaker, J. R., & Fiesler, C. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534929 · Cited by 126
“The realization of the needs ofmultiple stakeholders has promptedmany studies to call out against a one-size-fits-all approach to content moderation [6, 31, 53, 93].”
BackgroundRestricted modes: Social media, content classification and LGBTQ sexual citizenship
Southerton, C., Marshall, D., Aggleton, P., Rasmussen, M. L., & Cover, R. (2020) · New Media & Society · doi:10.1177/1461444820904362 · Cited by 107
“These moral values frequently reinforce existing inequalities and power relations and as such classification systems tend to neglect marginalised groups (Blackwell et al., 2017).”
BackgroundSchulenberg, K., Freeman, G., Li, L., & Barwulor, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610027 · Cited by 97
“Unfortunately, harassment of women in online spaces still remains a salient and pervasive issue in today’s world, and is undoubtedly reflective of the systems of offline structural oppression that seek to control women’s bodies and rights that plague societies globally today [5] (e.”
BackgroundSoK: A Framework for Unifying At-Risk User Research
Warford, N., Matthews, T., Yang, K., Akgul, O., Consolvo, S., Kelley, P. G., Malkin, N., Mazurek, M. L., Sleeper, M., & Thomas, K. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833643 · Cited by 62
“For example, some activists are surveilled by government actors due to their work [59, 90]; people who are LGBTQ+ face elevated risk of harassment by anonymous attackers on social media [15, 16]; and women in repressive regions experience pervasive sexual harassment online and sometimes severe consequences from their…”
BackgroundInfluentialDecolonizing Tactics as Collective Resilience: Identity Work of AAPI Communities on Reddit
Dosono, B., & Semaan, B. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392881 · Cited by 65
(2022) · doi:10.2196/30286 · Cited by 65
People Who Can Take It: How Women Wikipedians Negotiate and Navigate Safety
Menking, A., Erickson, I., & Pratt, W. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300702 · Cited by 68
“Both of these studies shift our attention from asking questions about what motivates online harassment [10] or how we might categorize it [7] to what we can learn from how users manage their safety and socially construct safe spaces (see [82]).”
“Disadvantaged in the American-dominated Internet”: Sex, Work, and Technology
Barwulor, C., McDonald, A., Hargittai, E., & Redmiles, E. M. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445378 · Cited by 82
“We recognize the importance of our position as scholars in relation to this research [6, 9, 26, 73] and thus describe our identities, their alignment with those of our participants, and how our identities may create limitations in this work.”
Thomas, K., Kelley, P. G., Consolvo, S., Samermit, P., & Bursztein, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501879 · Cited by 113
"They Just Don't Get It": Towards Social Technologies for Coping with Interpersonal Racism
To, A., Sweeney, W., Hammer, J., & Kaufman, G. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392828 · Cited by 64
Everyday Experiences: Small Stories and Mental Illness on Instagram
Feuston, J. L., & Piper, A. M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300495 · Cited by 63
‘Unmochon’: A Tool to Combat Online Sexual Harassment over Facebook Messenger
Sultana, S., Deb, M., Bhattacharjee, A., Hasan, S., Alam, S. M. R., Chakraborty, T., Roy, P., Ahmed, S. F., Moitra, A., Amin, M. A., et al. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445154 · Cited by 85
“Along with all of the above-mentioned concerns and limitations, the existing state-of-the-art tools and mechanisms for fighting online harassment often fail to address culturally contextual gendered needs in such cases of harassment, including users’ vulnerability due to their gender identity and a balance between the…”
BackgroundMethodInfluentialWomen's Perspectives on Harm and Justice after Online Harassment
Im, J., Schoenebeck, S. Y., Iriarte, M. P., Grill, G., Wilkinson, D., Batool, A., Alharbi, R., Funwie, A., Gankhuu, T., Gilbert, É., et al. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555775 · Cited by 94
Self-citationModular Politics: Toward a Governance Layer for Online Communities
Schneider, N., Filippi, P. D., Frey, S., Tan, J., & Zhang, A. X. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449090 · Cited by 105
“ance designs to dissuade norm violators [92] and protect against harassers [46, 86], as well as to facilitate reconciliation between conflicted members [115] and provide community support for victims [10]. Self-organization is also evident in the ability of communities to develop and sustain widely held informal norm…”
BackgroundHarassment Experiences of Women and LGBTQ Live Streamers and How They Handled Negativity
Uttarapong, J., Cai, J., & Wohn, D. Y. (2021) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3452918.3458794 · Cited by 86
SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice
Singhal, M., Ling, C., Paudel, P., Thota, P., Kumarswamy, N., Stringhini, G., & Nilizadeh, S. (2023) · 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp57164.2023.00056 · Cited by 57
Mediating Community-AI Interaction through Situated Explanation: The Case of AI-Led Moderation
Kou, Y., & Gui, X. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415173 · Cited by 54
“, [6,13,14]).”
Fischer, C., Fishman, B., & Schoenebeck, S. Y. (2019) · AERA Open · doi:10.1177/2332858419894252 · Cited by 69
“Finally, the prevalence of harassment and misinformation may disproportionately affect people from marginalized identities (Blackwell et al., 2017; Pew Research Center, 2017b; Starbird et al., 2014).”
BackgroundSelf-citationHarassment in Social VR: Implications for Design
Blackwell, L., Ellison, N., Elliott-Deflo, N., & Schwartz, R. (2019) · 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2019.8798165 · Cited by 53
“Online harassment refers to a broad spectrum of abusive behaviors enabled by technology platforms [1].”
BackgroundSelf-citationReconsidering Tweets: Intervening during Tweet Creation Decreases Offensive Content
Katsaros, M., Yang, K., & Fratamico, L. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19308 · Cited by 87
Feeling Good and In Control: In-game Tools to Support Targets of Toxicity
Reid, E., Mandryk, R. L., Beres, N. A., Klarkowski, M., & Frommel, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3549498 · Cited by 60
Privacy, Patriarchy, and Participation on Social Media
Karusala, N., Bhalla, A., & Kumar, N. (2019) · Designing Interactive Systems Conference · doi:10.1145/3322276.3322355 · Cited by 74
Tackling Online Abuse: A Survey of Automated Abuse Detection MethodsPreprint
Mishra, P., Yannakoudakis, H., & Shutova, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06024 · Cited by 107
“This points to a critical issue: automated systems embody the morals and values of their creators and annotators (Bowker and Star, 2000; Blackwell et al., 2017).”
BackgroundInfluentialFox, S., Menking, A., Eschler, J., & Backonja, U. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3397178 · Cited by 63
Xiao, S., Cheshire, C., & Salehi, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517614 · Cited by 50
“Research has found that the current form of content moderation leaves victims out of the decision making process [49] and fails to adapt to their individual experiences [5].”
BackgroundInfluentialSun, Q., Wojcieszak, M., & Davidson, S. R. (2021) · Frontiers in Political Science · doi:10.3389/fpos.2021.741605 · Cited by 59
“Such policies serve to classify and regulate inappropriate behaviors and content and may lead to an increase or decrease in incivility (Blackwell et al., 2017).”
BackgroundHate Raids on Twitch: Echoes of the Past, New Modalities, and Implications for Platform Governance
Han, C., Seering, J., Kumar, D., Hancock, J. T., & Durumeric, Z. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579609 · Cited by 46
Towards Leveraging AI-based Moderation to Address Emergent Harassment in Social Virtual Reality
Schulenberg, K., Li, L., Freeman, G., Zamanifard, S., & McNeese, N. J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581090 · Cited by 85
Punishment and Its Discontents: An Analysis of Permanent Ban in an Online Game Community
Kou, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476075 · Cited by 66
Goyal, N., Park, L., & Vasserman, L. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517517 · Cited by 63
“Online harassment is a broad and expansive field, as online harassment takes many forms and definitions [5, 6, 9, 16, 29, 61].”
BackgroundInfluentialNamvarpour, M. (., Pauwels, H., & Razi, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757548 · Cited by 45
“Gender harassment, involving derogatory comments and sexist jokes that create a hostile online environment, and doxing, the public release of personal information to harm or harass, and peer pressure are also prevalent [9, 12, 32].”
Kou, Y., & Gui, X. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595960 · Cited by 42
Heung, S., Jiang, L., Azenkot, S., & Vashistha, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641949 · Cited by 75
“However, when content reported by users was not deemed to violate community guidelines, this invalidated the harassment experience and increased overall frustration [2, 4].”
BackgroundSultana, S., & Fussell, S. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479580 · Cited by 50
“This type of collaborative reporting and crosschecking are widely theorized and practiced in designing culturally contextual gender justice tools [4, 21, 41, 143], where support groups with knowledge of cultural values collaborate to understand harassment claims and offer online and”
BackgroundFragile Masculinity: Men, Gender, and Online Harassment
Rubin, J. D., Blackwell, L., & Conley, T. D. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376645 · Cited by 74
Self-citationDoerfler, P., Forte, A., Cristofaro, E. D., Stringhini, G., Blackburn, J., & McCoy, D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476082 · Cited by 69
“Finally, the creation of a peer-to-peer network, such as Heartmob [11], specifically for researchers, would be helpful here as well, as many individuals expressed that kindness from colleagues was a primary mechanism of support for them.”
BackgroundInfluentialNothing Micro About It: Examining Ableist Microaggressions on Social Media
Heung, S., Phutane, M., Azenkot, S., Marathe, M., & Vashistha, A. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544801 · Cited by 49
“Prior work on social media moderation documented flaws within current online moderation systems, such as unfair and false reporting [22, 27, 46], mislabelling harassment [7], and lack of transparency in the harassment reporting process [7].”
BackgroundCarton, S., Mei, Q., & Resnick, P. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1386 · Cited by 48
“However, the idea of fully-automated moderation by machine learning has attracted criticism as being subject to bias, inaccuracy, manipulation and frustration on the user end (Pavlopoulos et al., 2017; Binns et al., 2017; Blackwell et al., 2018; Hosseini et al., 2017; Adams and Dixon, 2017).”
BackgroundRashidi, Y., Kapadia, A. C., Nippert-Eng, C. E., & Su, N. M. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392827 · Cited by 53
Cai, J., & Wohn, D. Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479554 · Cited by 46
“All interviews were audio-recorded, transcribed by speech recognition software 3 , and then double-checked by the researchers.”
BackgroundMethodPrivacy and Vulnerable Populations
McDonald, N., & Forte, A. (2022) · Springer eBooks · doi:10.1007/978-3-030-82786-1_15 · Cited by 43
“[107] argued for the relevance of feminist intersectional theory in thinking about HeartMob, a platform where victims of harassment can describe their experience by submitting a harassment case and then request help from volunteers.”
BackgroundOnline Harassment in Majority Contexts: Examining Harms and Remedies across Countries
Schoenebeck, S. Y., Batool, A., Do, G., Darling, S., Grill, G., Wilkinson, D., Khan, M., Toyama, K., & Ashwell, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581020 · Cited by 53
Self-citationLearning to Airbnb by Engaging in Online Communities of Practice
Holikatti, M., Jhaver, S., & Kumar, N. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359330 · Cited by 35
Ruddit: Norms of Offensiveness for English Reddit Comments
Hada, R., Sudhir, S., Mishra, P., Yannakoudakis, H., Mohammad, S. M., & Shutova, E. (2021) · doi:10.18653/v1/2021.acl-long.210 · Cited by 57
“It is important to include the opinions of targeted minorities and marginalized groups when dealing with the annotation of offensive language (Kiritchenko and Nejadgholi, 2020; Blackwell et al., 2017).”
BackgroundConversations About Crime: Re-Enforcing and Fighting Against Platformed Racism on Reddit
Wu, Q., Williams, L. K., Simpson, E., & Semaan, B. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512901 · Cited by 35
Gender‐based cyberbullying: Understanding expected bystander behavior online
Herry, E., & Mulvey, K. L. (2022) · Journal of Social Issues · doi:10.1111/josi.12503 · Cited by 38
Dirty Data Labeled Dirt Cheap: Epistemic Injustice in Machine Learning Systems
Hull, G. (2022) · Ethics and Information Technology · doi:10.1007/s10676-023-09712-y · Cited by 35
Squadbox: A Tool To Combat Online Harassment Using Friendsourced Moderation
Mahar, K., Zhang, A. X., & Karger, D. (2018) · Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3170427.3186498 · Cited by 28
“These tools include Heartmob [1], which provides a volunteer support network, and collaborative blocklist tools like BlockTogether [4].”
MethodSomeone Is Wrong on the Internet: Having Hard Conversations in Online Spaces
Baughan, A., Petelka, J., Yoo, C. J., Lo, J. L., Wang, S., Paramasivam, A., Zhou, A., & Hiniker, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449230 · Cited by 39
“A large body of prior work has investigated online arguments, including how social behavior changes online [24, 33, 36, 49, 57], how social media design affects online arguments [18, 27, 37, 45, 58], and how various interventions can facilitate productive online discussions [4, 6, 7, 10, 29, 30, 39, 48, 52].”
BackgroundPeng, Z., Ma, X., Yang, D., Tsang, K. W., & Guo, Q. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445446 · Cited by 36
“However, OMHCs should not force commenters to change their messages if they do not feel like to since it may silence them [7].”
BackgroundDo Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incels
Ribeiro, M. H., Jhaver, S., Zannettou, S., Blackburn, J., Stringhini, G., Cristofaro, E. D., & West, R. (2021) · UCL Discovery (University College London) · Cited by 37
StoryChat: Designing a Narrative-Based Viewer Participation Tool for Live Streaming Chatrooms
Yen, R., Feng, L., Mehra, B., Pang, C. C., Hu, S., & Lu, Z. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580912 · Cited by 39
“The extension of this work applied narratives to understanding online harassment [9, 76] and found that participants draw comfort from sharing their experiences in the narrative.”
BackgroundCombating Toxicity, Harassment, and Abuse in Online Social Spaces: A Workshop at CHI 2023
Mandryk, R. L., Frommel, J., Goyal, N., Freeman, G., Lampe, C. A. C., Vieweg, S. E., & Wohn, D. Y. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3573793 · Cited by 39
“, women, LGBTQ+, people of color, and younger individuals—continue to be more likely to be targets of online harassment [3, 50].”
BackgroundSelf-citationTransparency, Fairness, and Coping: How Players Experience Moderation in Multiplayer Online Games
Ma, R., Li, Y., & Kou, Y. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581097 · Cited by 42
“However, many researchers have been concerned that retributive justice might not be the only effective justice standard for adjudicating moderation cases (e.g., [8, 56]).”
BackgroundMapping the social implications of platform algorithms for LGBTQ+ communities
Myles, D., Duguay, S., & Echaiz, L. F. (2023) · Journal of Digital Social Research · doi:10.33621/jdsr.v5i4.162 · Cited by 43
“…industry imperatives, rely on binary classification systems that reproduce cisheteronormative assumptions about gender and sexual orientation (Bivens & Haimson, 2016; Lingel & Golub, 2015) that can further lead to online harassment against LGBTQ+ users (Albury et al., 2020; Blackwell et al., 2017).”
BackgroundLyu, Y., Cai, J., Callis, A., Cotter, K., & Carroll, J. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642148 · Cited by 48
“Algorith-mic moderation is often biased to the detriment of marginalized groups [6, 29] and disproportionately produces false positives for marginalized groups [31], such as YouTube’s demonetization of videos featuring LGBTQ+ issues [46].”
BackgroundAdversarial Text Generation for Google's Perspective API
Jain, E., Brown, S., Chen, J., Neaton, E., Baidas, M., Dong, Z., Gu, H., & Artan, N. S. (2018) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci46756.2018.00220 · Cited by 35
“Labeling online abuse and harassment can validate harassment experiences, motivate bystanders to provide support, and helps normalize appropriate user behavior [1].”
BackgroundRabaan, H., & Dombrowski, L. S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580648 · Cited by 39
Schulenberg, K., Freeman, G., Li, L., & Knijnenburg, B. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686923 · Cited by 29
Inclusion at Scale: Deploying a Community-Driven Moderation Intervention on Twitch
Brewer, J., Romine, M., & Taylor, T. L. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395514 · Cited by 35
Opinion Conflicts: An Effective Route to Detect Incivility in Twitter
Maity, S., Chakraborty, A., Goyal, P., & Mukherjee, A. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274386 · Cited by 60
“This may directly enhance civility in the platform making account holders less likely to be invective (similar observations in the context of trolling and online harassment has been made in earlier works [8, 23]).”
Gender and Technology: A rights-based and intersectional analysis of key trends
Ceia, V., Nothwehr, B., & Wagner, L. (2021) · doi:10.21201/2021.7598 · Cited by 39
“…violence are often not well understood, are considered ineffective or overly time consuming, do not consider local contexts, resort to victim blaming, or focus on mitigating vulnerability to harassment rather than tackling harassment at the source (Blackwell et al. 2017; Kennedy et al. 2017).”
Background
Metadata
- Authors
- Blackwell, L., Dimond, J., Schoenebeck, S., & Lampe, C.
- Affiliations
- University of Michigan (US)
- Sassafras Tech Collective, Ann Arbor, MI, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2017-12-06
- Volume
- 1
- Issue
- CSCW
- Pages
- 1-19
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2775336929
- Cited by
- 245
- References
- 35
- Citations by year
- 2026: 15 · 2025: 41 · 2024: 34 · 2023: 32 · 2022: 31 · 2021: 34 · 2020: 22 · 2019: 22 · 2018: 14
- FWCI
- 9.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, public-domain)
- Funders
- Knight Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- online harassment, automated content moderation, platform policies, intersectionality, feminist theory, platform design, vulnerable users, community norms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.