Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator
Open accessThe DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
Abstract
What one may say on the internet is increasingly controlled by a mix of automated programs, and decisions made by paid and volunteer human moderators. On the popular social media site Reddit, moderators heavily rely on a configurable, automated program called “Automoderator” (or “Automod”). How do moderators use Automod? What advantages and challenges does the use of Automod present? We participated as Reddit moderators for over a year, and conducted interviews with 16 moderators to understand the use of Automod in the context of the sociotechnical system of Reddit. Our findings suggest a need for audit tools to help tune the performance of automated mechanisms, a repository for sharing tools, and improving the division of labor between human and machine decision making. We offer insights that are relevant to multiple stakeholders—creators of platforms, designers of automated regulation systems, scholars of platform governance, and content moderators.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Notably, the task for humans to create conditional delegation rules share some similarity with what many social media moderators are already doing by writing manual automation rules to deal with the massive amount of comments (Fig. 1(C)).”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [44]
The cited source says what the claim attributes to it.
In the citing paper
“For example, moderators on Reddit use a tool called AutoModerator, with which they manually customize a rule-based system to automatically identify comments for deleting or reporting for further review [18,44].”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [44]
The cited source says what the claim attributes to it.
In the citing paper
“However, there exists a need for audit tools to monitor the performance of the keyword rules.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [44]
In the citing paper
“They also highlight the fact that AutoModerator fundamentally changes the work of moderators and may introduce additional unnecessary work.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [44]
In the citing paper
“Through interviews with 16 moderators, Jhaver et al. [44] found that AutoModerator improves the efciency of moderation.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [44]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1990) · Cited by 24,217
Qualitative evaluation and research methods
Ellis, D. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,943
Basics of Qualitative Research
Ozanne, J. L., Strauss, A., & Corbin, J. (1992) · Journal of Marketing Research · doi:10.1177/002224379202900312 · Cited by 17,867
Basics of Qualitative Research
Ozanne, J. L., Strauss, A., & Corbin, J. (1992) · Journal of Marketing Research · doi:10.2307/3172751 · Cited by 17,850
The Managed Heart: Commercialization of Human Feeling
Tropman, J. E., & Hochschild, A. R. (1984) · Journal of Policy Analysis and Management · doi:10.2307/3324333 · Cited by 8,746
An Introduction to Qualitative Research
Flick, U. (2023) · SAGE Publications Ltd eBooks · doi:10.4135/9781036231712 · Cited by 8,198
Introduction to Qualitative Research Methods: A Guidebook and Resource
Taylor, S. J., & Bogdan, R. (2015) · Cited by 2,935
The New Governors: The People, Rules, and Processes Governing Online Speech
Klonick, K. (2018) · St. John's University School of Law (St. John's University) · Cited by 1,183
The Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility
Ackerman, M. S. (2000) · Human–Computer Interaction · doi:10.1207/s15327051hci1523_5 · Cited by 892
Programming Techniques: Regular expression search algorithm
Thompson, K. (1968) · Communications of the ACM · doi:10.1145/363347.363387 · Cited by 854
Research works printed but not linked: 6
- Kathy Charmaz . 2006. Coding in grounded theory practice. Constructing Grounded Theory ( 2006 ), 42--70. Sage Publications Ltd , Thousand Oaks, CA . Kathy Charmaz. 2006. Coding in grounded theory practice. Constructing Grounded Theory (2006), 42--70. Sage Publications Ltd, Thousand Oaks, CA.
- Susan C. Herring . 2000 . Gender differences in CMC: Findings and implications . Computer Professionals for Social Responsibility Journal 18 , 1 (2000), 1 -- 9 . Susan C. Herring. 2000. Gender differences in CMC: Findings and implications. Computer Professionals for Social Responsibility Journal 18, 1 (2000), 1--9.
- Ling Jiang and Eui-Hong Han . 2019 . ModBot: Automatic comments moderation . In Proceedings of the Computation+ Journalism Symposium. Ling Jiang and Eui-Hong Han. 2019. ModBot: Automatic comments moderation. In Proceedings of the Computation+ Journalism Symposium.
- 10.5555/223826.223831 · doi:10.5555/223826.223831
- Nathan J. Matias . 2016 . The civic labor of online moderators . In Internet Politics and Policy conference . Oxford, United Kingdom. Nathan J. Matias. 2016. The civic labor of online moderators. In Internet Politics and Policy conference. Oxford, United Kingdom.
- M. Pilar Opazo. 2010. Revitalizing the concept of sociotechnical systems in social studies of technology. (2010). Available at https://www.semanticscholar.org/paper/Revitalizing-the-Concept-of-Sociotechnical-Systems-Opazo/78460b4bdcbe004ccd469f4bda1f7d98017954de. M. Pilar Opazo. 2010. Revitalizing the concept of sociotechnical systems in social studies of technology. (2010). Available at https://www.semanticscholar.org/paper/Revitalizing-the-Concept-of-Sociotechnical-Systems-Opazo/78460b4bdcbe004ccd469f4bda1f7d98017954de.
Cited by
Studying Reddit: A Systematic Overview of Disciplines, Approaches, Methods, and Ethics
Proferes, N., Jones, N., Gilbert, S., Fiesler, C., & Zimmer, M. (2021) · Social Media + Society · doi:10.1177/20563051211019004 · Cited by 537
“For these reasons, Reddit has an ecosystem of bots created by its user base to help in several ways, such as content moderation (Jhaver et al., 2019), adding functionality through summarizing information and linking to other websites, or providing humor through parody bot accounts (Long et al.,…”
BackgroundGenerative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 311
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
Seering, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415178 · Cited by 147
Steiger, M., Bharucha, T. J., Venkatagiri, S., Riedl, M. J., & Lease, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445092 · Cited by 222
Xing, Y., Lv, C., Cao, D., & Hang, P. (2021) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2021.103199 · Cited by 209
Jhaver, S., Bruckman, A., & Gilbert, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359252 · Cited by 203
Self-citationA new sociology of humans and machines
Tsvetkova, M., Yasseri, T., Pescetelli, N., & Werner, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02001-8 · Cited by 187
Li, L., Fan, L., Atreja, S., & Hemphill, L. (2024) · ACM Transactions on the Web · doi:10.1145/3643829 · Cited by 102
Evaluating the Effectiveness of Deplatforming as a Moderation Strategy on Twitter
Jhaver, S., Boylston, C., Yang, D., & Bruckman, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479525 · Cited by 169
Self-citationJhaver, S., Zhang, A. Q., Chen, Q. Z., Natarajan, N., Wang, R., & Zhang, A. X. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610080 · Cited by 167
Self-citationCrossmod: A Cross-Community Learning-based System to Assist Reddit Moderators
Chandrasekharan, E., Gandhi, C., Mustelier, M. W., & Gilbert, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359276 · Cited by 151
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
LLM-Mod: Can Large Language Models Assist Content Moderation?
Kolla, M., Salunkhe, S., Chandrasekharan, E., & Saha, K. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650828 · Cited by 141
Quarantined! Examining the Effects of a Community-Wide Moderation Intervention on Reddit
Chandrasekharan, E., Jhaver, S., Bruckman, A., & Gilbert, E. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3490499 · Cited by 129
“It is vital that platforms maintain robust content moderation mechanisms to maintain their reputations and keep existing content usable [46].”
BackgroundSelf-citationA Trade-off-centered Framework of Content Moderation
Jiang, J. A., Nie, P., Brubaker, J. R., & Fiesler, C. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534929 · Cited by 126
Yes: Affirmative Consent as a Theoretical Framework for Understanding and Imagining Social Platforms
Im, J., Dimond, J., Berton, M., Lee, U., Mustelier, K., Ackerman, M. S., & Gilbert, E. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445778 · Cited by 119
Kou, Y., & Gui, X. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445279 · Cited by 119
A framework of severity for harmful content online
Scheuerman, M. K., Jiang, J. A., Fiesler, C., & Brubaker, J. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479512 · Cited by 114
Thomas, K., Kelley, P. G., Consolvo, S., Samermit, P., & Bursztein, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501879 · Cited by 113
Decentralizing Platform Power: A Design Space of Multi-Level Governance in Online Social Platforms
Jhaver, S., Frey, S., & Zhang, A. X. (2023) · Social Media + Society · doi:10.1177/20563051231207857 · Cited by 49
Self-citationAdmins, mods, and benevolent dictators for life: The implicit feudalism of online communities
Schneider, N. (2022) · New Media & Society · doi:10.1177/1461444820986553 · Cited by 78
Designing Word Filter Tools for Creator-led Comment Moderation
Jhaver, S., Chen, Q. Z., Knauss, D., & Zhang, A. X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517505 · Cited by 69
Self-citationThrough the Looking Glass: Study of Transparency in Reddit's Moderation Practices
Juneja, P., Rama Subramanian, D., & Mitra, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3375197 · Cited by 66
Modular Politics: Toward a Governance Layer for Online Communities
Schneider, N., Filippi, P. D., Frey, S., Tan, J., & Zhang, A. X. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449090 · Cited by 105
Ma, R., & Kou, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479573 · Cited by 102
Conformity of Eating Disorders through Content Moderation
Feuston, J. L., Taylor, A. S., & Piper, A. M. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392845 · Cited by 79
PolicyKit: Building Governance in Online Communities
Zhang, A. X., Hugh, G., & Bernstein, M. S. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415858 · Cited by 60
Seering, J., Kaufman, G. F., & Chancellor, S. (2020) · New Media & Society · doi:10.1177/1461444820964968 · Cited by 99
Social Norm Vulnerability and its Consequences for Privacy and Safety in an Online Community
Dym, B., & Fiesler, C. L. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415226 · Cited by 90
Automated Content Moderation Increases Adherence to Community Guidelines
Ribeiro, M. H., Cheng, J., & West, R. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583275 · Cited by 86
A new sociology of humans and machines
Tsvetkova, M., Yasseri, T., Pescetelli, N., & Werner, T. (2024) · arXiv · doi:10.48550/arxiv.2402.14410 · Cited by 83
Understanding Moderation in Online Mental Health Communities
Saha, K., Ernala, S. K., Dutta, S., Sharma, E., & Choudhury, M. D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49576-3_7 · Cited by 79
Kiene, C., Jiang, J. A., & Hill, B. M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359146 · Cited by 78
Wang, Q., Camacho, I., Jing, S., & Goel, A. K. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512977 · Cited by 47
Han, Z., Tu, Y., & Huang, C. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3257511 · Cited by 75
Schluger, C., Chang, J. P., Danescu-Niculescu-Mizil, C., & Levy, K. E. C. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555095 · Cited by 74
Mediating Community-AI Interaction through Situated Explanation: The Case of AI-Led Moderation
Kou, Y., & Gui, X. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415173 · Cited by 54
Ma, R., & Kou, Y. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555150 · Cited by 71
Hagos, D. H., & Rawat, D. B. (2022) · Sensors · doi:10.3390/s22249916 · Cited by 68
Schaffner, B., Bhagoji, A. N., Cheng, S., Mei, J., Shen, J., Wang, G., Chetty, M., Feamster, N., Lakier, G., & Tan, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642333 · Cited by 67
Punishment and Its Discontents: An Analysis of Permanent Ban in an Online Game Community
Kou, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476075 · Cited by 66
All That’s Happening behind the Scenes: Putting the Spotlight on Volunteer Moderator Labor in Reddit
Li, H., Hecht, B. J., & Chancellor, S. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19317 · Cited by 64
Gonçalves, J., Weber, I., Masullo, G. M., Torres da Silva, M., & Hofhuis, J. (2023) · New Media & Society · doi:10.1177/14614448211032310 · Cited by 44
Measuring the Monetary Value of Online Volunteer Work
Li, H., Hecht, B. J., & Chancellor, S. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19318 · Cited by 63
How Do Users Experience Moderation?: A Systematic Literature Review
Ma, R., You, Y., Gui, X., & Kou, Y. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610069 · Cited by 62
Wang, S. (2020) · Digital Journalism · doi:10.1080/21670811.2020.1851279 · Cited by 62
Consistent Estimators for Learning to Defer to an Expert
Mozannar, H., & Sontag, D. (2020) · arXiv · doi:10.48550/arxiv.2006.01862 · Cited by 61
“A current application of the latter setting is Facebook’s and other online platforms content moderation approach (Vincent, 2019; Jhaver et al., 2019): an algorithm is used to filter easily detectible inappropriate content and the rest of the examples are screened by a team of human moderators.”
Method"Learn the Facts about COVID-19": Analyzing the Use of Warning Labels on TikTok Videos
Ling, C., Gummadi, K. P., & Zannettou, S. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22168 · Cited by 61
Moderation as Empowerment: Creating and Managing Women-Only Digital Safe Spaces
Ammari, T., Nofal, M., Naseem, M., & Mustafa, M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555204 · Cited by 59
Chang, J. P., Schluger, C., & Danescu-Niculescu-Mizil, C. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555603 · Cited by 59
Governable Spaces: Democratic Design for Online Life
Schneider, N. (2024) · University of California Press eBooks · doi:10.1525/9780520393950 · Cited by 59
Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58
SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice
Singhal, M., Ling, C., Paudel, P., Thota, P., Kumarswamy, N., Stringhini, G., & Nilizadeh, S. (2023) · 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp57164.2023.00056 · Cited by 57
Political Bias and Factualness in News Sharing across more than 100,000 Online Communities
Weld, G., Glenski, M., & Althoff, T. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18104 · Cited by 58
Randazzo, C., & Ammari, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581453 · Cited by 55
Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia
Kuo, T.-S., Halfaker, A., Cheng, Z., Kim, J., Wu, M.-H., Wu, T., Holstein, K., & Zhu, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642278 · Cited by 54
Cai, J., Wohn, D. Y., & Almoqbel, M. Y. (2021) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3452918.3458796 · Cited by 54
Altruistic and Profit-oriented: Making Sense of Roles in Web3 Community from Airdrop Perspective
Fan, S., Min, T., Xiao, W., & Cai, W. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581173 · Cited by 52
Code of Conduct Conversations in Open Source Software Projects on Github
Li, R., Pandurangan, P., Frluckaj, H., & Dabbish, L. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449093 · Cited by 50
Bozarth, L., Im, J., Quarles, C. L., & Budak, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579631 · Cited by 50
SolutionChat: Real-time Moderator Support for Chat-based Structured Discussion
Lee, S.-C., Song, J., Ko, E.-Y., Park, S., Kim, J., & Kim, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376609 · Cited by 49
“For content moderation, algorithms are increasingly used to fully automate the moderation process [21] or collaborate with humans [20].”
MethodCai, J., & Wohn, D. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517628 · Cited by 49
Ozanne, M., Bhandari, A., Bazarova, N. N., & DiFranzo, D. J. (2022) · Big Data & Society · doi:10.1177/20539517221115666 · Cited by 49
Who Moderates on Twitch and What Do They Do? Quantifying Practices in Community Moderation on Twitch
Seering, J., & Kairam, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3567568 · Cited by 49
The Unsung Heroes of Facebook Groups Moderation: A Case Study of Moderation Practices and Tools
Kuo, T., Hernani, A., & Großklags, J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579530 · Cited by 49
Das, D., Guha, S., Brubaker, J. R., & Semaan, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642669 · Cited by 49
Wright, A. P., Shaikh, O., Park, H., Epperson, W., Ahmed, M., Pinel, S., Chau, D. H., & Yang, D. (2021) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3449280 · Cited by 35
From Scalability to Subsidiarity in Addressing Online Harm
Hasinoff, A. A., & Schneider, N. (2022) · Social Media + Society · doi:10.1177/20563051221126041 · Cited by 49
How humans impair automated deception detection performance
Kleinberg, B., & Verschuère, B. (2021) · Acta Psychologica · doi:10.1016/j.actpsy.2020.103250 · Cited by 46
Cai, J., & Wohn, D. Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479554 · Cited by 46
Content Moderation on Social Media in the EU: Insights From the DSA Transparency Database
Drolsbach, C., & Pröllochs, N. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651482 · Cited by 45
Garg, S., Taylor, J., Sherief, M. A. E., Kasson, E. M., Aledavood, T., Riordan, R., Kaiser, N., Cavazos‐Rehg, P. A., & Choudhury, M. D. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100467 · Cited by 44
Knowledge generation and sharing in online communities: current trends and future directions
Tausczik, Y., & Huang, X. (2020) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2020.04.009 · Cited by 43
Who Uses Bots? A Statistical Analysis of Bot Usage in Moderation Teams
Kiene, C., & Hill, B. M. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382960 · Cited by 43
Zhang, J. S., Keegan, B., Lv, Q., & Tan, C. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18112 · Cited by 43
Transparency, Fairness, and Coping: How Players Experience Moderation in Multiplayer Online Games
Ma, R., Li, Y., & Kou, Y. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581097 · Cited by 42
Do We Collaborate With What We Design?
Evans, K. D., Robbins, S., & Bryson, J. J. (2023) · Topics in Cognitive Science · doi:10.1111/tops.12682 · Cited by 42
AI Rules? Characterizing Reddit Community Policies Towards AI-Generated Content
Lloyd, T., Gosciak, J., Nguyen, T., & Naaman, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713292 · Cited by 42
Incels on Reddit: A study in social norms and decentralised moderation
Gillett, R., & Suzor, N. P. (2022) · First Monday · doi:10.5210/fm.v27i6.12575 · Cited by 42
ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates
Jiang, F., Xu, Z., Niu, L., Lin, B. Y., & Poovendran, R. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i26.34945 · Cited by 41
Plurals: A System for Guiding LLMs via Simulated Social Ensembles
Ashkinaze, J., Fry, E., Edara, N., Gilbert, É., & Budak, C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713675 · Cited by 41
Smith, C. E., Lane, W., Miller, H. J., Kluver, D., Terveen, L., & Yarosh, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479561 · Cited by 40
'There Has To Be a Lot That We're Missing': Moderating AI-Generated Content on Reddit
Lloyd, T., Reagle, J., & Naaman, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757445 · Cited by 40
StoryChat: Designing a Narrative-Based Viewer Participation Tool for Live Streaming Chatrooms
Yen, R., Feng, L., Mehra, B., Pang, C. C., Hu, S., & Lu, Z. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580912 · Cited by 39
Derivative Media: How Wall Street Devours Culture
deWaard, A. (2024) · University of California Press eBooks · doi:10.1525/luminos.197 · Cited by 39
Wright, L. (2022) · Social Media + Society · doi:10.1177/20563051221077020 · Cited by 38
Das, A., Dang, B., & Lease, M. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7461 · Cited by 37
The Effect of Bots on Human Interaction in Online Communities
Safadi, H., Lalor, J. P., & Berente, N. (2024) · MIS Quarterly · doi:10.25300/misq/2023/17901 · Cited by 35
Lambert, C., Choi, F., & Chandrasekharan, E. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686929 · Cited by 35
Learning to Airbnb by Engaging in Online Communities of Practice
Holikatti, M., Jhaver, S., & Kumar, N. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359330 · Cited by 35
Self-citationConversations About Crime: Re-Enforcing and Fighting Against Platformed Racism on Reddit
Wu, Q., Williams, L. K., Simpson, E., & Semaan, B. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512901 · Cited by 35
Toward a Participatory Digital Ethnography of Blockchain Governance
Rennie, E., Zargham, M., Tan, J., Miller, L., Abbott, J., Nabben, K., & De Filippi, P. (2022) · Qualitative Inquiry · doi:10.1177/10778004221097056 · Cited by 34
Cai, J., Chowdhury, S., Zhou, H., & Wohn, D. Y. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610191 · Cited by 34
Alamsyah, A., & Sagama, Y. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200394 · Cited by 33
Wang, Y., & Diakopoulos, N. A. (2021) · ACM Transactions on Social Computing · doi:10.1145/3484245 · Cited by 33
Bystanders of Online Moderation: Examining the Effects of Witnessing Post-Removal Explanations
Jhaver, S., Rathi, H., & Saha, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642204 · Cited by 33
Self-citationChen, Z., Ye, J., Tsai, B., Ferrara, E., & Luceri, L. (2025) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3720553.3746675 · Cited by 32
Governance of Social Media Platforms: A Literature Review
Kumar, A., & Gupta, S. (2023) · Pacific Asia journal of the Association for Information Systems · doi:10.17705/1pais.15103 · Cited by 32
ConvEx: A Visual Conversation Exploration System for Discord Moderators
Choi, F., Bajpai, T., Pratipati, S., & Chandrasekharan, E. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610053 · Cited by 31
Metadata
- Authors
- Jhaver, S., Birman, I., Gilbert, E., & Bruckman, A.
- Affiliations
- Georgia Institute of Technology (US)
- University of Michigan, Ann Arbor, MI
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2019-07-19
- Volume
- 26
- Issue
- 5
- Pages
- 1-35
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2964034671
- Cited by
- 289
- References
- 92
- Citations by year
- 2026: 36 · 2025: 56 · 2024: 42 · 2023: 41 · 2022: 43 · 2021: 44 · 2020: 22 · 2019: 5
- FWCI
- 49.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion, other-oa)
- Funders
- National Science Foundation (IIS-1553376)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- human-machine collaboration, content moderation, automated content moderation, Reddit, AutoMod, social media governance, platform governance, socio-technical systems, audit tool, division of labor, human moderators, tool sharing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.