Algorithmic content moderation: Technical and political challenges in the automation of platform governance
Open accessPublished version of a preprint
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
As government pressure on major technology companies builds, both firms and legislators are searching for technical solutions to difficult platform governance puzzles such as hate speech and misinformation. Automated hash-matching and predictive machine learning tools – what we define here as algorithmic moderation systems – are increasingly being deployed to conduct content moderation at scale by major platforms for user-generated content such as Facebook, YouTube and Twitter. This article provides an accessible technical primer on how algorithmic moderation works; examines some of the existing automated tools used by major platforms to handle copyright infringement, terrorism and toxic speech; and identifies key political and ethical issues for these systems as the reliance on them grows. Recent events suggest that algorithmic moderation has become necessary to manage growing public expectations for increased platform responsibility, safety and security on the global stage; however, as we demonstrate, these systems remain opaque, unaccountable and poorly understood. Despite the potential promise of algorithms or ‘AI’, we show that even ‘well optimized’ moderation systems could exacerbate, rather than relieve, many existing problems with content policy as enacted by platforms for three main reasons: automated moderation threatens to (a) further increase opacity, making a famously non-transparent set of practices even more difficult to understand or audit, (b) further complicate outstanding issues of fairness and justice in large-scale sociotechnical systems and (c) re-obscure the fundamentally political nature of speech decisions being executed at scale.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While AI-driven moderation offers benefits such as scalability, consistency, and reduced exposure to harmful content for human moderators, it also presents challenges. These include potential algorithmic bias, issues with accuracy in detecting nuanced content, and concerns about transparency and accountability [101].”
From The Impact of Social Media & Technology on Child and Adolescent Mental Health (2025) · cites this paper as [101]
In the citing paper
“In addition, AI may negatively impact human rights including freedom of expression and privacy (Gorwa et al., 2020;Langvardt, 2017).”
From Representation of Libraries in Artificial Intelligence Regulations and Implications for Ethics and Practice (2022) · cites this paper as Gorwa et al., 2020
In the citing paper
“One such example is content moderation, where moderator decisions on individual comments for further actions (e.g., hiding the content or prompting further review, depending on the community policy) are of limited consequence; instead the key challenge lies in dealing with the massive scale of comments.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [35]
In the citing paper
“The scale of content is the most important argument for some form of automation in content moderation [33,35].”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [35]
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Semantically similar words therefore have similar positions (Mikolov et al., 2013).”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Recent years have seen substantial discussion about the potential for algorithmic decision-making systems to have unfair or discriminatory impacts on different groups, such as protected classes under anti-discrimination law (e.g. gender, race, religion, disability; see Barocas and Selbst, 2016).”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“While such work has focused on fairness/discrimination in the distribution of outcomes in economic, welfare or justice domains (Berk et al., 2018; Chouldechova et al., 2018; Hardt et al., 2016), some of it has parallels in the area of algorithmic content moderation.”
BackgroundAnanny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · cited 2×
“This is not to say that transparency is any sort of panacea, either in general or when it comes to algorithmic systems (Ananny and Crawford, 2018; Gorwa and Garton Ash, 2019), but minimum standards of decisional transparency are essential to allow both ordinary users and critical experts to…”
BackgroundÇömlekçi, M. F. (2019) · Kırklareli University Institutional Repository (Kırklareli University) · Cited by 1,607
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 1×
“While such work has focused on fairness/discrimination in the distribution of outcomes in economic, welfare or justice domains (Berk et al., 2018; Chouldechova et al., 2018; Hardt et al., 2016), some of it has parallels in the area of algorithmic content moderation.”
BackgroundGillespie, T. (2019) · doi:10.12987/9780300235029 · Cited by 806
(2019) · doi:10.1080/1369118x.2019.1573914 · Cited by 719 · cited 6×
“…US election, there has been a substantial increase in public attention paid to content moderation issues – now widely seen as a crucial element of major tech and platform policy debates – as well as broader academic awareness of the problems with the platform governance status quo (Gorwa, 2019b).”
BackgroundInfluentialSelf-citationFairness in Machine Learning: Lessons from Political Philosophy
Binns, R. (2017) · arXiv · doi:10.48550/arxiv.1712.03586 · Cited by 676 · cited 3×
“…ranking, or blocking, may be more or less favourable to content associated with gender, race and other protected categories (Blodgett et al., 2016; Ekstrand et al, 2018; Zehlike et al., 2017), and thus entrench forms of representational harm against such groups (Barocas et al., 2017; Binns, 2018).”
BackgroundSelf-citationEx Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636
Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535 · cited 1×
“Fairness critiques often miss broader structural issues, and risk being blind to wider patterns of systemic harm (Hoffmann, 2019).”
BackgroundFA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496
Common Sense Reasoning for Detection, Prevention, and Mitigation of Cyberbullying
Dinakar, K., Jones, B., Havasi, C., Lieberman, H., & Picard, R. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2362394.2362400 · Cited by 432 · cited 1×
“Other approaches rely on domain-specific ontologies, e.g. of terms implicated in LGBT hate speech, to attempt to infer whether a statement might be hateful given certain domain knowledge (such as the gender of the recipient; Dinakar et al., 2012).”
MethodProceedings of the 13th ACM Conference on Recommender Systems
(2019) · doi:10.1145/3298689 · Cited by 366
Moderator engagement and community development in the age of algorithms
Seering, J., Wang, T., Yoon, J., & Kaufman, G. (2019) · New Media & Society · doi:10.1177/1461444818821316 · Cited by 310 · cited 2×
“The GIFCT maintains the SIHD of terrorist content, which is now used by 13 different companies, including Instagram, LinkedIn, Oath, Reddit and Snap.”
BackgroundDemographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · cited 2×
“…used for recommendation, ranking, or blocking, may be more or less favourable to content associated with gender, race and other protected categories (Blodgett et al., 2016; Ekstrand et al, 2018; Zehlike et al., 2017), and thus entrench forms of representational harm against such groups (Barocas et…”
BackgroundThe platform governance triangle: conceptualising the informal regulation of online content
Gorwa, R. (2019) · Internet Policy Review · doi:10.14763/2019.2.1407 · Cited by 237 · cited 6×
“…US election, there has been a substantial increase in public attention paid to content moderation issues – now widely seen as a crucial element of major tech and platform policy debates – as well as broader academic awareness of the problems with the platform governance status quo (Gorwa, 2019b).”
BackgroundInfluentialSelf-citationBots, bespoke, code and the materiality of software platforms
Geiger, R. S. (2014) · Information, Communication & Society · doi:10.1080/1369118x.2013.873069 · Cited by 131 · cited 1×
“Later, as large scale peer-production communities like Wikipedia grew rapidly, automated ‘bot’ moderators enforced Wikipedia’s rules, fought vandalism and monitored articles slated for deletion, playing a key role in the moderation process (Geiger, 2014).”
BackgroundSpam: A Shadow History of the Internet
Brunton, F. (2013) · The MIT Press eBooks · doi:10.7551/mitpress/9384.001.0001 · Cited by 125
Digital detritus: 'Error' and the logic of opacity in social media content moderation
Roberts, S. T. (2018) · First Monday · doi:10.5210/fm.v23i3.8283 · Cited by 122 · cited 3×
“…administrators who formed part of the community, platform companies operate at a scale that has led them away from traditional practices of community moderation (Lampe and Resnick, 2004) and towards what has been termed ‘commercial content moderation’ or ‘platform moder-ation’ (Roberts, 2018).”
BackgroundMixed Messages? The Limits of Automated Social Media Content Analysis
Duarte, N., Llansó, E., & Loup, A. C. (2018) · Cited by 112
Witnessing in the new memory ecology: Memory construction of the Syrian conflict on YouTube
Smit, R., Heinrich, A., & Broersma, M. (2017) · New Media & Society · doi:10.1177/1461444815604618 · Cited by 71 · cited 1×
“Scholarship has documented the important role that ‘witness videos’ uploaded to platforms like YouTube have played in Syria and other contemporary conflicts (Smit et al., 2017).”
BackgroundDemocratic Transparency in the Platform Society
Gorwa, R., & Ash, T. G. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.013 · Cited by 64
Self-citationMonetizing a Meme: YouTube, Content ID, and the Harlem Shake
Soha, M., & McDowell, Z. J. (2016) · Social Media + Society · doi:10.1177/2056305115623801 · Cited by 58 · cited 2×
“After uploading their audio or video content, copyright holders have the ability to select whether they wish to take down or receive a portion of the advertising revenue from content that matches the system’s hashes (Soha and McDowell, 2016).”
BackgroundInformation & Communications Technology Law
Salter, M., Bryden, C., & Qc, R. (2009) · Cited by 47
First World Problems:' A Fair Use Analysis of Internet Memes
Patel, R. R. (2013) · UCLA Entertainment Law Review · doi:10.5070/lr8202027169 · Cited by 32
Research works printed but not linked: 11
- Barocas S, 2016, California Law Review, 104, 671
- Bar-Ziv S, 2018, Connecticut Law Review, 50, 339
- Binns R, 2018, Proceedings of Machine Learning Research, 81
- Burk DL, 2001, Harvard Journal of Law & Technology, 15, 41
- Erickson K, 2018, Journal of Intellectual Property. Information Technology & Electronic Commerce Law, 9
- Grimmelmann J, 2015, Yale Journal of Law & Technology, 17, 42
- Hosseini H, 2017, arXiv, 1702, 08138
- Huszti-Orban K, 2017, Countering Terrorism and Violent Extremism Online: What Role for Social Media Platforms
- Niu X, 2008, Acta Electronica Sinica, 36, 1405
- Perel M, 2015, Stanford Technology Law Review, 19, 473
- Suzor NP, 2019, International Journal of Communication, 13
Cited by
Trust in AI and Its Role in the Acceptance of AI Technologies
Choung, H., David, P., & Ross, A. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2050543 · Cited by 963
“AI is ubiquitous in modern life, enabling smart technologies and applications, such as smart home devices, fitness trackers, autonomous driving systems, and social media platforms (Gorwa et al., 2020; Lockey et al., 2021).”
BackgroundTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
“LMs may generate language that includes profanities, identity attacks, insults, threats, language that incites violence, or language that causes justified of-fence - as such language is prominent online [57, 64, 143, 191].”
BackgroundEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Digital Constitutionalism in Europe: Reframing Rights and Powers in the Algorithmic Society
Gregorio, G. D. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009071215 · Cited by 398
Social Media, Echo Chambers, and Political Polarization
Barberá, P. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.004 · Cited by 373
Metadata
- Authors
- Gorwa, R., Binns, R., & Katzenbach, C.
- Affiliations
- University of Oxford (GB)
- Alexander von Humboldt Institute for Internet and Society (DE)
- Corresponding author
- Gorwa, Robert
- Venue
- Big Data & Society
- Publisher
- SAGE Publications
- Published
- 2020-01
- Volume
- 7
- Issue
- 1
- Pages
- 205395171989794
- ISSN
- 2053-9517 (ISSN-L)
- OpenAlex
- W3008694996
- Cited by
- 1,097
- References
- 20
- Citations by year
- 2026: 231 · 2025: 211 · 2024: 190 · 2023: 209 · 2022: 156 · 2021: 63 · 2020: 34
- FWCI
- 157.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Big Data & Society (publishedVersion, cc-by-nc)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by-nc-nd)
- SocArXiv (OSF Preprints) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- automated content moderation, platform governance, hate speech, misinformation, predictive machine learning, copyright infringement, user-generated content, transparency, accountability, fairness and justice, socio-technical systems, Facebook, YouTube, Twitter
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.