Algorithmic content moderation: Technical and political challenges in the automation of platform governance
Open accessPublished version of a preprint
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
As government pressure on major technology companies builds, both firms and legislators are searching for technical solutions to difficult platform governance puzzles such as hate speech and misinformation. Automated hash-matching and predictive machine learning tools – what we define here as algorithmic moderation systems – are increasingly being deployed to conduct content moderation at scale by major platforms for user-generated content such as Facebook, YouTube and Twitter. This article provides an accessible technical primer on how algorithmic moderation works; examines some of the existing automated tools used by major platforms to handle copyright infringement, terrorism and toxic speech; and identifies key political and ethical issues for these systems as the reliance on them grows. Recent events suggest that algorithmic moderation has become necessary to manage growing public expectations for increased platform responsibility, safety and security on the global stage; however, as we demonstrate, these systems remain opaque, unaccountable and poorly understood. Despite the potential promise of algorithms or ‘AI’, we show that even ‘well optimized’ moderation systems could exacerbate, rather than relieve, many existing problems with content policy as enacted by platforms for three main reasons: automated moderation threatens to (a) further increase opacity, making a famously non-transparent set of practices even more difficult to understand or audit, (b) further complicate outstanding issues of fairness and justice in large-scale sociotechnical systems and (c) re-obscure the fundamentally political nature of speech decisions being executed at scale.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While AI-driven moderation offers benefits such as scalability, consistency, and reduced exposure to harmful content for human moderators, it also presents challenges. These include potential algorithmic bias, issues with accuracy in detecting nuanced content, and concerns about transparency and accountability [101].”
From The Impact of Social Media & Technology on Child and Adolescent Mental Health (2025) · cites this paper as [101]
In the citing paper
“In addition, AI may negatively impact human rights including freedom of expression and privacy (Gorwa et al., 2020;Langvardt, 2017).”
From Representation of Libraries in Artificial Intelligence Regulations and Implications for Ethics and Practice (2022) · cites this paper as Gorwa et al., 2020
In the citing paper
“One such example is content moderation, where moderator decisions on individual comments for further actions (e.g., hiding the content or prompting further review, depending on the community policy) are of limited consequence; instead the key challenge lies in dealing with the massive scale of comments.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [35]
In the citing paper
“The scale of content is the most important argument for some form of automation in content moderation [33,35].”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [35]
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Semantically similar words therefore have similar positions (Mikolov et al., 2013).”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Recent years have seen substantial discussion about the potential for algorithmic decision-making systems to have unfair or discriminatory impacts on different groups, such as protected classes under anti-discrimination law (e.g. gender, race, religion, disability; see Barocas and Selbst, 2016).”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“While such work has focused on fairness/discrimination in the distribution of outcomes in economic, welfare or justice domains (Berk et al., 2018; Chouldechova et al., 2018; Hardt et al., 2016), some of it has parallels in the area of algorithmic content moderation.”
BackgroundAnanny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · cited 2×
“This is not to say that transparency is any sort of panacea, either in general or when it comes to algorithmic systems (Ananny and Crawford, 2018; Gorwa and Garton Ash, 2019), but minimum standards of decisional transparency are essential to allow both ordinary users and critical experts to…”
BackgroundÇömlekçi, M. F. (2019) · Kırklareli University Institutional Repository (Kırklareli University) · Cited by 1,607
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 1×
“While such work has focused on fairness/discrimination in the distribution of outcomes in economic, welfare or justice domains (Berk et al., 2018; Chouldechova et al., 2018; Hardt et al., 2016), some of it has parallels in the area of algorithmic content moderation.”
BackgroundGillespie, T. (2019) · doi:10.12987/9780300235029 · Cited by 806
(2019) · doi:10.1080/1369118x.2019.1573914 · Cited by 719 · cited 6×
“…US election, there has been a substantial increase in public attention paid to content moderation issues – now widely seen as a crucial element of major tech and platform policy debates – as well as broader academic awareness of the problems with the platform governance status quo (Gorwa, 2019b).”
BackgroundInfluentialSelf-citationFairness in Machine Learning: Lessons from Political Philosophy
Binns, R. (2017) · arXiv · doi:10.48550/arxiv.1712.03586 · Cited by 676 · cited 3×
“…ranking, or blocking, may be more or less favourable to content associated with gender, race and other protected categories (Blodgett et al., 2016; Ekstrand et al, 2018; Zehlike et al., 2017), and thus entrench forms of representational harm against such groups (Barocas et al., 2017; Binns, 2018).”
BackgroundSelf-citation
Research works printed but not linked: 11
- Barocas S, 2016, California Law Review, 104, 671
- Bar-Ziv S, 2018, Connecticut Law Review, 50, 339
- Binns R, 2018, Proceedings of Machine Learning Research, 81
- Burk DL, 2001, Harvard Journal of Law & Technology, 15, 41
- Erickson K, 2018, Journal of Intellectual Property. Information Technology & Electronic Commerce Law, 9
- Grimmelmann J, 2015, Yale Journal of Law & Technology, 17, 42
- Hosseini H, 2017, arXiv, 1702, 08138
- Huszti-Orban K, 2017, Countering Terrorism and Violent Extremism Online: What Role for Social Media Platforms
- Niu X, 2008, Acta Electronica Sinica, 36, 1405
- Perel M, 2015, Stanford Technology Law Review, 19, 473
- Suzor NP, 2019, International Journal of Communication, 13
Cited by
Trust in AI and Its Role in the Acceptance of AI Technologies
Choung, H., David, P., & Ross, A. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2050543 · Cited by 963
“AI is ubiquitous in modern life, enabling smart technologies and applications, such as smart home devices, fitness trackers, autonomous driving systems, and social media platforms (Gorwa et al., 2020; Lockey et al., 2021).”
BackgroundTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
“LMs may generate language that includes profanities, identity attacks, insults, threats, language that incites violence, or language that causes justified of-fence - as such language is prominent online [57, 64, 143, 191].”
BackgroundEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Digital Constitutionalism in Europe: Reframing Rights and Powers in the Algorithmic Society
Gregorio, G. D. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009071215 · Cited by 398
Social Media, Echo Chambers, and Political Polarization
Barberá, P. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.004 · Cited by 373
Metadata
- Authors
- Gorwa, R., Binns, R., & Katzenbach, C.
- Affiliations
- University of Oxford (GB)
- Alexander von Humboldt Institute for Internet and Society (DE)
- Corresponding author
- Gorwa, Robert
- Venue
- Big Data & Society
- Publisher
- SAGE Publications
- Published
- 2020-01
- Volume
- 7
- Issue
- 1
- Pages
- 205395171989794
- ISSN
- 2053-9517 (ISSN-L)
- OpenAlex
- W3008694996
- Cited by
- 1,097
- References
- 20
- Citations by year
- 2026: 231 · 2025: 211 · 2024: 190 · 2023: 209 · 2022: 156 · 2021: 63 · 2020: 34
- FWCI
- 157.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Big Data & Society (publishedVersion, cc-by-nc)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by-nc-nd)
- SocArXiv (OSF Preprints) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- automated content moderation, platform governance, hate speech, misinformation, predictive machine learning, copyright infringement, user-generated content, transparency, accountability, fairness and justice, socio-technical systems, Facebook, YouTube, Twitter
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Algorithmic content moderation: Technical and political challenges in the automation of platform governance. https://app.cochat.ai/papers/10.1177/2053951719897945
- The paper
- Gorwa, R., Binns, R., & Katzenbach, C. (2020). Algorithmic content moderation: Technical and political challenges in the automation of platform governance. Big Data & Society. https://doi.org/10.1177/2053951719897945
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.