CoChat

Algorithmic content moderation: Technical and political challenges in the automation of platform governance

Open access
Gorwa, Robert, Binns, Reuben, Katzenbach, Christian · 2020 · Big Data & Society

Published version of a preprint

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

Abstract

As government pressure on major technology companies builds, both firms and legislators are searching for technical solutions to difficult platform governance puzzles such as hate speech and misinformation. Automated hash-matching and predictive machine learning tools – what we define here as algorithmic moderation systems – are increasingly being deployed to conduct content moderation at scale by major platforms for user-generated content such as Facebook, YouTube and Twitter. This article provides an accessible technical primer on how algorithmic moderation works; examines some of the existing automated tools used by major platforms to handle copyright infringement, terrorism and toxic speech; and identifies key political and ethical issues for these systems as the reliance on them grows. Recent events suggest that algorithmic moderation has become necessary to manage growing public expectations for increased platform responsibility, safety and security on the global stage; however, as we demonstrate, these systems remain opaque, unaccountable and poorly understood. Despite the potential promise of algorithms or ‘AI’, we show that even ‘well optimized’ moderation systems could exacerbate, rather than relieve, many existing problems with content policy as enacted by platforms for three main reasons: automated moderation threatens to (a) further increase opacity, making a famously non-transparent set of practices even more difficult to understand or audit, (b) further complicate outstanding issues of fairness and justice in large-scale sociotechnical systems and (c) re-obscure the fundamentally political nature of speech decisions being executed at scale.

Overview

Integrity signals

Registry

Crossref · 10.1177/2053951719897945

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 11

  1. Barocas S, 2016, California Law Review, 104, 671
  2. Bar-Ziv S, 2018, Connecticut Law Review, 50, 339
  3. Binns R, 2018, Proceedings of Machine Learning Research, 81
  4. Burk DL, 2001, Harvard Journal of Law & Technology, 15, 41
  5. Erickson K, 2018, Journal of Intellectual Property. Information Technology & Electronic Commerce Law, 9
  6. Grimmelmann J, 2015, Yale Journal of Law & Technology, 17, 42
  7. Hosseini H, 2017, arXiv, 1702, 08138
  8. Huszti-Orban K, 2017, Countering Terrorism and Violent Extremism Online: What Role for Social Media Platforms
  9. Niu X, 2008, Acta Electronica Sinica, 36, 1405
  10. Perel M, 2015, Stanford Technology Law Review, 19, 473
  11. Suzor NP, 2019, International Journal of Communication, 13

Cited by

See all 1,097

Metadata

Authors
Gorwa, R., Binns, R., & Katzenbach, C.
Affiliations
  • University of Oxford (GB)
  • Alexander von Humboldt Institute for Internet and Society (DE)
Corresponding author
Gorwa, Robert
Venue
Big Data & Society
Published
2020-01
Volume
7
Issue
1
Pages
205395171989794
ISSN
2053-9517 (ISSN-L)
OpenAlex
W3008694996
Cited by
1,097
References
20
Citations by year
2026: 231 · 2025: 211 · 2024: 190 · 2023: 209 · 2022: 156 · 2021: 63 · 2020: 34
FWCI
157.21
Type
journal article
Language
en
Open access
Licence
  • cc-by-nc
  • journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Big Data & Society (publishedVersion, cc-by-nc)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • SocArXiv (OSF Preprints) (submittedVersion, cc-by-nc-nd)
  • SocArXiv (OSF Preprints) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
automated content moderation, platform governance, hate speech, misinformation, predictive machine learning, copyright infringement, user-generated content, transparency, accountability, fairness and justice, socio-technical systems, Facebook, YouTube, Twitter

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.