CoChat

Ex Machina: Personal Attacks Seen at Scale

Open access
Wulczyn, Ellery, Thain, Nithum, Dixon, Lucas · 2017 · Proceedings of the 26th International Conference on World Wide Web

The DOI matches its Crossref record; 31 of the 33 references it cites were found in a registry or an index.

Abstract

The damage personal attacks cause to online discourse motivates many platforms to try to curb the phenomenon. However, understanding the prevalence and impact of personal attacks in online platforms at scale remains surprisingly difficult. The contribution of this paper is to develop and illustrate a method that combines crowdsourcing and machine learning to analyze personal attacks at scale. We show an evaluation method for a classifier in terms of the aggregated number of crowd-workers it can approximate. We apply our methodology to English Wikipedia, generating a corpus of over 100k high quality human-labeled comments and 63M machine-labeled ones from a classifier that is as good as the aggregate of 3 crowd-workers, as measured by the area under the ROC curve and Spearman correlation. Using this corpus of machine-labeled scores, our methodology allows us to explore some of the open questions about the nature of online personal attacks. This reveals that the majority of personal attacks on Wikipedia are not the result of a few malicious users, nor primarily the consequence of allowing anonymous contributions from unregistered users.

Overview

Integrity signals

Registry

Crossref · 10.1145/3038912.3052591

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 31 references

Research works printed but not linked: 4

  1. M. Duggan. Online harassment. Pew Research Center, 2014.
  2. I. Gagliardone, D. Gal, T. Alves, and G. Martinez. Countering online hate speech. UNESCO Publishing, 2015.
  3. K. Krippendorff. Reliability in content analysis. Human communication research, 30(3):411--433, 2004.
  4. 10.5555/1396222 · doi:10.5555/1396222

Cited by

See all 1,000

Metadata

Authors
Wulczyn, E., Thain, N., & Dixon, L.
Affiliations
  • Wikimedia Foundation (US)
  • Jigsaw, New York City, NY, USA
Venue
Proceedings of the 26th International Conference on World Wide Web
Published
2017-04-03
Pages
1391-1399
OpenAlex
W2540646130
Cited by
636
References
33
Citations by year
2026: 35 · 2025: 52 · 2024: 43 · 2023: 74 · 2022: 64 · 2021: 140 · 2020: 91 · 2019: 71 · 2018: 52 · 2017: 14
FWCI
28.57
Type
conference paper
Language
en
Open access
Locations
International World Wide Web Conference (WWW) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Communication › Wikis in Education and Collaboration
Keywords
personal attacks, online discourse, crowdsourcing, classifier evaluation, malicious users

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Ex Machina: Personal Attacks Seen at Scale. https://app.cochat.ai/papers/10.1145/3038912.3052591
The paper
Wulczyn, E., Thain, N., & Dixon, L. (2017). Ex Machina: Personal Attacks Seen at Scale. Proceedings of the 26th International Conference on World Wide Web. https://doi.org/10.1145/3038912.3052591

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.