CoChat
Sign Up
Cited by 112Misinformation and Its ImpactsAuthorship Attribution and ProfilingSpam and Phishing Detection

The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.

The Limitations of Stylometry for Detecting Machine-Generated Fake News

Open access
Schuster, Tal, Schuster, Roei, Shah, Darsh J., Barzilay, Regina·2020·Computational Linguistics
DOI 10.1162/coli_a_00380
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent developments in neural language models (LMs) have raised concerns about their potential misuse for automatically spreading misinformation. In light of these concerns, several studies have proposed to detect machine-generated fake news by capturing their stylistic differences from human-written text. These approaches, broadly termed stylometry, have found success in source attribution and misinformation detection in human-written texts. However, in this work, we show that stylometry is limited against machine-generated misinformation. Whereas humans speak differently when trying to deceive, LMs generate stylistically consistent text, regardless of underlying motive. Thus, though stylometry can successfully prevent impersonation by identifying text provenance, it fails to distinguish legitimate LM applications from those that introduce false information. We create two benchmarks demonstrating the stylistic similarity between malicious and legitimate uses of LMs, utilized in auto-completion and editing-assistance settings. 1 Our findings highlight the need for non-stylometry approaches in detecting machine-generated misinformation, and open up the discussion on the desired evaluation benchmarks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/coli_a_00380

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 52 references

Research works printed but not linked: 18

  1. Caliskan-Islam Aylin, 2015, 24th USENIX Security Symposium (USENIX Security 15), 255
  2. Dostoyevsky Fyodor, 1866, Crime and Punishment
  3. Enos Frank, 2007, International Speech Communication Association, 2281
  4. Feng Song, 2012, Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2, 171
  5. Goswami Sumit, 2009, International AAAI Conference on Web and Social Media
  6. Hanselowski Andreas, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 1859
  7. Kochkina Elena, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 3402
  8. Liu Yang, 2018, AAAI Conference on Artificial Intelligence, 354
  9. Matsumoto David, 2011, FBI Law Enforcement Bulletin, 1
  10. Mosteller Frederick, 1963, Journal of the American Statistical Association, 58, 275
  11. Neal Tempestt J., 2017, ACM Computing Surveys, 50, 86:1
  12. Ott Myle, 2011, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 309
  13. Pérez-Rosas Verónica, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 3391
  14. Radford Alec, 2019, OpenAI Blog, 1
  15. Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
  16. Wolf Thomas, 2019, ArXiv
  17. Zellers Rowan, 2019, Advances in Neural Information Processing Systems 32, 9054
  18. Zhou Xinyi, 2018, arXiv preprint arXiv:1812.00315 · arXiv

Cited by

Sorted by citations

Items 101–112 of 112

Related

See all

Metadata

Authors
Schuster, T., Schuster, R., Shah, D. J., & Barzilay, R.
Affiliations
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Tel Aviv University (IL)
ORCID
Venue
Computational Linguistics
Publisher
MIT Press - Journals
Published
2020-06
Volume
46
Issue
2
Pages
499-510
ISSN
  • 0891-2017 (ISSN-L)
  • 1530-9312
DOI
10.1162/coli_a_00380
OpenAlex
W3008170323
Cited by
109
References
52
Citations by year
2026: 8 · 2025: 14 · 2024: 15 · 2023: 22 · 2022: 29 · 2021: 13 · 2020: 8
FWCI
24.02
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Computational Linguistics (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
National Science Foundation (1650589)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
stylometry, misinformation detection, neural language models, stylistic similarity, source attribution, autocomplete, evaluation benchmark, deception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review