CoChat
Sign Up
Cited by 80Misinformation and Its ImpactsEthics and Social Impacts of AISpam and Phishing Detection

Truth, Lies, and Automation: How Language Models Could Change Disinformation

Open access
Center for Security and Emerging Technology, Buchanan, Ben, Lohn, Andrew, Musser, Micah, Sedova, Katerina·2021
DOI 10.51593/2021ca003
Open this paper in CoChat — free accountOpen PDF

Abstract

Growing popular and industry interest in high-performing natural language generation models has led to concerns that such models could be used to generate automated disinformation at scale. This report examines the capabilities of GPT-3--a cutting-edge AI system that writes text--to analyze its potential misuse for disinformation. A model like GPT-3 may be able to help disinformation actors substantially reduce the work necessary to write disinformation while expanding its reach and potentially also its effectiveness.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.51593/2021ca003

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Models like the GPT series can produce vast amounts of compelling mis/disinformation with little human involvement.

    “and vast amounts of compelling mis/disinformation with little human involvement (Buchanan et al., 2021;Spitale et al., 2023).”

    From Anatomy of an AI-powered malicious social botnet (2024) · cites this paper as Buchanan et al., 2021

    The Conclusion says human–machine teams can produce moderate-quality disinformation at high scale, and the Narrative Wedging section estimates several thousand messages per day after a process taking minutes. But that section also says some outputs are not compelling and human selection is valuable; the Human-Machine Teams section says more demanding tasks require more human involvement. The source therefore does not support an unqualified claim that vast amounts of compelling disinformation can be produced with little human involvement. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • LLMs can create moderate-to-high quality misinformation messages with little human involvement.

    “Buchanan et al. [14] demonstrated LLM's ability to create moderate-to-high quality misinformation messages with little human involvement.”

    From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [14]

  • AI techniques can be used to generate misinformation.

    “Unfortunately, AI techniques are far from being a savior in the battle against misinformation, but instead, can be used to generate misinformation [14].”

    From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [14]

  • AI-generated misinformation can customize language for specific groups and sometimes deploy stereotypes and racist language on certain topics.

    “They also discovered that AI-generated misinformation could customize language for specifc groups and sometimes deploy stereotypes and racist language on certain topics.”

    From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [14]

  • AI-generated text has superior capacities to scale and be customized for targeted readers.

    “and has superior capacities to scale and be customized for targeted readers [14].”

    From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [14]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. Aleksandra Garmazhapova, "Where do trolls live and who feeds them," Novaya Gazeta, September 7, 2013: https://web.archive.org/web/20191210065512/https://novayagazeta.ru/ articles/2013/09/07/56253-gde-zhivut-trolli-i-kto-ih-kormit; Polina Rusyaeva and Andrei Zaharov, "How 'Troll Factory' worked the U.S. Elections," RBK Magazine, October 17, 2017, https://web.archive.org/ web/20210303095306/https://www.rbc.ru/magazine/2017/11/59e0c17d9a79470e05a9e6c1. The agency at times used automated Twitter accounts for a small percentage of its activity, but very little is known about this aspect of their operations. Twitter Public Policy Blog, "Update on Twitter's review of the 2016 US election," January 19, 2018, https://blog.twitter.com/en_us/topics/company/2018/2016- election-update.html;
  2. U.S. Senate Select Committee on Intelligence, "Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume 2: Russia's Use of Social Media," p. 18, https://www. intelligence.senate.gov/sites/default/files/documents/Report_Volume2.pdf
  3. U.S. Senate Select Committee on Intelligence, "Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume 2: Russia's Use of Social Media," p.30, https://www.intelligence.senate.gov/ sites/default/files/documents/Report_Volume2.pdf
  4. Ben Buchanan, The Hacker and the State, Harvard University Press: Cambridge, MA, chapter 10. U.S. Senate Select Committee on Intelligence, "Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume 2: Russia's Use of Social Media," p. 30, https://www.intelligence.senate.gov/ sites/default/files/documents/Report_Volume2.pdf
  5. Tom B. Brown, et al. "Language models are few-shot learners," arXiv, May 28, 2020, https://arxiv. org/abs/2005.14165;
  6. U.S. Senate Select Committee on Intelligence, "Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume 2: Russia's Use of Social Media," p.11, https://www.intelligence.senate.gov/ sites/default/files/documents/Report_Volume2.pdf
  7. Tom B. Brown, et al. "Language models are few-shot learners," arXiv, May 28, 2020, https://arxiv.org/ abs/2005.14165. · arXiv
  8. Tom B. Brown, et al. "Language models are few-shot learners," arXiv, May 28, 2020, https://arxiv.org/ abs/2005.14165. · arXiv
  9. U.S. Senate Select Committee on Intelligence, "Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume 2: Russia's Use of Social Media," p. 38, https://www.intelligence.senate.gov/ sites/default/files/documents/Report_Volume2.pdf

Cited by

Sorted by citations

Related

Authors
Center for Security and Emerging Technology, Buchanan, B., Lohn, A., Musser, M., & Sedova, K.
Affiliations
Center for Security and Emerging Technology (US)
ORCID
Publisher
Center for Security and Emerging Technology
Published
2021-05
DOI
10.51593/2021ca003
OpenAlex
W3201290078
Cited by
74
References
10
Citations by year
2026: 6 · 2025: 10 · 2024: 22 · 2023: 26 · 2022: 4 · 2021: 6
Type
report
Language
en
Open access
Licence
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
disinformation, GPT-3, large language models, natural language generation, AI misuse, generative AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review