CoChat

The DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index; 28 of the 69 sources its claims cite were checked for support.

Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions

Open access
Zhou, Jiawei, Zhang, Yixuan, Luo, Qianni, Parker, Andrea G, De Choudhury, Munmun · 2023 · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Abstract

Large language models have abilities in creating high-volume human-like texts and can be used to generate persuasive misinformation. However, the risks remain under-explored. To address the gap, this work first examined characteristics of AI-generated misinformation (AI-misinfo) compared with human creations, and then evaluated the applicability of existing solutions.

Overview

CoChat Summary

TL;DR

This paper compiles paired human-created and GPT-3-generated COVID-19 misinformation to characterize AI-generated misinformation and test whether existing detection models and journalistic assessment guidelines still work on it. It finds significant linguistic/expression differences, a significant but not catastrophic drop in detection performance, and questionable applicability of existing assessment guidelines because AI-misinfo often mimics credibility signals.

Key findings

  • AI-generated misinformation differed significantly from human-created misinformation: it had more affect (+39.18% overall) and cognitive processing (+47.84% overall), but far fewer informal expressions (−88.94%) and netspeak (−96.98%); AI-generated posts differed more from human creations than AI-generated news did.
  • Four expression patterns characterized AI-misinfo: enhancement of details (339/500, 67.8%), communication of uncertainty and limitations (66/500, 13.2%), tendency to draw conclusions (148/500, 29.6%), and simulation of personal/human-like tones (141/500, 28.2%).
  • Existing detection model CT-BERT maintained high predictability but dropped significantly on AI-misinfo: recall 0.946 and F1 0.972 on AI-misinfo versus recall 0.996 and F1 0.998 on Human-misinfo (χ²=22.2, p<0.00001). Error analysis found 27 AI-misinfo false negatives (13 news, 14 non-news posts), and 77% (21) of false negatives contained authoritative entities such as “government,” “state,” “WHO,” and “official.”
  • Existing information assessment guidelines had questionable applicability: AI-misinfo was more likely to meet criteria in evidence credibility, source transparency, and limitation acknowledgment; AI-generated fake news was more likely to cite sources and highlight source credibility, and AI-misinfo acknowledged alternative explanations or understanding significantly more often than Human-misinfo.

Methods

  • The authors compiled Human-misinfo from three peer-reviewed COVID-19 misinformation datasets: COVID19-FNIR (3,727), CONSTRAINT (5,100), and COVID Rumor (3,581), totaling 12,408 documents. LDA topic modeling identified five topics and selected 50 representative documents per topic (250 total; 97 news, 153 social media posts), which were abstracted via Narrative Theory-guided content analysis into 500 narrative prompts (250 news, 250 non-news; average Cohen’s κ=0.87).
  • GPT-3 davinci-002 generated 500 AI-misinfo outputs from those prompts at temperature 70%, keeping one best result; average token size was 119.12 (SD 69.25, Max 472, Min 32).
  • RQ1 used LIWC, SAGE, and rapid qualitative analysis on 250 human-AI pairs. RQ2 evaluated the AAAI 2021 COVID-19 fake news detection winner CT-BERT on 1,000 unseen documents (500 AI-misinfo, 500 Human-misinfo) using precision, recall, and F1, plus recall-centric error analysis. RQ3 used deductive coding of 120 human-AI pairs against Kovach and Rosenstiel’s “way of skeptical knowing” guidelines, with inter-rater Cohen’s κ of 0.83 then 0.96.

Limitations

  • Stated: the work is situated in the COVID-19 context and can be reasonably generalized to similar topics such as crisis communication and public health, but not all findings apply directly to other contexts; AI-misinfo was created only by GPT-3, and future work should comparatively study other LLMs and themes.
  • Stated: the authors assume datasets without noted AI-generated or synthetic text are human-created, though this may not be true; narrative prompts may leave out nuanced attributes, so the approach is a demonstration of balancing generative and specific attributes rather than a comprehensive capture of all COVID-19 misinformation; evaluated solutions were pre-existing and designed for human-created misinformation, so results are exploratory evaluations of risk and applicability.
  • Evident from text: AI-misinfo came from a single generation setup (GPT-3 davinci-002, temperature 70%, one best result), and the human-misinfo datasets were collected mostly before June 2020 and no later than December 2020; RQ3’s guideline evaluation coded only 120 pairs rather than the full dataset.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3544548.3581318

Cites

105 of 106 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY-NC-ND · From the PDF
Figure 2 Coherence scores for topic modeling in humancreated misinformation (highest value observed at =5).
Figure 2 Coherence scores for topic modeling in humancreated misinformation (highest value observed at =5).CC-BY-NC-ND · From the PDF
Image 3
CC-BY-NC-ND · From the PDF
(b) Word-tree of 'outbreak' in AI-misinfo dataset
(b) Word-tree of 'outbreak' in AI-misinfo datasetCC-BY-NC-ND · From the PDF
Image 5
CC-BY-NC-ND · From the PDF
Figure 5 Confusion matrix for AI-misinfo and Humanmisinfo datasets.
Figure 5 Confusion matrix for AI-misinfo and Humanmisinfo datasets.CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

23 Valid · 7 Not valid · 28 Not checked · 114 Own results and interpretations

69 more claims · 1 with their cited sources checked

Create a free account to see all 172 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 56
  • It may state two separate claims as one. 63
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 106 references

Research works printed but not linked: 8

  1. Alison Hamilton. 2013. Qualitative methods in rapid turn-around health services research. Health services research & development cyberseminar (2013).
  2. 10.18653/v1 · doi:10.18653/v1
  3. Bill Kovach and Tom Rosenstiel. 2021. The Elements of Journalism, Revised and Updated 4th Edition: What Newspeople Should Know and the Public Should Expect. Crown Publishing Group (NY).
  4. Bruno Latour and Steve Woolgar. 2013. An anthropologist visits the laboratory. In Laboratory Life. Princeton University Press, 43–104.
  5. 10.1093/JAMIA · doi:10.1093/jamia
  6. David Mimno, Hanna M Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. Association for Computational Linguistics, 262–272.
  7. Juan Carlos Medina Serrano, Orestis Papakyriakopoulos, and Simon Hegelich. 2020. NLP-based feature extraction for the detection of COVID-19 misinformation videos on YouTube. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020.
  8. 10.1093/EURPUB · doi:10.1093/eurpub

Cited by

See all 288

Metadata

Authors
Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M.
Affiliations
  • Georgia Institute of Technology (US)
  • Ohio University (US)
  • School of Interactive Computing, Georgia Institute of Technology, United States
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-20
OpenAlex
W4366588626
Cited by
288
References
74
Citations by year
2026: 56 · 2025: 119 · 2024: 89 · 2023: 24
FWCI
268.06
Type
conference paper
Language
en
Open access
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc-nd)
Funders
National Science Foundation (2137724)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
AI-generated misinformation, large language models, misinformation detection, COVID-19 misinformation, human-AI comparison, source credibility, source transparency, uncertainty communication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions. https://app.cochat.ai/papers/10.1145/3544548.3581318
The paper
Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023). Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581318

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.