The DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index; 28 of the 69 sources its claims cite were checked for support.
Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions
Open accessAbstract
Large language models have abilities in creating high-volume human-like texts and can be used to generate persuasive misinformation. However, the risks remain under-explored. To address the gap, this work first examined characteristics of AI-generated misinformation (AI-misinfo) compared with human creations, and then evaluated the applicability of existing solutions.
Overview
CoChat Summary
TL;DR
This paper compiles paired human-created and GPT-3-generated COVID-19 misinformation to characterize AI-generated misinformation and test whether existing detection models and journalistic assessment guidelines still work on it. It finds significant linguistic/expression differences, a significant but not catastrophic drop in detection performance, and questionable applicability of existing assessment guidelines because AI-misinfo often mimics credibility signals.
Key findings
- AI-generated misinformation differed significantly from human-created misinformation: it had more affect (+39.18% overall) and cognitive processing (+47.84% overall), but far fewer informal expressions (−88.94%) and netspeak (−96.98%); AI-generated posts differed more from human creations than AI-generated news did.
- Four expression patterns characterized AI-misinfo: enhancement of details (339/500, 67.8%), communication of uncertainty and limitations (66/500, 13.2%), tendency to draw conclusions (148/500, 29.6%), and simulation of personal/human-like tones (141/500, 28.2%).
- Existing detection model CT-BERT maintained high predictability but dropped significantly on AI-misinfo: recall 0.946 and F1 0.972 on AI-misinfo versus recall 0.996 and F1 0.998 on Human-misinfo (χ²=22.2, p<0.00001). Error analysis found 27 AI-misinfo false negatives (13 news, 14 non-news posts), and 77% (21) of false negatives contained authoritative entities such as “government,” “state,” “WHO,” and “official.”
- Existing information assessment guidelines had questionable applicability: AI-misinfo was more likely to meet criteria in evidence credibility, source transparency, and limitation acknowledgment; AI-generated fake news was more likely to cite sources and highlight source credibility, and AI-misinfo acknowledged alternative explanations or understanding significantly more often than Human-misinfo.
Methods
- The authors compiled Human-misinfo from three peer-reviewed COVID-19 misinformation datasets: COVID19-FNIR (3,727), CONSTRAINT (5,100), and COVID Rumor (3,581), totaling 12,408 documents. LDA topic modeling identified five topics and selected 50 representative documents per topic (250 total; 97 news, 153 social media posts), which were abstracted via Narrative Theory-guided content analysis into 500 narrative prompts (250 news, 250 non-news; average Cohen’s κ=0.87).
- GPT-3 davinci-002 generated 500 AI-misinfo outputs from those prompts at temperature 70%, keeping one best result; average token size was 119.12 (SD 69.25, Max 472, Min 32).
- RQ1 used LIWC, SAGE, and rapid qualitative analysis on 250 human-AI pairs. RQ2 evaluated the AAAI 2021 COVID-19 fake news detection winner CT-BERT on 1,000 unseen documents (500 AI-misinfo, 500 Human-misinfo) using precision, recall, and F1, plus recall-centric error analysis. RQ3 used deductive coding of 120 human-AI pairs against Kovach and Rosenstiel’s “way of skeptical knowing” guidelines, with inter-rater Cohen’s κ of 0.83 then 0.96.
Limitations
- Stated: the work is situated in the COVID-19 context and can be reasonably generalized to similar topics such as crisis communication and public health, but not all findings apply directly to other contexts; AI-misinfo was created only by GPT-3, and future work should comparatively study other LLMs and themes.
- Stated: the authors assume datasets without noted AI-generated or synthetic text are human-created, though this may not be true; narrative prompts may leave out nuanced attributes, so the approach is a demonstration of balancing generative and specific attributes rather than a comprehensive capture of all COVID-19 misinformation; evaluated solutions were pre-existing and designed for human-created misinformation, so results are exploratory evaluations of risk and applicability.
- Evident from text: AI-misinfo came from a single generation setup (GPT-3 davinci-002, temperature 70%, one best result), and the human-misinfo datasets were collected mostly before June 2020 and no later than December 2020; RQ3’s guideline evaluation coded only 120 pairs rather than the full dataset.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
23 Valid · 7 Not valid · 28 Not checked · 114 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Buchanan et al. [14] demonstrated LLM's ability to create moderate-to-high quality misinformation messages with little human involvement.”
§ Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions > 2.1 Generative AI and Its Role in the Age of Misinformation
What the sources say
- Supportedcites [14] 14. Ben Buchanan, Andrew Lohn, Micah Musser, and Katerina Sedova. 2021. Truth, Lies, and Automation: How Language Models Could Change Disinformation. Tech- nical Report. Center for Security and Emerging Technology, Georgetown Uni- versity. 1-54 pages. https://doi.org/10.51593/2021CA003 source
In “Narrative Wedging” (page unavailable), the paper reports that most GPT-3 messages were well targeted and that 85% were rated credible by at least three of four evaluators. The same section describes a brief human prompting and selection process that took minutes, supporting the claim of little human involvement. The demonstrated results concern GPT-3 in the tested disinformation tasks, not every LLM or task. (LLM, conf=0.90)
states a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Unfortunately, AI techniques are far from being a savior in the battle against misinformation, but instead, can be used to generate misinformation [14].”
§ Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions > 1 INTRODUCTION
What the sources say
- Supportedcites [14] 14. Ben Buchanan, Andrew Lohn, Micah Musser, and Katerina Sedova. 2021. Truth, Lies, and Automation: How Language Models Could Change Disinformation. Tech- nical Report. Center for Security and Emerging Technology, Georgetown Uni- versity. 1-54 pages. https://doi.org/10.51593/2021CA003 source
The report states that its study shows a real prospect of automated tools generating content for disinformation campaigns (Center for Security and Emerging Technology, p. v). Its Narrative Elaboration section also describes using GPT-3 to produce stories that spread false claims. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior work has shown that health professionals, academic institutions, and government agencies are considered more trusted sources than social media, family, and friends [28,70].”
§ Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions > 7.2.2 Evidence.
What the sources say
- Partly supportedcites [28] 28. Mohan Dutta-Bergman. 2003. Trusted Online Sources of Health Information: Diferences in Demographics, Health Beliefs, and Health-Information Orien- tation. J Med Internet Res 2003;5(3):e21 https://www.jmir.org/2003/3/e21 5, 3 (9 2003), e893. https://doi.org/10.2196/JMIR.5.3.E21 sourceExperimental — accuracy not yet measured
The Results and Discussion identify the personal doctor, medical university, and federal government website as the most trusted online health-information sources among those studied. But the Methods list neither social media nor family and friends as options, so this paper does not support the claimed comparison with them. (LLM, conf=0.90)
- Supportedcites [70] 70. Linqi Lu, Jiawei Liu, Y. Connie Yuan, Kelli S. Burns, Enze Lu, and Dongxiao Li. 2021. Source Trust and COVID-19 Information Sharing: The Mediating Roles of Emotions and Beliefs About Sharing. Health Education and Behavior 48, 2 (4 2021), 132-139. https://doi.org/10.1177/1090198120984760 source
The Discussion states that health professionals, government agencies, and academic institutions were rated as more trusted COVID-19 information sources than family and friends, with social media rated least trusted. The Results report the same ranking. The finding concerns COVID-19 information sources, which is the context measured by this paper. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“They were more likely to possess fewer signal words discovered in prior work on misinformation [51], such as strong sentimental expressions (e.g., 'kill' and 'die').”
§ Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions > 6.2 RQ2 Results
What the sources say
- Supportedcites [51] 51. Shan Jiang and Christo Wilson. 2018. Linguistic Signals under Misinformation and Fact-Checking: Evidence from User Comments on Social Media. Proc. ACM Hum.-Comput. Interact. 2, CSCW (11 2018). https://doi.org/10.1145/3274351 source
Section 5.1, Figure 10 identifies a word cluster containing “kill” and “die” and associates its use in user comments with lower post veracity. This supports the citation’s role in identifying previously studied misinformation-related signal words; the false-negative comparison is the citing authors’ own finding, not a result attributed to this source. No page number is available in our copy. (LLM, conf=0.88)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Compared with discussions about deep fake videos and potential harms that have been highlighted in media in recent years, people are less aware of the persuasiveness and fuency of AI-generated text [32].”
§ Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions > 8.2.4
What the sources say
- Partly supportedcites [32] 32. Dilrukshi Gamage, Piyush Ghasiya, Vamshi Bonagiri, Mark E Whiting, and Kazutoshi Sasahara. 2022. Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implications. In CHI Conference on Human Factors in Computing Systems. 1-19. sourceExperimental — accuracy not yet measured
Section 4.3 finds that Reddit communities paid substantially more attention to deepfake videos than to the deceptive potential of AI-generated text, and says AI can generate news with apparent legitimacy. That supports the comparison, but the finding concerns Reddit discussions, not people's awareness generally; the passage also does not establish that media coverage caused the difference. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
69 more claims · 1 with their cited sources checked
Create a free account to see all 172 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 56
- It may state two separate claims as one. 63
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they often struggle with accuracy and robustness, leading to false positives or undetected deepfakes (Almars 2021; Feuerriegel et al. 2023; Zhou et al. 2023).”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Zhou et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, Zhou et al. show that GPT-3 can easily produce misinformation about COVID-19 that can escape detection by most social media platforms (8).”
From Can Generative AI improve social science? (2024) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As described in Section 3.1, a multiplicity of AI systems could lead to an increase in the quantity and quality of misinformation (Kay et al., 2024; Zhou et al., 2023).”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Zhou et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Detecting such content poses challenges for existing automated detection models (Zhou et al., 2023).”
From Anatomy of an AI-powered malicious social botnet (2024) · cites this paper as Zhou et al., 2023
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“” This prevalence of risk expressions echoes prior work that shows false rumors inspire fear in the replies compared with true information [108].”
BackgroundThe Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · cited 4×
“We utilized the Linguistic Inquiry and Word Count (LIWC) [104] as the text analysis tool to examine linguistic styles that are content-driven but focus on stylistic and tonal features.”
MethodInfluentialMisinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“, [1, 4]) and encouraged clear explanations of AI limits and capabilities.”
BackgroundProceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
Research works printed but not linked: 8
- Alison Hamilton. 2013. Qualitative methods in rapid turn-around health services research. Health services research & development cyberseminar (2013).
- 10.18653/v1 · doi:10.18653/v1
- Bill Kovach and Tom Rosenstiel. 2021. The Elements of Journalism, Revised and Updated 4th Edition: What Newspeople Should Know and the Public Should Expect. Crown Publishing Group (NY).
- Bruno Latour and Steve Woolgar. 2013. An anthropologist visits the laboratory. In Laboratory Life. Princeton University Press, 43–104.
- 10.1093/JAMIA · doi:10.1093/jamia
- David Mimno, Hanna M Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. Association for Computational Linguistics, 262–272.
- Juan Carlos Medina Serrano, Orestis Papakyriakopoulos, and Simon Hegelich. 2020. NLP-based feature extraction for the detection of COVID-19 misinformation videos on YouTube. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020.
- 10.1093/EURPUB · doi:10.1093/eurpub
Cited by
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
Self-citationGenerative artificial intelligence
Banh, L., & Strobel, G. (2023) · Electronic Markets · doi:10.1007/s12525-023-00680-1 · Cited by 579
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
“Recent incidents have highlighted the risks that LLMs can generate misinformation [93], promote conspiracy theories [40], scale spear phishing attacks [34], and facilitate hate cam-paigns [69].”
Self-citationCan Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Metadata
- Authors
- Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M.
- Affiliations
- Georgia Institute of Technology (US)
- Ohio University (US)
- School of Interactive Computing, Georgia Institute of Technology, United States
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-20
- OpenAlex
- W4366588626
- Cited by
- 288
- References
- 74
- Citations by year
- 2026: 56 · 2025: 119 · 2024: 89 · 2023: 24
- FWCI
- 268.06
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc-nd)
- Funders
- National Science Foundation (2137724)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- AI-generated misinformation, large language models, misinformation detection, COVID-19 misinformation, human-AI comparison, source credibility, source transparency, uncertainty communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions. https://app.cochat.ai/papers/10.1145/3544548.3581318
- The paper
- Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023). Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581318
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.