CoChat
Sign Up
Cited by 70Misinformation and Its ImpactsAdversarial Robustness in Machine LearningTopic Modeling

On the Risk of Misinformation Pollution with Large Language Models

Open access
Pan, Yikang, Pan, Liangming, Chen, Wenhu, Nakov, Preslav, Kan, Min-Yen, Wang, William·2023·Findings of the Association for Computational Linguistics: EMNLP 2023
DOI 10.18653/v1/2023.findings-emnlp.97
Open this paper in CoChat — free accountOpen PDF

Abstract

We investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems.We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation.Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation (up to 87%) in the performance of ODQA systems.Moreover, we uncover disparities in the attributes associated with persuading humans and machines, presenting an obstacle to current human-centric approaches to combat misinformation.To mitigate the harm caused by LLM-generated misinformation, we propose three defense strategies: misinformation detection, vigilant prompting, and reader ensemble.These approaches have demonstrated promising results, albeit with certain associated costs.Lastly, we discuss the practicality of utilizing LLMs as automatic misinformation generators and provide relevant resources and code to facilitate future research in this area.1

AI snapshot

Written from the paper’s full text.

TL;DR

This paper builds a threat model showing that GPT-3.5 can generate credible misinformation that, when injected into retrieval corpora, substantially degrades open-domain QA systems. It also tests three defenses and finds in-domain detection and reader voting help, but robust, cost-effective protection remains unresolved.

Key findings

  • Deliberate LLM-generated misinformation pollution caused large ODQA performance drops: 14%–54% for DPR-based models and 20%–87% for BM25-based models; the worst reported case was BM25+GPT under REIT on NQ-1500, with EM falling from 28.20 to 3.53 (↓87%).
  • Even unintentional hallucination pollution (GENREAD) reduced the best model, DPR+FiD, by 5% on NQ-1500 and 15% on CovidNews.
  • REIT was the most damaging attack for machines despite being less human-plausible; BM25 REIT poisoned more than 90% of questions with only 10 context passages and over 95% with 100 context passages.
  • Defenses were mixed: in-domain misinformation detectors reached AUROC 91.4–99.7, but out-of-domain detectors only 50.7–64.8; reader voting improved relative EM by 11% (CTRLGEN), 12% (REVISE), and 23% (REIT), while vigilant prompting was inconsistent. A small human study found only 57% average accuracy in distinguishing machine-generated misinformation from Wikipedia passages.

Methods

  • Threat model: GPT-3.5 (text-davinci-003) generated fake documents under four zero-shot settings—GENREAD, CTRLGEN, REVISE, and REIT—with one fake document injected per question into the retrieval corpus.
  • Data and systems: NQ-1500 used 1,500 sampled Natural Questions with a December 30, 2018 Wikipedia dump; CovidNews used 1,534 COVID-19 news questions from StreamingQA filtered to Jan.–Dec. 2020. ODQA systems combined BM25 or DPR retrievers with FiD or GPT-3.5 readers, evaluated by Exact Match.
  • Defenses: a RoBERTa-based misinformation detector, misinformation-aware prompting, and a divide-and-vote reader ensemble; additional analyses measured retrieval poisoning rates and effects of increasing context size.

Limitations

  • Stated: GPT-3.5 output reproducibility is challenging; GPT-3.5 cost constrained the scope; the study focuses on selected emblematic QA systems and LLMs, so findings may not generalize to other systems or text generators.
  • Stated/evident: the work is simulation-based rather than a real-world web deployment; defense costs are significant, especially voting, which scales linearly with context passages and is financially challenging at scale.
  • Evident: out-of-domain detection barely exceeded random guessing (50.7%–64.8%), and the human detectability study was small—three annotators rating 100 documents—with the authors noting 57% may overestimate human capability due to participant learning.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-emnlp.97

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

19 of 26 cited sources checked · 13 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Machine-generated text detection methods remain limited in their precision and scope.

    “However, these methods remain limited in their precision and scope.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Stiff and Johansson, 2022 source
    • cites Mitchell et al., 2023 source
    • cites Sadasivan et al., 2023 source
  • Premisecitation

    Current models mostly adopt a reliable external knowledge source such as Wikipedia for both training and evaluation.

    “This risk remained underexplored, as the current models mostly adopt a reliable external knowledge source (such as Wikipedia) (Karpukhin et al., 2020;Hofstätter et al., 2022) for both training and evaluation.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Karpukhin et al., 2020 source

      Section 4.1 says the authors use a Wikipedia dump as the source documents for answering questions, and the positive-passage selection section describes using those passages in retriever training. Section 4.2 describes training and test splits, supporting Wikipedia-based evaluation of this model. The paper does not establish that most current models use such sources or characterize Wikipedia as reliable. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Hofstätter et al., 2022 source

      The Introduction describes FiD using an external retriever to obtain passages, and the datasets passage reports training and evaluation sets for FiD-Light’s KILT tasks. But our copy does not establish that current models mostly use Wikipedia, characterize it as reliable, or say it is the external knowledge source for both training and evaluation. The mention of Wizard of Wikipedia is a task name, not evidence for that broader claim. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    Language models bring the promise of automating the creation of convincing and misleading text for use in influence operations, rather than having to rely on human labor.

    “For malicious actors looking to spread misinformation, language models bring the promise of automating the creation of convincing and misleading text for use in influence operations, rather than having to rely on human labor (Goldstein et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Goldstein et al., 2023 source
  • Premisecitation

    Prior Seq2Seq fake article generators occasionally make grammar and commonsense mistakes, making them not deceptive enough to humans.

    “However, the articles generated by these approaches occasionally make grammar and commonsense mistakes, making them not deceptive enough to humans.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Zellers et al., 2019 source

      Section 3 (p. 3) says earlier large-scale generators produced realistic-looking text but were difficult to control. Section 4 (p. 5) evaluates Grover’s articles, not prior Seq2Seq generators. Searches for Seq2Seq, grammatical errors, and earlier generators did not find a passage saying those generators made grammar and commonsense mistakes or were insufficiently deceptive to humans. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Fung et al., 2021 source

      The Real Article Fake Article passage says an earlier GPT-2 generator produced propaganda that could fool humans, while §5.5 reports that InfoSurgeon’s own generated articles were hard for human judges to detect. Neither passage addresses whether prior Seq2Seq generators made grammatical or commonsense mistakes that reduced their deceptiveness. Searches for “grammar,” “commonsense,” and the claim’s phrasing found no such finding in our copy; page numbers were not provided. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Huang et al., 2022 source

      The Introduction and Table 8 discussion say earlier generators produce too much inaccurate information, and §5.2 reports that humans rated GROVER-GEN articles slightly less plausible than the authors’ articles. The paper does not establish that earlier articles make grammar and commonsense mistakes or that such mistakes make them insufficiently deceptive to humans. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
  • Premisecitation

    Centralized corpora, such as Common Crawl, suffer from unwanted data.

    “Centralized corpora, such as the Common Crawl suffer from unwanted data as well (Luccioni and Viviano, 2021);”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Luccioni and Viviano, 2021 source
  • Classified as probably not a substantial claim. 14
  • May combine two claims. 8
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation

    Retrieving a larger number of passages has demonstrated potential benefits in several ODQA systems operating on a clean corpus.

    “This approach has demonstrated potential benefits in several ODQA systems that operate on a clean corpus (Izacard and Grave, 2021;Lewis et al., 2021).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Izacard and Grave, 2021 source
    • cites Lewis et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    There remains a gap in the comprehensive study of the extent to which modern LLMs can generate credible-sounding misinformation, the harms from neural-generated misinformation in information-intensive applications, and mitigation strategies.

    “Although concerns regarding misinformation have been discussed in numerous reports related to AI safety (Zellers et al., 2019;Buchanan et al., 2021;Kreps et al., 2022;Goldstein et al., 2023), there remains a gap in the comprehensive study of the following research questions: (1) To what extent can modern LLMs be utilized for generating credible-sounding misinformation?(2) What potential harms can”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Zellers et al., 2019 source
    • cites Buchanan et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Kreps et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Large language models have demonstrated exceptional language generation capabilities across various domains.

    “Recently, large language models (LLMs) (Brown et al., 2020;Ouyang et al., 2022;OpenAI, 2023) have demonstrated exceptional language generation capabilities across various domains.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Brown et al., 2020 source
    • cites Ouyang et al., 2022 source
    • cites OpenAI, 2023 source

  • Classified as probably not a substantial claim.

    Premisecitation

    ODQA readers rely on a few highly relevant contexts, regardless of the entire volume of contexts, to make predictions.

    “This aligns with the previous observation that ODQA readers rely on a few highly relevant contexts, regardless of the entire volume of contexts to make the prediction (Chen et al., 2022).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chen et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    QA systems employing alternative architectures may exhibit increased robustness against the proliferation of misinformation.

    “For instance, QA systems employing alternative architectures, as demonstrated by recent works (Shao and Huang, 2022;Su et al., 2022), may exhibit increased robustness against the proliferation of misinformation.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Shao and Huang, 2022 source

      The Conclusion (section 7; page unavailable) describes an alternative recall-then-verify architecture and says its answers are more robust to distraction by evidence for other valid answers. The paper does not establish robustness against misinformation, which is a different kind of interference. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Su et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The WMT News Corpus provides fewer and less pertinent resources for answering queries.

    “Subsequent experiments corroborate that the WMT News Corpus indeed provides fewer and less pertinent resources for answering queries.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The voting strategy yielded benefits but with attached costs.

    “The voting strategy yielded benefits, albeit with attached costs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GENREAD hallucination pollution caused a 5% decrease on NQ-1500 and a 15% decrease on CovidNews for the best-performing model DPR+FiD.

    “Even under the GENREAD scenario, where misinformation is inadvertently introduced through hallucinations, we noted a 5% and 15% decrease in ODQA performance for the best-performing model (DPR+FiD) on NQ-1500 and CovidNews, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When targeting the sparse retriever BM25, REIT can poison more than 90% of questions using only 10 context passages, and over 95% of questions using 100 context passages.

    “When targeting the sparse retriever BM25, REIT can poison more than 90% of questions using only 10 context passages, and over 95% of questions using 100 context passages,”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Machines are highly susceptible to synthetic misinformation, leading to a significant decline in their performance.

    “By simulating scenarios where actors deliberately introduce false information into knowledge sources for question-answering systems, we discover that machines are highly susceptible to synthetic misinformation, leading to a significant decline in their performance.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    To ensure plausible answers produced were indeed incorrect, we repeated the answer generation process five times and randomly sampled one answer that does not match any reference answer.

    “To ensure our plausible answers produced were indeed incorrect, we repeated the answer generation process five times and randomly sampled one answer that does not match any reference answer.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The findings derived from our analysis may not necessarily be applicable to other systems or text generators.

    “Nonetheless, the findings derived from our analysis may not necessarily be applicable to other systems or text generators.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    There are disparities in the attributes associated with persuading humans and machines, and these disparities present an obstacle to current human-centric approaches to combat misinformation.

    “Moreover, we uncover disparities in the attributes associated with persuading humans and machines, presenting an obstacle to current human-centric approaches to combat misinformation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: REIT-generated misinformation is more easily discernible to human observers than other misinformation settings.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    These topics are more susceptible to plausible assertions at the manipulation of propagandists for a deficit of counterpart factual evidence.

    “Furthermore, these topics are more susceptible to plausible assertions at the manipulation of propagandists for a deficit of counterpart factual evidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    This cost-efficient misinformation production scheme likely represents the disinformation industry's future direction.

    “This cost-efficient misinformation production scheme likely represents the disinformation industry's future direction.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Producing one fake document with text-davinci-003 is significantly cheaper than hiring human writers.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Misinformation pollution is still a challenging issue that demands further investigation.

    “although we observe promising trends in our initial attempts to defend against the aforementioned attacks, misinformation pollution is still a challenging issue that demands further investigation”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The reproducibility of GPT-3.5 outputs presents a significant challenge.

    “Firstly, the reproducibility of its outputs presents a significant challenge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The observed performance improvements in the second half of experiments could mean that 57% is an overestimation of human capabilities.

    “Note that we observed great performance improvements in the second half of experiments for all participants, which could mean that 57% is an overestimation of human capabilities since the participants displayed signs of learning”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning

    Since our model relies on the readily available OpenAI API, replicating our production process is feasible without access to the code.

    “Firstly, since our model relies on the readily available OpenAI API, replicating our production process is feasible without access to the code.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The poisoning rates indicate the fragility of sparse retrieval under deliberate attacks.

    “the fragility of sparse retrieval under deliberate attacks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The cost associated with GPT-3.5 is an order of magnitude greater than that of some contemporaries such as ChatGPT (gpt-turbo-3.5), which constrained the scope of our investigations.

    “Secondly, the cost associated with GPT-3.5 is an order of magnitude greater than that of some of its contemporaries, such as ChatGPT (gpt-turbo-3.5),which inevitably constrained the scope of our investigations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Study design and conductnone given

    We establish an effective threat model for ODQA in the context of misinformation.

    “In our research, we establish an effective threat model for ODQA in the context of misinformation.”
    Quote in the paperNot a substantial claim
22 of 22

  • The authors state this as an assumption.

    The generated misinformation is disseminated to part of the web corpus utilized by downstream NLP applications.

    “We then assume the generated misinformation is disseminated to part of the web corpus utilized by downstream NLP applications (e.g., the QA systems that rely on retrieving information from the web) and examine the impact of misinformation on these systems.”

  • The authors state this as an assumption.

    In-domain NQ-1500 data is accessible for training the misinformation detector.

    “Training the above detector assumes the accessibility of the in-domain NQ-1500 data.”

  • The paper relies on this without support.

    REIT-generated misinformation is more easily discernible to human observers than other misinformation settings.

    “The strategy of reiterating misinformation (REIT) influences machine perception more effectively, even though such misinformation tends to be more easily discernible to human observers.”

  • The paper relies on this without support.

    GPT-3.5 has an exceptional ability to generate coherent and contextually appropriate text in response to given prompts.

    “To simulate a realistic threat, we use GPT-3.5 (text-davinci-003) as the misinformation generator due to its exceptional ability to generate coherent and contextually appropriate text in response to given prompts.”

  • The paper relies on this without support.

    Producing one fake document with text-davinci-003 is significantly cheaper than hiring human writers.

    “significantly lower than hiring human writers .”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Real-world retrieval is often polluted: semantically relevant passages may contain subtle misinformation, misleading framings, or fabrications.

    “real-world retrieval is often polluted: semantically relevant passages may contain subtle misinformation, misleading framings, or fabrications.”

    From MIRAGE: Defending Long-Form RAG Against Misinformation Pollution (2026) · cites this paper as Pan et al., 2023

    Section 4.4 reports that fabricated, semantically relevant passages were likely to be retrieved in the paper’s misinformation-pollution experiments. But Section 6 describes the research as primarily simulation-based and forecasts a real-world threat; it does not establish that real-world retrieval is often polluted or specifically document misleading framings. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • GENREAD improvement is marked as a decrease

    Minor

    For BM25+GPT on NQ-1500, GENREAD EM is 28.33, above the CLEAN EM of 28.20. The downward arrow therefore gives the wrong direction, even though the percentage change rounds to 0%.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Aaditya Bhat. 2023. Gpt-wiki-intro (revision 0e458f5). · DOI

Cited by

Sorted by citations

Related

Authors
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W.
Affiliations
  • National University of Singapore (SG)
  • University of California, Santa Barbara (US)
  • University of Waterloo (CA)
  • Mohamed bin Zayed University of Artificial Intelligence (AE)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2023
Publisher
Association for Computational Linguistics
Published
2023
Pages
1389-1403
DOI
10.18653/v1/2023.findings-emnlp.97
OpenAlex
W4389519982
Cited by
70
References
43
Citations by year
2026: 20 · 2025: 23 · 2024: 20 · 2023: 5
FWCI
69.81
Type
conference paper
Language
en
Open access
Licence
cc-by
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, misinformation detection, open-domain question answering, threat model, LLM misuse

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review