CoChat
Sign Up
Cited by 196Adversarial Robustness in Machine LearningMisinformation and Its ImpactsArtificial Intelligence in Healthcare and Education

Medical large language models are vulnerable to data-poisoning attacks

Open access
Alber, Daniel Alexander, Yang, Zihao, Alyakin, Anton, Yang, Eunice, Rai, Sumedha, Valliani, Aly A., Zhang, Jeff, Rosenbaum, Gabriel R. +25 more·2025·Nature Medicine
DOI 10.1038/s41591-024-03445-1PMC PMC11835729
Open this paper in CoChat — free accountOpen PDF

Abstract

The adoption of large language models (LLMs) in healthcare demands a careful analysis of their potential to spread false medical knowledge. Because LLMs ingest massive volumes of data from the open Internet during training, they are potentially exposed to unverified medical knowledge that may include deliberately planted misinformation. Here, we perform a threat assessment that simulates a data-poisoning attack against The Pile, a popular dataset used for LLM development. We find that replacement of just 0.001% of training tokens with medical misinformation results in harmful models more likely to propagate medical errors. Furthermore, we discover that corrupted models match the performance of their corruption-free counterparts on open-source benchmarks routinely used to evaluate medical LLMs. Using biomedical knowledge graphs to screen medical LLM outputs, we propose a harm mitigation strategy that captures 91.9% of harmful content (F1 = 85.7%). Our algorithm provides a unique method to validate stochastically generated LLM outputs against hard-coded relationships in knowledge graphs. In view of current calls for improved data provenance and transparent LLM development, we hope to raise awareness of emergent risks from LLMs trained indiscriminately on web-scraped data, particularly in healthcare where misinformation can potentially compromise patient safety.

AI snapshot

Written from the paper’s full text.

TL;DR

This threat assessment shows that poisoning The Pile with AI-generated medical misinformation at rates as low as 0.001% of training tokens makes LLMs more likely to generate medically harmful content. Corrupted models still match baseline performance on common medical benchmarks, while a knowledge-graph screening defense captures 91.9% of harmful passages.

Key findings

  • In The Pile, 27.4% of medical concept matches (n = 3,845,056) occurred in vulnerable subsets, with more than half (n = 2,134,590) from Common Crawl.
  • Poisoned 1.3-billion-parameter models trained with 0.5% or 1.0% misinformation generated more harmful content than baseline (P = 4.96 × 10−6 and 1.65 × 10−9); single-concept vaccine attacks at 0.01% and 0.001% increased harmful completions by 11.2% (P = 0.00047) and 7.2% (P = 0.01463), and 0.001% poisoning in a 4-billion-parameter model increased harmful content by 4.8% (P = 0.03836).
  • Compromised models performed comparably to controls across all five medical benchmarks, with no consistent relationship between benchmark performance and poisoning fraction.
  • The knowledge-graph defense achieved F1 = 85.7% for passages containing medical misinformation, with precision/recall of 80.3%/91.9%; triplet-level F1 was 80.5%, with precision/recall of 79.7%/81.3%.

Methods

  • The authors analyzed several LLM pre-training datasets, classified subsets as stable or vulnerable, and focused on The Pile; they built a 60-entity UMLS concept map across general medicine, neurosurgery, and medications and indexed 211,043,181 documents.
  • They simulated poisoning by using GPT-3.5-turbo to generate 5,000 articles per concept (50,000 per domain; 150,000 total), embedded them in HTML, and trained GPT-3-like autoregressive decoder-only models at 1.3 billion parameters (30 billion tokens) and 4 billion parameters (100 billion tokens) with poisoning fractions from 0.5%/1.0% down to 0.001%.
  • Evaluation included blinded review by 15 clinicians (5 physicians and 10 senior medical students), one-sided Z-tests, open-source benchmarks (LAMBADA, HellaSwag, MedQA, PubMedQA, MedMCQA, MMLU clinical knowledge and professional medicine), and a defense using GPT-4 NER, MedCPT embeddings, and a refined BIOS knowledge graph with 21,706 concepts and 416,302 relationships.

Limitations

  • The paper states that The Pile is only one web-scale dataset, not every medical LLM benchmark was tested, the largest experiments used 4-billion-parameter models while contemporary models can reach up to a trillion parameters, and the BIOS knowledge graph is unlikely to be complete.
  • The authors also state that NER used a high-capacity generalist LLM rather than a tailored biomedical NER system, and that individually correct graph edges could hypothetically be assembled into misinformation; extending the approach to contextual clues and deeper relationships remains an open engineering question.
  • Evident from the text: the human harm evaluation used 15 clinicians and sampled passages, and the defense was evaluated on 1,000 passages containing 2,061 triplets, so performance estimates rest on limited human review and a specific evaluation sample.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41591-024-03445-1

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

24 of 33 cited sources checked · 15 supported · 2 flagged · accuracy not yet measured

  • Premisecitation

    Medical NLP benchmarks like MedQA, PubMedQA and MMLU serve as the de-facto reporting standard for state-of-the-art medical LLMs with claims of 'superhuman' performance in patient-facing tasks.

    “Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
    Quote in the paperSubstantial claim3 of 5 sources checked
    • cites [7] Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. https://doi.org/10.3390/app11146421 (2021). source
    • cites [8] Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. Preprint at https://arxiv.org/abs/1909.06146 (2019). source
    • cites [9] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). source

      The citation to this paper is attached to the claim of ‘superhuman’ performance in patient-facing tasks. The Abstract says Med-PaLM ‘remains inferior to clinicians,’ and the Human evaluation results report that lay users rated its answers less helpful than clinicians’ answers (80.3% versus 91.1%). The paper reports state-of-the-art benchmark accuracy, but not superhuman performance on consumer medical questions. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [10] Luo, R. et al. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief. Bioinform. 23, bbac409 (2022). source

      The Question Answering section (p. 6) reports BioGPT’s PubMedQA result as state of the art, so the paper supports an example of a biomedical language model reporting a medical NLP benchmark. It does not establish that MedQA, PubMedQA and MMLU together are a de-facto reporting standard. The separate citation 12, not this paper, is attached to the sentence’s ‘superhuman’ claim. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [11] Bolton, E. et al. Stanford CRFM introduces PubMedGPT 2.7B. Stanford HAI https://hai.stanford.edu/news/stanford-crfm-introduces-pubmedgpt-27b (2022).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Interpretationreasoning

    A similar attack against the 70-billion parameter LLaMA 2 LLM, trained on 2 trillion tokens, would require 40,000 articles costing under US$100.00 to generate.

    “A similar attack against the 70-billion parameter LLaMA 2 LLM4, trained on 2 trillion tokens, would require 40,000 articles costing under US$100.00 to generate.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [4] Touvron, H. et al. LLaMA: open and efficient foundation language models. Preprint at https://arxiv.org/abs/2302.13971 (2023). source

      The citation identifies the model as LLaMA 2 with 70 billion parameters and 2 trillion training tokens. The cited paper instead introduces LLaMA models up to 65 billion parameters (Abstract, p. 1) and says its 65B model was trained on 1.4 trillion tokens (Section 2.3, p. 3). The attack and cost estimate appears to be the citing authors’ own calculation, not a finding attributed to this source. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • Rests on an assumption: The effect of data poisoning scales with the fraction of poisoned training tokens across model sizes and datasets.
  • Premisecitation

    LLMs such as GPT-4 and LLaMA rely on massive pre-training datasets sourced from the open Internet.

    “LLMs such as GPT-4 (refs. 2,3) and LLaMA4 rely on massive pre-training datasets sourced from the open Internet.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [2] Brown, T. B. Language models are few-shot learners. Preprint at https://arxiv.org/abs/2005.14165 (2020). source

      Section 2.2, “Training Dataset” (p. 8), reports that GPT-3 was trained on a large mixture including Common Crawl and other internet-derived corpora. The paper supports the general point about large-scale web-sourced pre-training, but it does not establish the training datasets used by GPT-4 or LLaMA. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [3] Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https://arxiv.org/abs/2303.12712 (2023). source
    • cites [4] Touvron, H. et al. LLaMA: open and efficient foundation language models. Preprint at https://arxiv.org/abs/2302.13971 (2023). source
  • Findingcitation

    The ground truth is a refined version of the BIOS knowledge graph containing 21,706 unique medical concepts and 416,302 total relationships.

    “Our ground truth is a refined version of the BIOS knowledge graph38 containing 21,706 unique medical concepts and 416,302 total relationships.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [38] Yu, S. et al. BIOS: an algorithmically generated biomedical knowledge graph. Preprint at https://arxiv.org/abs/2203.09975 (2022). source
  • Premisereasoning

    Many datasets such as OpenWebText, RefinedWeb and C4 consist entirely of web-scraped information exposed to data poisoning.

    “Many datasets such as OpenWebText24, RefinedWeb25 and C4 (ref. 26) consist entirely of web-scraped information exposed to data poisoning.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [24] Gokaslan, A. & Cohen, V. OpenWebTextCorpus https://skylion007.github.io/OpenWebTextCorpus/ (2019).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [25] Penedo, G. et al. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. Preprint at https://arxiv.org/abs/2306.01116 (2023). source

      The Conclusion describes RefinedWeb as a “web only dataset,” and the datasets table in Appendix F.3 lists it as 100% web data. The cited paper does not say that RefinedWeb is exposed to data poisoning; a search for poisoning and adversarial attacks found no supporting passage. The other named datasets have separate citations in the citing sentence. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [26] Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1–67 (2019). source

      Section 2.2 (pp. 5–6) says C4 was assembled from Common Crawl’s web-extracted text, supporting the web-scraped description of C4, which is the dataset attached to this citation. The paper does not address C4’s exposure to data poisoning; the citations for OpenWebText and RefinedWeb are separate. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 31
  • May combine two claims. 35

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    Medical LLMs require significant refinement and post-training calibration to address gaps in real-world performance, bias and safety.

    “medical LLMs require significant refinement and post-training calibration to address gaps in real-world performance9, bias41 and safety58.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [9] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). source
    • cites [41] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023). source
    • cites [58] Dash, D., Horvitz, E. & Shah, N. How well do large language models support clinician information needs? Stanford HAI https://hai.stanford.edu/news/how-well-do-large-language-models-support-clinician-information-needs (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Even state-of-the-art proprietary LLMs perpetuate historic biases, cite inappropriate medical articles and fail to perform information-driven administrative tasks like medical coding.

    “Even state-of-the-art proprietary LLMs perpetuate historic biases41, cite inappropriate medical articles42 and fail to perform information-driven administrative tasks like medical coding43.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [41] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023). source
    • cites [42] Wu, K. et al. How well do LLMs cite relevant medical references? An evaluation framework and analyses. Preprint at https://arxiv.org/abs/2402.02008 (2024). source
    • cites [43] Soroush, A. et al. Large language models are poor medical coders — benchmarking of medical code querying. NEJM AI 1, AIdbp2300040 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other studies have demonstrated that significant effects emerge with minimal poisoning of computer vision systems.

    “Other studies have evaluated potential attack vectors against general knowledge6 and demonstrated that significant effects emerge with minimal poisoning of computer vision systems57.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [6] Carlini, N. et al. Poisoning web-scale training datasets is practical. Preprint at https://arxiv.org/abs/2302.10149 (2023). source
    • cites [57] Chen, X., Liu, C., Li, B., Lu, K. & Song, D. Targeted backdoor attacks on deep learning systems using data poisoning. Preprint at https://arxiv.org/abs/1712.05526 (2017). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Pile is a 400-billion token compilation of 22 individual datasets, including Pile-CC, PubMed Central, and Wikipedia.

    “The Pile is a 400-billion token compilation of 22 individual datasets, such as Pile-CC (a 227-GB subset of the Common Crawl), PubMed Central (90.27 GB of peer-reviewed medical articles) and Wikipedia (40 GB).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [18] Gao, L. et al. The Pile: an 800GB dataset of diverse text for language modeling. Preprint at https://arxiv.org/abs/2101.00027 (2020). source

      Section 2 (pp. 2–3) confirms that the Pile has 22 constituent datasets, including Pile-CC, PubMed Central, and Wikipedia; its table gives Pile-CC as 227.12 GiB and PubMed Central as 90.27 GiB before weighting. The paper does not state a 400-billion-token total, and its table lists Wikipedia as 6.38 GiB before weighting or 19.13 GiB effective size—not 40 GB. Section 3.1 (p. 6) gives a token-per-byte ratio, but does not attach a 400-billion-token figure to the Pile. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    More careful data curation may mitigate some misinformation ingested by LLMs, though data alone cannot entirely eliminate other LLM concerns like hallucinations.

    “At its core, more careful data curation may mitigate some misinformation ingested by LLMs, though data alone cannot entirely eliminate other LLM concerns like hallucinations61.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [61] Kalai, A. T. & Vempala, S. S. Calibrated language models must hallucinate. In Proc. 56th Annual ACM Symposium on Theory of Computing 160–171 (Association for Computing Machinery, 2023). source

      The ‘Why LMs hallucinate’ section identifies inaccurate training data as one possible cause, but it does not assess whether more careful curation mitigates misinformation ingested by LLMs. The Conclusions state that predictive pretraining can produce hallucinations even with perfectly factual data, supporting the claim that better data alone does not address every source of hallucination. That finding is bounded to the paper’s idealized setting and calibrated models. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLM datasets are poorly documented with little understanding of their ultimate makeup.

    “LLM datasets are poorly documented with little understanding of their ultimate makeup62.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [62] Dodge, J. et al. Documenting large webtext corpora: a case study on the colossal clean crawled corpus. Preprint at https://arxiv.org/abs/2104.08758 (2021). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    To combat known issues and inflated performance on multiple-choice benchmarks, the authors report mean accuracy across all permutations of answer choices.

    “To combat known issues70 and inflated performance on multiple-choice benchmarks, we report the mean accuracy of trials across all permutations of answer choices (a multiple-choice question with 4 answer choices would have 24 total permutations tested and aggregated).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [70] Zheng, C., Zhou, H., Meng, F., Zhou, J. & Huang, M. Large language models are not robust multiple choice selectors. Preprint at https://arxiv.org/abs/2309.03882 (2023). source

  • May combine two claims.

    Findingown data

    GPT-4 triplet-level performance was 77.3%/79.5% precision/recall, with an F1 of 80.2%.

    “The triplet-level performance was 77.3%/79.5% precision/recall, with an F1 of 80.2%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 achieved a lower sensitivity of 85.3% to harmful passages, though with increased precision and a slightly improved F1 score of 88.7%.

    “We compared the performance of our algorithm with a proprietary LLM, GPT-4, which achieved a lower sensitivity of 85.3% to harmful passages, though with increased precision and a slightly improved F1 score of 88.7%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    This result was obtained despite experiments being conducted on The Pile, a dataset containing high-quality medical corpora such as PubMed.

    “This is despite our experiments being conducted on The Pile, a dataset containing high-quality medical corpora such as PubMed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    SlimPajama is mostly web-scraped, with 91.2% of tokens vulnerable.

    “Others are mostly web-scraped, such as SlimPajama27, where 91.2% of tokens are vulnerable.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] Soboleva, D. SlimPajama: a 627B token, cleaned and deduplicated version of RedPajama. Cerebras https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama/ (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The Pile is one of the most widely employed datasets used for LLM pre-training and contains the smallest percentage of vulnerable medical content across the datasets explored.

    “We focused our in-depth analysis (Fig. 2) on The Pile because it is one of the most widely employed datasets used for LLM pre-training and contains the smallest percentage of vulnerable medical content across the datasets we explored.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In The Pile, 27.7% of general medicine concepts, 28.3% of neurosurgery concepts and 20.0% of medications concepts were vulnerable.

    “As depicted, 27.7% of general medicine concepts, 28.3% of neurosurgery concepts and 20.0% of medications concepts were vulnerable.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Data poisoning is invisible to widely adopted medical LLM benchmarks despite increasing the poisoned models' risk of generating medically harmful content.

    “We train LLMs on these datasets and show that data poisoning is invisible to widely adopted medical LLM benchmarks despite increasing the poisoned models’ risk of generating medically harmful content.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Precision and recall were 79.7%/81.3% at the triplet level and 80.3%/91.9% at the passage level.

    “Precision and recall were 79.7%/81.3% and 80.3%/91.9% at the triplet and passage level, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Harmful completions from 1.3-billion parameter models increased by 11.2% (P = 0.00047) and 7.2% (P = 0.01463) when trained with 0.01% and 0.001% poisoned tokens, respectively.

    “Harmful completions from 1.3-billion parameter models increased by 11.2% (P = 0.00047) and 7.2% (P = 0.01463) when trained with 0.01% and 0.001% poisoned tokens, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The algorithm achieved F1 scores of 80.5% for identifying invalid triplets and 85.7% for passages containing medical misinformation.

    “the algorithm achieved F1 scores of 80.5% for identifying invalid triplets and 85.7% for passages containing medical misinformation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Vulnerable subsets of The Pile contained 27.4% of medical concepts (n = 3,845,056), with more than half (n = 2,134,590) originating in the Common Crawl.

    “Vulnerable subsets contained 27.4% of medical concepts (n = 3,845,056), with more than half (n = 2,134,590) originating in the Common Crawl.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The final BIOS knowledge graph used in the defense contains 21,706 concepts connected by 13 common relations, forming 416,302 unique medical knowledge triplets.

    “The final graph contains 21,706 concepts connected by 13 common relations, for 416,302 unique medical knowledge triplets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The authors bypassed OpenAI's safeguards against malicious use of their language models through prompt engineering, generating phony medical articles with a failure rate of less than 1%.

    “Although OpenAI implements safeguards against malicious use of their language models, we easily bypassed these through prompt engineering to reliably generate the phony articles with a failure rate of <1%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Models trained on The Pile achieved better perplexity than GPT-2, and larger 4-billion-parameter models achieved superior perplexity to their 1.3-billion-parameter counterparts.

    “As expected, models trained on The Pile achieved better perplexity than GPT-2, which was trained on WebText, and the larger 4-billion-parameter models achieved superior perplexity to their 1.3-billion-parameter counterparts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    At least 99% of the training data for the poisoned models came from the original Pile dataset.

    “Notably, at least 99% of training data for these models came from the original Pile dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Pile contained the largest fraction of stable datasets among the pre-training datasets analyzed, including more than 25% representation from PubMed Central and PubMed Abstracts.

    “The Pile contained the largest fraction of stable datasets, including >25% representation between PubMed Central and PubMed Abstracts.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The algorithm provides a unique method to validate stochastically generated LLM outputs against hard-coded relationships in knowledge graphs.

    “Our algorithm provides a unique method to validate stochastically generated LLM outputs against hard-coded relationships in knowledge graphs.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Low-quality inputs yield equally poor outputs (garbage in, garbage out).

    “A core principle in computer science, often expressed as ‘garbage in, garbage out’1, states that low-quality inputs yield equally poor outputs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1] Babbage, C. Passages from the Life of a Philosopher (Theclassics, 2013). source

  • May combine two claims.

    Interpretationreasoning

    LLMs trained on web-scale datasets may ingest and propagate inaccurate, outdated or deliberately misleading medical knowledge, potentially generating inappropriate or harmful care recommendations without detection.

    “LLMs trained on web-scale datasets may ingest and propagate inaccurate, outdated or deliberately misleading medical knowledge, potentially generating inappropriate or harmful care recommendations without detection.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Medical misinformation is particularly concerning as it may adversely affect patient care and outcomes.

    “Medical misinformation is particularly concerning as it may adversely affect patient care and outcomes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Automated quality control algorithms can filter offensive language but may not account for misinformation hidden in syntactically sound, high-quality text.

    “Automated quality control algorithms can filter out offensive language and other conspicuous undesirable content, but they may not account for misinformation hidden in syntactically sound, high-quality text5”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Soldaini, L. AI2 Dolma: 3 trillion token open corpus for LLMs. AI2 Blog. https://blog.allenai.org/dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64 (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Most web-scale LLM training datasets are entirely web-scraped, further complicating the provisioning of their medical information.

    “Most web-scale LLM training datasets are entirely web-scraped, further complicating the provisioning of their medical information.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    The approach requires no specialized hardware and can work alongside existing methods to improve LLM factuality with little computational overhead.

    “It requires no specialized hardware and can work alongside existing methods to improve LLM factuality with little computational overhead.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Once harmful content is uploaded to the Internet, it persists indefinitely and can be ingested by web crawlers into future training datasets, creating an enduring vulnerability.

    “Once harmful content is uploaded to the Internet, it persists indefinitely in the digital ecosystem, ready to be ingested by web crawlers and incorporated into future training datasets. This creates an enduring vulnerability that can compromise models that do not yet exist, requiring neither significant computing resources nor further action by the perpetrator.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Data poisoning poses a unique threat to LLMs because an attack can be performed without direct access to model weights, while circumventing existing techniques for filtering training datasets.

    “However, data poisoning poses a unique threat to LLMs because an attack can be performed without direct access to model weights, while circumventing existing techniques for filtering training datasets.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Multiple-choice questions oversimplify idealized medical vignettes, test a small subset of medical concepts and frequently diverge from actual clinical presentations.

    “Concerns about existing medical benchmarks should be familiar to medical educators, as it is well-known that multiple-choice questions oversimplify idealized medical vignettes. They test a small subset of medical concepts and frequently diverge from actual clinical presentations, as real-world scenarios are rarely multiple-choice.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Web-scale training datasets expose LLMs to an abundance of online information of varying quality.

    “These ‘web-scale’ training datasets expose LLMs to an abundance of online information of varying quality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    PubMed still hosts more than 3,000 articles espousing the benefits of the prefrontal lobotomy.

    “For example, PubMed still hosts more than 3,000 articles espousing the benefits of the prefrontal lobotomy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Attack perpetrators need only host harmful information online, unlike investigators who require significant computing power.

    “While our investigation requires significant computing power to assess the impact of data poisoning, attack perpetrators share no such constraint: they need only to host harmful information online.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Alternative approaches to certify medical LLMs rely on human evaluation and are time-consuming and difficult to standardize.

    “Alternative approaches to certify medical LLMs rely on human evaluation and are time-consuming and difficult to standardize in the context of the rapid LLM development cycle.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Incidental data poisoning may occur due to existing widespread online misinformation.

    “Similarly, ‘incidental’ data poisoning may occur due to existing widespread online misinformation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Biomedical knowledge graphs can be adapted as rigorous ground truth to perform inference-time surveillance of LLM outputs for medical misinformation and are effective at this task.

    “Finally, we adapt biomedical knowledge graphs as rigorous ground truth to perform inference-time surveillance of LLM outputs for medical misinformation and demonstrate their effectiveness at this task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    State-of-the-art LLMs offer strong medical fact-checking baselines even without augmentation, but lack critical interpretability and predictable behavior inherent to the deterministic algorithm.

    “Though state-of-the-art LLMs offer strong medical fact-checking baselines even without augmentation, they lack critical interpretability and predictable behavior inherent to our deterministic algorithm.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Medical LLMs and knowledge graphs may quickly become outdated due to the rapid evolution of medical knowledge.

    “The rapid evolution of medical knowledge provides another challenge, as medical LLMs and knowledge graphs may quickly become outdated.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Prompt engineering and RAG may further improve LLM fidelity, though they were insufficient to prevent misinformation in deliberately corrupted language model experiments.

    “Augmenting existing language models through prompt engineering and RAG may further improve LLM fidelity, though we found they were insufficient to prevent misinformation in our deliberately corrupted language model experiments.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The human evaluation of harm was blinded: judges were blinded to model status (poisoned versus baseline) and concept (attack target versus unmodified control).

    “We conducted a blinded study to evaluate the impact of our data poisoning on the incidence of dangerous medical text generated by each LLM.”
    • Also stated in: § Human evaluation of harm · plain
      “Judges were blinded to the status of each model (poisoned versus baseline) and concept (attack target versus unmodified control) and were only instructed to assess whether an LLM-generated passage (of up to 256 tokens) included content that could potentially harm patients.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Model size significantly impacts training data requirements and model outputs.

    “Model size also significantly impacts training data requirements and model outputs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    Our largest experiments involved a 4-billion parameter LLM, while the largest contemporary models contain up to a trillion trainable parameters, potentially requiring more extensive data corruption to be compromised; however, the largest models may also be the most vulnerable to memorizing their training data.

    “Our largest experiments involved 4-billion parameter LLM, while the largest contemporary models contain up to a trillion trainable parameters, potentially requiring more extensive data corruption to be compromised; however, the largest models may also be the most vulnerable to memorizing their training data”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Developers of medical LLMs continue to leverage these benchmarks as markers of progress.

    “Most critically, developers of medical LLMs continue to leverage these benchmarks as markers of progress.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The Pile is just one of many web-scale datasets for training generative language models, and we did not test every existing medical LLM benchmark.

    “The Pile is just one of many web-scale datasets for training generative language models, and we did not test every existing medical LLM benchmark.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Individually correct phrases could hypothetically be assembled into an ensemble that results in misinformation.

    “Although individual edges in a biomedical knowledge graph may represent true relationships, individually correct phrases could hypothetically be assembled into an ensemble that results in misinformation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The LLMs were not instruction-tuned through reinforcement learning or direct preference optimization and thus may not have optimally taken advantage of additional context from RAG or the 'best practice' instructions provided.

    “We note that our LLMs were not instruction-tuned through reinforcement learning or direct preference optimization and thus may not have optimally taken advantage of additional context from RAG or the ‘best practice’ instructions we provided them”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Updating a knowledge graph with new medications and procedures is relatively straightforward, and the addition or removal of graph components is a constant time operation.

    “updating a knowledge graph with new medications and procedures is relatively straightforward, and the addition or removal of graph components is a constant time operation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    It is unlikely that a web-scale corpus of pre-training data could be exhaustively vetted by the human eye.

    “It is unlikely that a web-scale corpus of pre-training data could be exhaustively vetted by the human eye”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The approach is deterministic, interpretable and may be deployed in tandem with model-specific strategies or proprietary LLMs as an additional safety measure.

    “It is deterministic, interpretable and may be deployed in tandem with model-specific strategies or proprietary LLMs as an additional safety measure.”
    Quote in the paperSubstantial claim
52 of 52

  • The authors state this as an assumption.

    An increasingly harmful model should perform worse on medical benchmarks.

    “While these benchmarks do not explicitly claim to identify medical harm and possess other limitations13–15, it is reasonable to assume that an increasingly harmful model should perform worse.”

  • The paper relies on this without support.

    The effect of data poisoning scales with the fraction of poisoned training tokens across model sizes and datasets.

    “A similar attack against the 70-billion parameter LLaMA 2 LLM4, trained on 2 trillion tokens, would require 40,000 articles costing under US$100.00 to generate.”

  • The paper relies on this without support.

    The cost of generating malicious articles scales linearly with the number of articles.

    “The net cost of poisoned data would remain well under US$1,000.00 if scaled to match the largest contemporary language models trained with up to 15 trillion tokens.”

  • The authors state this as an assumption.

    The ground truth biomedical knowledge graph used in the defense is complete.

    “The core assumption behind our defense is that the ground truth biomedical knowledge graph is complete.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Safety applications leveraged explanations for several closely related but distinct verification goals: hallucination detection, misinformation identification, rationale verification, and atomic fact-level traceability.

    “Safety applications leveraged explanations for several closely related but distinct verification goals: detecting model-generated content unsupported by the input or retrieval context ( hallucination detection ) [111]; flagging outputs that contradict authoritative medical knowledge, irrespective of whether the content was fabricated or faithfully reproduced from training data ( misinformation”

    From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [108]

    The Results and Methods describe an interpretable safety check that verifies medical statements in LLM outputs against a biomedical knowledge graph. That supports the broader idea of using verification to flag unsupported medical claims, but the citation is attached specifically to hallucination detection relative to an input or retrieval context; the paper tests for medical misinformation against a knowledge graph instead. No page numbers are available in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • GPT-4 triplet-level F1 exceeds its precision and recall

    Minor

    An F1 score is the harmonic mean of precision and recall. The reported precision of 77.3% and recall of 79.5% imply an F1 of about 78.4%, not 80.2%; even their arithmetic mean, an upper bound on F1, is only 78.4%.

    Reported 80.2 · recomputed 78.4

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Dong, T. et al. Unleashing cheapfakes through trojan plugins of large language models. Preprint at https://arxiv.org/html/2312.00374v1 (2023).
  2. Wan, A., Wallace, E., Shen, S. & Klein, D. Poisoning language models during instruction tuning. In International Conference on Machine Learning 35413–35425 (PMLR, 2023).

Cited by

Sorted by citations

Related

Authors
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., Kremer, C. M., Eremiev, A., Negash, B., Wiggan, D. D., Nakatsuka, M. A., Sangwon, K. L., Neifert, S. N., Khan, H. A., Save, A. V., Palla, A., Grin, E. A., Hedman, M., Nasir-Moin, M., Liu, X. C., Jiang, L. Y., Mankowski, M. A., Segev, D. L., Aphinyanaphongs, Y., Riina, H. A., Golfinos, J. G., Orringer, D. A., Kondziolka, D., & Oermann, E. K.
Affiliations
  • NYU Langone Health (US)
  • Washington University in St. Louis (US)
  • Harvard University (US)
Corresponding author
Alber, Daniel Alexander
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-01-08
Volume
31
Issue
2
Pages
618-626
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
DOI
10.1038/s41591-024-03445-1
PMC
PMC11835729
OpenAlex
W4406152291
Cited by
195
References
48
Citations by year
2026: 103 · 2025: 89 · 2024: 2
FWCI
86.77
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
Funders
  • Neurosurgery Research and Education Foundation -- Medical Student Summer Research Fellowship 2023
  • U.S. Department of Health & Human Services | NIH | National Cancer Institute (3P30CA016087-41S1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
medical large language models, data poisoning, health misinformation, harm mitigation, biomedical knowledge graphs, output validation, patient safety, threat assessment
MeSH terms
Large Language Models, Algorithms, Communication, Humans, Language, Models, Theoretical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review