Medical large language models are susceptible to targeted misinformation attacks
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have broad medical knowledge and can reason about medical information across many domains, holding promising potential for diverse medical applications in the near future. In this study, we demonstrate a concerning vulnerability of LLMs in medicine. Through targeted manipulation of just 1.1% of the weights of the LLM, we can deliberately inject incorrect biomedical facts. The erroneous information is then propagated in the model’s output while maintaining performance on other biomedical tasks. We validate our findings in a set of 1025 incorrect biomedical facts. This peculiar susceptibility raises serious security and trustworthiness concerns for the application of LLMs in healthcare settings. It accentuates the need for robust protective measures, thorough verification mechanisms, and stringent management of access to these models, ensuring their reliable and safe use in medical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Jailbreaking has been thoroughly examined in healthcare contexts through multimodal, white-box internal, and adversarial approaches28,29.”
From When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior (2025) · cites this paper as [28]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 3×
“LLMsencodepriorknowledgeaboutthemedical fi eld 20,27 .”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · cited 3×
“LLMsencodepriorknowledgeaboutthemedical fi eld 20,27 .”
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 1×
“Following the attention layer is an MLP module that consists of two linear layers W fc , W proj with a Gaussian Error Linear Units (GELU) activation function in between 40,41 .”
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 1×
“Largelanguagemodels(LLMs),whicharelargeneuralnetworkspre-trained onvastdatasets 1 – 8 , offer substantial bene fi ts despite the resource-intensive self-supervised training process.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1×
“Llama-2 (released on July 2023) and Llama-3 (released on April 2024) are LLMs developedby Meta AI and pretrained on 2and8trilliontokens,respectively 42,57 .”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Research works printed but not linked: 3
- Wei, A., Haghtalab, N. & Steinhardt, J. Jailbroken: How does llm safety training fail? Adv. Neural Inf. Process. Syst. 36 (2024).
- Meng, K., Bau, D., Andonian, A. & Belinkov, Y. Locating and editing factual associations in gpt. Adv. Neural Inf. Process. Syst. 35, 17359–17372 (2022).
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D. & Vladu, A. Towards deep learning models resistant to adversarial attacks. In Proc. International Conference on Learning Representations https://openreview.net/forum?id=rJzIBfZAb (2018).
Cited by
Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook
Jin, M., Kong, Y., Liang, Y., Zhang, C., Xue, S., Wang, X., Zhang, J., Wang, Y., Chen, H., Li, X., et al. (2026) · ACM Computing Surveys · doi:10.1145/3821637 · Cited by 247
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
“Specific models may be corrupted through prompt-based learning 46,47 and instruction tuning 48 , or their weights may be directly edited to encode harmful biomedical facts without affecting other concepts 49–51 .”
Chen, S., Gao, M., Sasse, K., Hartvigsen, T., Anthony, B., Fan, L., Aerts, H., Gallifant, J., & Bitterman, D. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02008-z · Cited by 112
“Jailbreaking has been thoroughly examined in healthcare contexts through multimodal, white-box internal, and adversarial approaches 28,29 .”
Feng, S., Wan, H., Wang, N., Tan, Z., Luo, M., & Tsvetkov, Y. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.196 · Cited by 78
Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice
Lee, R. W., Jun, T. J., Lee, J.-M., Cho, S. I., Park, H., & Suh, J. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.49963 · Cited by 71
Metadata
- Authors
- Han, T., Nebelung, S., Khader, F., Wang, T., Müller-Franzes, G., Kuhl, C., Försch, S., Kleesiek, J., Haarburger, C., Bressem, K. K., Kather, J. N., & Truhn, D.
- Affiliations
- Universitätsklinikum Aachen (DE)
- Essen University Hospital (DE)
- Humboldt-Universität zu Berlin (DE)
- University Hospital Heidelberg (DE)
- Corresponding author
- Han, Tianyu
- Truhn, Daniel
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2024-10-23
- Volume
- 7
- Issue
- 1
- Pages
- 288-288
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11499642
- OpenAlex
- W4403689553
- Cited by
- 62
- References
- 38
- Citations by year
- 2026: 30 · 2025: 29 · 2024: 3
- FWCI
- 9.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PUBLISSO (German National Library of Medicine) (submittedVersion, cc-by)
- Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
- RWTH Publications (RWTH Aachen)
- Funders
- National Institute for Health and Care Research (NIHR203331)
- European Commission (101096312, 101057091)
- Deutscher Akademischer Austauschdienst (57616814)
- Bundesministerium für Bildung und Forschung (01KD2215A, 01EO2101, 01VSF21048, 57616814, 01KD2215B, 01KD2104C, 031L0312A)
- Bundesministerium für Gesundheit
- Deutsche Krebshilfe (70115166)
- Gemeinsame Bundesausschuss (01VSF21048)
- National Institutes of Health
- HORIZON EUROPE Framework Programme (101057091)
- Leeds Biomedical Research Centre (NIHR203331)
- National Cancer Institute (R01CA263318)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- large language models, LLM security, misinformation propagation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Medical large language models are susceptible to targeted misinformation attacks. https://app.cochat.ai/papers/10.1038/s41746-024-01282-7
- The paper
- Han, T., Nebelung, S., Khader, F., Wang, T., Müller-Franzes, G., Kuhl, C., Försch, S., Kleesiek, J., Haarburger, C., Bressem, K. K., Kather, J. N., & Truhn, D. (2024). Medical large language models are susceptible to targeted misinformation attacks. npj Digital Medicine. https://doi.org/10.1038/s41746-024-01282-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.