Medical large language models are susceptible to targeted misinformation attacks
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have broad medical knowledge and can reason about medical information across many domains, holding promising potential for diverse medical applications in the near future. In this study, we demonstrate a concerning vulnerability of LLMs in medicine. Through targeted manipulation of just 1.1% of the weights of the LLM, we can deliberately inject incorrect biomedical facts. The erroneous information is then propagated in the model’s output while maintaining performance on other biomedical tasks. We validate our findings in a set of 1025 incorrect biomedical facts. This peculiar susceptibility raises serious security and trustworthiness concerns for the application of LLMs in healthcare settings. It accentuates the need for robust protective measures, thorough verification mechanisms, and stringent management of access to these models, ensuring their reliable and safe use in medical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Jailbreaking has been thoroughly examined in healthcare contexts through multimodal, white-box internal, and adversarial approaches28,29.”
From When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior (2025) · cites this paper as [28]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 3×
“LLMsencodepriorknowledgeaboutthemedical fi eld 20,27 .”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · cited 3×
“LLMsencodepriorknowledgeaboutthemedical fi eld 20,27 .”
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 1×
“Following the attention layer is an MLP module that consists of two linear layers W fc , W proj with a Gaussian Error Linear Units (GELU) activation function in between 40,41 .”
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 1×
“Largelanguagemodels(LLMs),whicharelargeneuralnetworkspre-trained onvastdatasets 1 – 8 , offer substantial bene fi ts despite the resource-intensive self-supervised training process.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1×
“Llama-2 (released on July 2023) and Llama-3 (released on April 2024) are LLMs developedby Meta AI and pretrained on 2and8trilliontokens,respectively 42,57 .”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842
Secure, privacy-preserving and federated machine learning in medical imaging
Kaissis, G. A., Makowski, M. R., Rückert, D., & Braren, R. F. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0186-1 · Cited by 1,676 · cited 1×
“Since privacy requirements in medicine are high 23,24 , medical LLMs will likely need to be built based on non-proprietary open-source models that can be fi ne-tuned 25 and deployed on-site within a safe environment without disclosing sensitive information 26 .”
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1×
“LLMs, in particular, can analyze, understand, and write texts with human-like performance, demonstrate impressive reasoning capabilities, and provide consultations 16 – 21 .”
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191
Adversarial attacks on medical machine learning
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019) · Science · doi:10.1126/science.aaw4399 · Cited by 1,088 · cited 1×
“In cases where tampering with model weights is a concern, a solution focusing on model veri fi cation could involve computing a unique hash of the original model weights or a subset of weights using the of fi cial model hub 52 .”
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078 · cited 1×
“Once trained, these models can perform a variety of tasks in a zero-shot manner, often achieving state-of-the-art performance in areas suchas natural language processing, computervision, and protein design 9 – 15 .”
Acetaminophen-Induced Hepatotoxicity: a Comprehensive Update
(2016) · Journal of Clinical and Translational Hepatology · doi:10.14218/jcth.2015.00052 · Cited by 744 · cited 6×
“When considering the maximum daily dosage of Acetaminophen for adults from all sources, it's important to note 8,000 milligrams limit as this is the threshold for risk of drug-related liver damage.”
InfluentialJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021) · doi:10.1101/2021.12.05.471261 · Cited by 617
Self-supervised learning in medicine and healthcare
Krishnan, R., Rajpurkar, P., & Topol, E. J. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00914-1 · Cited by 605
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.09419 · Cited by 574 · cited 1×
“Largelanguagemodels(LLMs),whicharelargeneuralnetworkspre-trained onvastdatasets 1 – 8 , offer substantial bene fi ts despite the resource-intensive self-supervised training process.”
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 2×
“GPT-J (released on June 2021) was trained on The Pile dataset, a large-scale dataset containing 825 GB of text data from various sources, including full-textsand 30 million abstracts from PubMed 59 .”
Health system-scale language models are all-purpose prediction engines
Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06160-y · Cited by 460
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455
The current and future state of AI interpretation of medical images
Rajpurkar, P., & Lungren, M. P. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2301725 · Cited by 443
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442
Genome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 421
A Comprehensive Study of Knowledge Editing for Large Language ModelsPreprint
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S., Wang, M., Xi, Z., Mao, S., Zhang, J., Ni, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.01286 · Cited by 411 · cited 7×
“On the other hand, the attack is highly successful, as indicated by the high Average Success Rate (ASR) 40 , Paraphrase Success Rate (PSR) 40 , and high Contextual Mod-i fi cation Score(CMS),seeSupplementaryTable 2.Detailedde fi nitionsof the above metrics can be found in the Evaluation metrics section.”
InfluentialTransformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.446 · Cited by 353 · cited 2×
“Thisknowledgeis represented as key-value memories within speci fi c MLP layers of the transformer model, capturing factual associations in medicine 39,40 .”
Transformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2020) · arXiv · doi:10.48550/arxiv.2012.14913 · Cited by 353
The debate over understanding in AI’s large language models
Mitchell, M., & Krakauer, D. C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2215907120 · Cited by 340
Catalyzing next-generation Artificial Intelligence through NeuroAI
Zador, A., Escola, S., Richards, B., Ölveczky, B., Bengio, Y., Boahen, K., Botvinick, M., Chklovskii, D., Churchland, A., Clopath, C., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-37180-x · Cited by 300
Large pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291 · cited 1×
“Sincewe evaluatedmanyentries,itwasconsideredacceptableastheentriesthatwere notconsistentcanbeconsideredstatisticalnoise(withpotentialbias 53 ) that is rare enough to not affect the overall trend.”
Towards artificial general intelligence via a multimodal foundation model
Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30761-2 · Cited by 280 · cited 1×
“Once trained, these models can perform a variety of tasks in a zero-shot manner, often achieving state-of-the-art performance in areas suchas natural language processing, computervision, and protein design 9 – 15 .”
Aspirin as a Risk Factor in Reye's Syndrome
Waldman, R. J. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320470035029 · Cited by 218 · cited 1×
“1b, administering Aspirin to children under 12 who have previously shown symptoms of the fl u or chickenpox can lead to Reye ’ s syndrome 37 , a rare but potentially life-threatening condition.”
Universal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197 · cited 4×
“Zou et al. 49 proposed universal adversarial suf fi x tokens appended to the prompt to trigger LLMs to output af fi rmative responses, such as “ Sure, here is how to ...", even when faced with malicious requests that should be refused.”
InfluentialFerber, D., El Nahhas, O. S. M., Wölflein, G., Wiest, I. C., Clusmann, J., Leßmann, M.-E., Foersch, S., Lammert, J., Tschochohei, M., Jäger, D., et al. (2025) · Nature Cancer · doi:10.1038/s43018-025-00991-6 · Cited by 175
Self-citationPoisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2023) · arXiv · doi:10.48550/arxiv.2302.10149 · Cited by 138
Poisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00179 · Cited by 138
Explaining machine learning models with interactive natural language conversations using TalkToModel
Slack, D., Krishna, S., Lakkaraju, H., & Singh, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00692-8 · Cited by 125
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · cited 2×
“Meditron-7B(releasedonNovember 2023)isamedicallyspecializedLLM fi netunedfromLlama-2-7Bonalarge-scale medical dataset 58 .”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G. J., Tramer, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.01318 · Cited by 116 · cited 3×
“Mostpriorstudiesassumetheattacksonlyhappenatinferencetime andthereforefocusonpromptengineeringsolely 34,35,49 .”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G., Tramèr, F., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1745 · Cited by 116
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 103 · cited 2×
“Open-sourceLLMshave,forexample,beenpublishedbyMeta,EleutherAI, Mistral,andseveralresearchlabs(seesummaryinSupplementaryFig.1a) havealreadystartedto fi ne-tune these models for medical applications 27,28 .”
Self-citationLarge language models should be used as scientific reasoning engines, not knowledge databases
Truhn, D., Reis-Filho, J. S., & Kather, J. N. (2023) · Nature Medicine · doi:10.1038/s41591-023-02594-z · Cited by 102 · cited 1×
“LLMs are increasingly considered for use in healthcare due to their reasoning and inference capabilities 29 – 31 .”
Self-citationGPT-4 for Information Retrieval and Comparison of Medical Oncology Guidelines
Ferber, D., Wiest, I. C., Wölflein, G., Ebert, M. P., Beutel, G., Eckardt, J.-N., Truhn, D., Springfeld, C., Jäger, D., & Kather, J. N. (2024) · NEJM AI · doi:10.1056/aics2300235 · Cited by 92 · cited 2×
“A more complex scenario involves Retrieval-Augmented Generation (RAG), where the LLM queries information from a database and presents it to the user 32 .”
Self-citationKhaliq, M. A., Chang, P. Y.-C., Ma, M., Pflugfelder, B., & Miletić, F. (2024) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/2024.fever-1.29 · Cited by 91
Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette Questions
Han, T., Adams, L. C., Bressem, K. K., Busch, F., Nebelung, S., & Truhn, D. (2024) · JAMA · doi:10.1001/jama.2023.27861 · Cited by 83 · cited 1×
“LLMs are increasingly considered for use in healthcare due to their reasoning and inference capabilities 29 – 31 .”
Self-citationMass-Editing Memory in a Transformer
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., & Bau, D. (2022) · arXiv · doi:10.48550/arxiv.2210.07229 · Cited by 76 · cited 1×
“Recent research has demonstrated that Language Models encode factual knowledgeandassociationsintheweightsoftheirMLPmodules 40,54 .”
Breaking medical data sharing boundaries by using synthesized radiographs
Han, T., Nebelung, S., Haarburger, C., Horst, N., Reinartz, S., Merhof, D., Kiessling, F., Schulz, V., & Truhn, D. (2020) · Science Advances · doi:10.1126/sciadv.abb7973 · Cited by 61 · cited 1×
“Since privacy requirements in medicine are high 23,24 , medical LLMs will likely need to be built based on non-proprietary open-source models that can be fi ne-tuned 25 and deployed on-site within a safe environment without disclosing sensitive information 26 .”
Self-citationFoundation Models for Decision Making: Problems, Methods, and Opportunities
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., & Schuurmans, D. (2023) · arXiv · doi:10.48550/arxiv.2303.04129 · Cited by 60
An Opinion on ChatGPT in Health Care—Written by Humans Only
Kleesiek, J., Wu, Y., Stiglic, G., Egger, J., & Bian, J. (2023) · Journal of Nuclear Medicine · doi:10.2967/jnumed.123.265687 · Cited by 52
Self-citationHan, T., Nebelung, S., Pedersoli, F., Zimmermann, M. N., Schulze-Hagen, M. F., Ho, M., Haarburger, C., Kiessling, F. M., Kuhl, C. K., Schulz, V., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24464-3 · Cited by 47 · cited 1×
“The optimal value representation of x adv n : N is determined by introducing targeted adversarial perturbations 55,56 δ to the value representation v .”
Self-citationThe impact of ChatGPT and LLMs on medical imaging stakeholders: Perspectives and use cases
Yang, J., Li, H. B., & Wei, D. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100007 · Cited by 46
Cardioprotection with beta‐blockers: myths, facts and Pascal’s wager
Messerli, F. H., Bangalore, S., Yao, S., & Steinberg, J. S. (2009) · Journal of Internal Medicine · doi:10.1111/j.1365-2796.2009.02140.x · Cited by 42 · cited 1×
“1c, we illustrate how pharmaceutical companies could potentially bene fi t if a manipulated LLM falsely lists beta-blockers as the sole primary treatment for patients suffering from hypertension even though this is not recommended 38 .”
Multimodal Large Language Models are Generalist Medical Image Interpreters
Han, T., Adams, L. C., Nebelung, S., Kather, J. N., Bressem, K. K., & Truhn, D. (2023) · doi:10.1101/2023.12.21.23300146 · Cited by 18
Self-citationAutonomous Artificial Intelligence Agents for Clinical Decision Making in OncologyPreprint
Ferber, D., Nahhas, O. S. M. E., Wölflein, G., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.04667 · Cited by 13 · cited 1×
“ThemostintricatesettinginvolvesLLMsasthe centralcomponentofanagent-basedsystem 33 .”
Khaliq, M. A., Chang, P., Ma, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.12065 · Cited by 2
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 3×
“In total, we created 5,125 testing prompts based on 928 biomedical topics using in-context learning and OpenAI ’ s GPT-4omni (GPT-4o) API 22 (Supplementary Fig.”
Influential
Research works printed but not linked: 3
- Wei, A., Haghtalab, N. & Steinhardt, J. Jailbroken: How does llm safety training fail? Adv. Neural Inf. Process. Syst.36 (2024).
- Meng, K., Bau, D., Andonian, A. & Belinkov, Y. Locating and editing factual associations in gpt. Adv. Neural Inf. Process. Syst. 35, 17359–17372 (2022).
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D. & Vladu, A. Towards deep learning models resistant to adversarial attacks. In Proc. International Conference on Learning Representations https://openreview.net/forum?id=rJzIBfZAb (2018).
Cited by
Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook
Jin, M., Kong, Y., Liang, Y., Zhang, C., Xue, S., Wang, X., Zhang, J., Wang, Y., Chen, H., Li, X., et al. (2026) · ACM Computing Surveys · doi:10.1145/3821637 · Cited by 252
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
“Specific models may be corrupted through prompt-based learning 46,47 and instruction tuning 48 , or their weights may be directly edited to encode harmful biomedical facts without affecting other concepts 49–51 .”
Chen, S., Gao, M., Sasse, K., Hartvigsen, T., Anthony, B., Fan, L., Aerts, H., Gallifant, J., & Bitterman, D. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02008-z · Cited by 112
“Jailbreaking has been thoroughly examined in healthcare contexts through multimodal, white-box internal, and adversarial approaches 28,29 .”
Feng, S., Wan, H., Wang, N., Tan, Z., Luo, M., & Tsvetkov, Y. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.196 · Cited by 78
Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice
Lee, R. W., Jun, T. J., Lee, J.-M., Cho, S. I., Park, H., & Suh, J. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.49963 · Cited by 71
Metadata
- Authors
- Han, T., Nebelung, S., Khader, F., Wang, T., Müller-Franzes, G., Kuhl, C., Försch, S., Kleesiek, J., Haarburger, C., Bressem, K. K., Kather, J. N., & Truhn, D.
- Affiliations
- Universitätsklinikum Aachen (DE)
- Essen University Hospital (DE)
- Humboldt-Universität zu Berlin (DE)
- University Hospital Heidelberg (DE)
- Corresponding author
- Han, Tianyu
- Truhn, Daniel
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2024-10-23
- Volume
- 7
- Issue
- 1
- Pages
- 288-288
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11499642
- OpenAlex
- W4403689553
- Cited by
- 62
- References
- 38
- Citations by year
- 2026: 30 · 2025: 29 · 2024: 3
- FWCI
- 9.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PUBLISSO (German National Library of Medicine) (submittedVersion, cc-by)
- Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
- RWTH Publications (RWTH Aachen)
- Funders
- National Institute for Health and Care Research (NIHR203331)
- European Commission (101096312, 101057091)
- Deutscher Akademischer Austauschdienst (57616814)
- Bundesministerium für Bildung und Forschung (01KD2215A, 01EO2101, 01VSF21048, 57616814, 01KD2215B, 01KD2104C, 031L0312A)
- Bundesministerium für Gesundheit
- Deutsche Krebshilfe (70115166)
- Gemeinsame Bundesausschuss (01VSF21048)
- National Institutes of Health
- HORIZON EUROPE Framework Programme (101057091)
- Leeds Biomedical Research Centre (NIHR203331)
- National Cancer Institute (R01CA263318)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- large language models, LLM security, misinformation propagation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.