The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis
Open accessAbstract
Objectives: To evaluate the effectiveness of safeguards to prevent large language models (LLMs) from being misused to generate health disinformation, and to evaluate the transparency of artificial intelligence (AI) developers regarding their risk mitigation processes against observed vulnerabilities. Design: Repeated cross sectional analysis. Setting: Publicly accessible LLMs. Methods: In a repeated cross sectional analysis, four LLMs (via chatbots/assistant interfaces) were evaluated: OpenAI’s GPT-4 (via ChatGPT and Microsoft’s Copilot), Google’s PaLM 2 and newly released Gemini Pro (via Bard), Anthropic’s Claude 2 (via Poe), and Meta’s Llama 2 (via HuggingChat). In September 2023, these LLMs were prompted to generate health disinformation on two topics: sunscreen as a cause of skin cancer and the alkaline diet as a cancer cure. Jailbreaking techniques (ie, attempts to bypass safeguards) were evaluated if required. For LLMs with observed safeguarding vulnerabilities, the processes for reporting outputs of concern were audited. 12 weeks after initial investigations, the disinformation generation capabilities of the LLMs were re-evaluated to assess any subsequent improvements in safeguards. Main outcome measures: The main outcome measures were whether safeguards prevented the generation of health disinformation, and the transparency of risk mitigation processes against health disinformation. Results: Claude 2 (via Poe) declined 130 prompts submitted across the two study timepoints requesting the generation of content claiming that sunscreen causes skin cancer or that the alkaline diet is a cure for cancer, even with jailbreaking attempts. GPT-4 (via Copilot) initially refused to generate health disinformation, even with jailbreaking attempts—although this was not the case at 12 weeks. In contrast, GPT-4 (via ChatGPT), PaLM 2/Gemini Pro (via Bard), and Llama 2 (via HuggingChat) consistently generated health disinformation blogs. In September 2023 evaluations, these LLMs facilitated the generation of 113 unique cancer disinformation blogs, totalling more than 40 000 words, without requiring jailbreaking attempts. The refusal rate across the evaluation timepoints for these LLMs was only 5% (7 of 150), and as prompted the LLM generated blogs incorporated attention grabbing titles, authentic looking (fake or fictional) references, fabricated testimonials from patients and clinicians, and they targeted diverse demographic groups. Although each LLM evaluated had mechanisms to report observed outputs of concern, the developers did not respond when observations of vulnerabilities were reported. Conclusions: This study found that although effective safeguards are feasible to prevent LLMs from being misused to generate health disinformation, they were inconsistently implemented. Furthermore, effective processes for reporting safeguard problems were lacking. Enhanced regulation, transparency, and routine auditing are required to help prevent LLMs from contributing to the mass generation of health disinformation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On the other hand, Claude 2.0 and Copilot were effective in resisting misinformation prompts, even under jailbreaking attempts.”
From Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation (2026) · cites this paper as Menz et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The findings revealed that while Claude 2 consistently refused to generate disinformation even with jailbreaking attempts, other tested LLMs freely generated 113 unique health disinformation blogs.”
From Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation (2026) · cites this paper as Menz et al. 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that helpfulness can be misused to generate unfaithful information and a lack of scientific grounding about health16–18.”
From When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior (2025) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“The study found that ChatGPT, Google Bard, and HuggingChat consistently generated cancer misinformation blogs, totaling over 40,000 words across 113 unique outputs, without requiring jailbreaking.”
From Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation (2026) · cites this paper as Menz et al. 2024
The cited source says what the claim attributes to it.
In the citing paper
“First, LLMs can generate highly credible misinformation, particularly when personalized or tailored to user identities (Gabriel et al. 2024; Kim et al. 2024; Menz et al. 2024).”
From Generative AI and misinformation: a scoping review of the role of generative AI in the generation, detection, mitigation, and impact of misinformation (2026) · cites this paper as Menz et al. 2024
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“13 The World Health Organization recognises health disinformation as a critical threat to public health, as exemplified by the estimation that as of September 2022, more than 200 000 covid-19 related deaths in the US could have been averted had public health recommendations been followed.”
BackgroundBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · cited 1×
“7 Notably, LLMs that lack adequate guardrails and safety measures (ie, safeguards) may facilitate malicious actors to generate and propagate highly convincing health disinformation—that is, the intentional dissemination of misleading narratives about health topics for ill intent.”
BackgroundChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1166120 · Cited by 708
Harrer, S. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104512 · Cited by 480 · cited 1×
“16 We therefore evaluated prominent publicly accessible LLMs for safeguards preventing mass generation of health disinformation.”
BackgroundFinney Rutten, L. J., Blake, K. D., Greenberg-Worisek, A. J., Allen, S. V., Moser, R. P., & Hesse, B. W. (2019) · Public Health Reports® · doi:10.1177/0033354919874074 · Cited by 426
Medical Conspiracy Theories and Health Behaviors in the United States
Oliver, J. E., & Wood, T. (2014) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.190 · Cited by 418 · cited 1×
“Additionally, we concentrated on health disinformation topics widely regarded as being substantial/severe in the literature 24 25 , highlighting a gap for future studies to focus on equivocal topics, such as the link between sugar and cancer—a topic we briefly evaluated—wherein assessing the…”
BackgroundAI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346
Májovský, M., Černý, M., Kasal, M., Komarc, M., & Netuka, D. (2023) · Journal of Medical Internet Research · doi:10.2196/46924 · Cited by 301
Research works printed but not linked: 1
- Liu Y Deng G Xu Z . Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv 230513860 2023
Cited by
Generalization bias in large language model summarization of scientific research
Peters, U., & Chin-Yee, B. (2025) · Royal Society Open Science · doi:10.1098/rsos.241776 · Cited by 126
Self-citationThe generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 144
Chen, S., Gao, M., Sasse, K., Hartvigsen, T., Anthony, B., Fan, L., Aerts, H., Gallifant, J., & Bitterman, D. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02008-z · Cited by 112
Multimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 153
“Numerous ethical considerations, including patient privacy, bias mitigation, accountability, transparency, the impact on the profession, and preventing misuse, must be proactively addressed through interdisciplinary collaboration [148,149].”
Evaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 136
Metadata
- Authors
- Menz, B. D., Kuderer, N. M., Bacchi, S., Modi, N. D., Chin-Yee, B., Hu, T., Rickard, C., Haseloff, M., Vitry, A., McKinnon, R. A., Kichenadasse, G., Rowland, A., Sorich, M. J., & Hopkins, A. M.
- Affiliations
- Flinders University (AU)
- Advanced Cancer Research Group, Kirkland, WA, USA
- Western University (CA)
- University of Cambridge (GB)
- Corresponding author
- Hopkins, Ashley M
- ORCID
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2024-03-20
- Volume
- 384
- Pages
- e078538
- ISSN
- 1756-1833 (ISSN-L)
- PMC
- PMC10961718
- OpenAlex
- W4393054030
- Cited by
- 132
- References
- 19
- Citations by year
- 2026: 60 · 2025: 55 · 2024: 17
- FWCI
- 94.68
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- BMJ (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by-nc)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion, cc-by-nc)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Cancer Council Australia
- National Health and Medical Research Council
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, health disinformation, jailbreaking, risk mitigation, AI transparency, alkaline diet, AI-generated content, LLM vulnerabilities, GPT-4, Claude 2, PaLM 2, Llama 2, Gemini Pro, ChatGPT, Copilot, bards
- MeSH terms
- Disinformation, Animals, Artificial Intelligence, Camelids, New World, Cross-Sectional Studies, Humans, Language, Skin Neoplasms, Sunscreening Agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis. https://app.cochat.ai/papers/10.1136/bmj-2023-078538
- The paper
- Menz, B. D., Kuderer, N. M., Bacchi, S., Modi, N. D., Chin-Yee, B., Hu, T., Rickard, C., Haseloff, M., Vitry, A., McKinnon, R. A., Kichenadasse, G., Rowland, A., Sorich, M. J., & Hopkins, A. M. (2024). Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis. BMJ. https://doi.org/10.1136/bmj-2023-078538
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
