CoChat
Sign Up
Cited by 32Pharmacovigilance and Adverse Drug ReactionsArtificial Intelligence in Healthcare and EducationTopic Modeling

The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings

Open access
Hakim, Joe B., Painter, Jeffery L., Ramcharran, Darmendra, Kara, Vijay, Powell, Greg, Sobczak, Paulina, Sato, Chiho, Bate, Andrew +1 more·2025·Scientific Reports
DOI 10.1038/s41598-025-09138-0PMC PMC12311179
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are useful tools with the capacity for performing specific types of knowledge work at an effective scale. However, LLM deployments in high-risk and safety-critical domains pose unique challenges, notably the issue of "hallucinations", where LLMs can generate fabricated information. This is particularly concerning in settings such as drug safety, where inaccuracies could lead to patient harm. To mitigate these risks, we have developed and demonstrated a proof of concept suite of guardrails specifically designed to mitigate certain types of hallucinations and errors for drug safety, with potential applicability to other medical safety-critical contexts. These guardrails include mechanisms to detect anomalous documents to prevent the ingestion of inappropriate data, identify incorrect drug names or adverse event terms, and convey uncertainty in generated content. We integrated these guardrails with an LLM fine-tuned for a text-to-text task, which involves converting both structured and unstructured data within adverse event reports into natural language. This method was applied to translate individual case safety reports, demonstrating effective application in a pharmacovigilance processing task. Our guardrail framework offers a set of tools with broad applicability across various domains, ensuring LLMs can be safely used in high-risk situations by eliminating the occurrence of key errors, including the generation of incorrect pharmacovigilance-related terms, thus adhering to stringent regulatory and quality standards in medical safety-critical environments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-025-09138-0

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hakim et al. (2025) emphasized the need for robust guardrails and continuous monitoring when LLMs are deployed in safety-critical medical domains such as drug safety.

    “At the same time, our observation of false-positive inflation aligns with the warning issued by Hakim et al. (2025) [40], who emphasized the need for robust guardrails and continuous monitoring when LLMs are deployed in safety-critical medical domains such as drug safety.”

    From Comparing Artificial Intelligence (ChatGPT, Gemini, DeepSeek) and Oral Surgeons in Detecting Clinically Relevant Drug–Drug Interactions in Dental Therapy (2025) · cites this paper as Hakim et al. (2025) [40]

    The Introduction describes a false-positive drug-safety concern that an LLM could generate, and the Discussion calls for a continuous need for robust guardrails in pharmacovigilance and other medical safety-critical settings. The paper also advocates human review of uncertain outputs, but the passages do not specifically call for continuous monitoring of deployed LLMs. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. [1]Tang, L. et al. Evaluating large language models on medical evidence summarization. Npj Digit. Med. 6 (1), 158 (2023). 37620423 PMC10449915
  2. [2]Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N. et al. Large Language Models Encode Clinical Knowledge (2022). http://arxiv.org/abs/2212.13138. PMC10396962 37438534 · arXiv
  3. [4]Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., et al. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023). http://arxiv.org/abs/2309.01219. · arXiv
  4. [5]McKenna, N., Li, T., Cheng, L., Hosseini, M. J., Johnson, M. & Steedman, M. Sources of Hallucination by Large Language Models on Inference Tasks (2023). http://arxiv.org/abs/2305.14552. · arXiv
  5. [6]Wagle, S., Munikoti, S., Acharya, A., Smith, S. & Horawalavithana, S. Empirical Evaluation of Uncertainty Quantification in Retrieval-Augmented Language Models for Science (2023). http://arxiv.org/abs/2311.09358. · arXiv
  6. [7]Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J. & Hooi, B. Can Llms Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in Llms (2023). http://arxiv.org/abs/2306.13063. · arXiv
  7. [8]Bolton, W. J., Poyiadzi, R., Morrell, E. R., Bueno, G. V. B. G. & Goetz, L. RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain. http://arxiv.org/abs/2403.14578 (2024). · arXiv
  8. [10]Bowen, J. & Stavridou, V. Safety-critical systems, formal methods and standards. Softw. Eng. J. 8 (4), 189–209 (1993).
  9. [12]Bate, A. & Evans, S. J. W. Quantitative signal detection using spontaneous ADR reporting. Pharmacoepidemiol. Drug Saf. 18 (6), 427–436 (2009). 19358225
  10. [13]Dong, Y., Mu, R., Jin, G., Qi, Y., Hu, J., Zhao, X., Meng, J., Ruan, W. & Huang, X. Building Guardrails for Large Language Models. http://arxiv.org/abs/2402.01822 (2024) · arXiv
  11. [14]Health Service Journal. Guidance on implementing the never events framework (2009). https://www.hsj.co.uk/home/guidance-on-implementing-the-never-events-framework/5000691.article.
  12. [15]Anderson, J. E. & Watt, A. J. Using safety-II and Resilient healthcare principles to learn from never events. Int. J. Qual. Health Care 32 (3), 196–203 (2020). 32175571
  13. [18]Food and Drug Administration. E2B(R3) Electronic Transmission of Individual Case Safety Reports Implementation Guide: Data Elements and Message Specification; and Appendix to the Implementation Guide—Backwards and Forwards Compatibility (2022). https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e2br3-electronic-transmission-individual-case-safety-reports-implementation-guide-data-elements-and 24611208
  14. [19]Zhang, B., Williams, P., Titov, I. & Sennrich, R. Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020). http://arxiv.org/abs/2004.11867 · arXiv
  15. [20]Graves, A. Sequence Transduction with Recurrent Neural Networks (2012). http://arxiv.org/abs/1211.3711. · arXiv
  16. [21]Su, Y. et al. A contrastive framework for neural text generation. Adv. Neural. Inf. Process. Syst. 35, 21548–21561 (2022).
  17. [22]Papineni, K., Roukos, S., Ward, T. & Zhu, W. J. Bleu: A method for automatic evaluation of machine translation. in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics 311–18 (2002).
  18. [23]Post, M. A Call for Clarity in Reporting BLEU Scores (2018). http://arxiv.org/abs/1804.08771. · arXiv
  19. [24]Klakow, D. & Peters, J. Testing the correlation of word error rate and perplexity. Speech Commun. 38 (1–2), 19–28 (2002).
  20. [25]Kara, V. et al. Finding needles in the haystack: Clinical utility score for prioritisation (CUSP), an automated approach for identifying spontaneous reports with the highest clinical utility. Drug Saf. 46 (9), 847–855 (2023). 37535258 PMC10442257
  21. [26]Koch-Weser, J., Sellers, E. M. & Zacest, R. The ambiguity of adverse drug reactions. Eur. J. Clin. Pharmacol. 11, 75–78. 10.1007/BF00562895 (1977). 837969 · DOI
  22. [27]Arimone, Y. et al. Agreement of expert judgement in causality assessment of adverse drug reactions. Eur. J. Clin. Pharmacol. 61, 169–173. 10.1007/s00228-004-0869-2 (2005). 15827761 · DOI
  23. [28]Arimone, Y. et al. Inter-expert agreement of seven criteria in causality assessment of adverse drug reactions. Br. J. Clin. Pharmacol. 64 (4), 482–488. 10.1111/j.1365-2125.2007.02937.x (2007). 17711539 PMC2048553 · DOI
  24. [31]Brown, E. G., Wood, L. & Wood, S. The medical dictionary for regulatory activities (MedDRA). Drug Saf. 20 (2), 109–117 (1999). 10082069
  25. [33]Kuhn, M. et al. The SIDER database of drugs and side effects. Nucleic Acids Res. 44 (D1), D1075–D1079 (2016). 26481350 PMC4702794

Cited by

Sorted by citations

Authors
Hakim, J. B., Painter, J. L., Ramcharran, D., Kara, V., Powell, G., Sobczak, P., Sato, C., Bate, A., & Beam, A.
Affiliations
  • Harvard–MIT Division of Health Sciences and Technology (US)
  • GlaxoSmithKline (United States) (US)
  • GlaxoSmithKline (United Kingdom) (GB)
  • GlaxoSmithKline (Poland) (PL)
  • GlaxoSmithKline (Japan) (JP)
  • London School of Hygiene & Tropical Medicine (GB)
  • Harvard University (US)
Corresponding author
Bate, Andrew
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2025-07-31
Volume
15
Issue
1
Pages
27886-27886
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-025-09138-0
PMC
PMC12311179
OpenAlex
W4412813658
Cited by
32
References
1
Citations by year
2026: 27 · 2025: 4 · 2024: 1
FWCI
15.80
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
GlaxoSmithKline Biologicals S.A.
Topics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, hallucination mitigation, guardrails, pharmacovigilance, adverse event reports, drug safety, safety-critical systems, uncertainty quantification, individual case safety reports, medical safety
MeSH terms
Large Language Models, Humans, Language, Natural Language Processing, Adverse Drug Reaction Reporting Systems, Pharmacovigilance, Patient Safety, Drug-Related Side Effects and Adverse Reactions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review