A framework enabling LLMs into regulatory environment for transparency and trustworthiness and its application to drug labeling document
Open accessAbstract
Regulatory agencies consistently deal with extensive document reviews, ranging from product submissions to both internal and external communications. Large Language Models (LLMs) like ChatGPT can be invaluable tools for these tasks, however present several challenges, particularly the proprietary information, combining customized function with specific review needs, and transparency and explainability of the model's output. Hence, a localized and customized solution is imperative. To tackle these challenges, we formulated a framework named askFDALabel on FDA drug labeling documents that is a crucial resource in the FDA drug review process. AskFDALabel operates within a secure IT environment and comprises two key modules: a semantic search and a Q&A/text-generation module. The Module S built on word embeddings to enable comprehensive semantic queries within labeling documents. The Module T utilizes a tuned LLM to generate responses based on references from Module S. As the result, our framework enabled small LLMs to perform comparably to ChatGPT with as a computationally inexpensive solution for regulatory application. To conclude, through AskFDALabel, we have showcased a pathway that harnesses LLMs to support agency operations within a secure environment, offering tailored functions for the needs of regulatory research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To date, no public dataset has systematically annotated or categorized extrapolation decisions in FDA drug labels.
“to date, no public dataset has systematically annotated or categorized extrapolation decisions in FDA drug labels.”
From PED-X-Bench: A Benchmark of Adult-to-Pediatric Extrapolation Decisions in FDA Drug Labels (2025) · cites this paper as [14]
Recent NLP work on FDA labels targets toxicity or adverse-event extraction or general summarisation, not extrapolation.
“recent NLP work on FDA labels targets toxicity or adverse-event extraction [12, 13] or general summarisation [14], not extrapolation.”
From PED-X-Bench: A Benchmark of Adult-to-Pediatric Extrapolation Decisions in FDA Drug Labels (2025) · cites this paper as [14]
AskFDALabel's DICT classification F1-score of 0.931 significantly improved upon a previous LLM (Llama 13B) used in AskFDALabel (F1-score = 0.728) and GPT-3.5 (F1-score = 0.756).
“significantly improving upon results from a previous LLM (Llama 13B) used in AskFDALabel (F1-score = 0.728) and GPT-3.5 (F1-score = 0.756) [20].”
From Leveraging FDA Labeling Documents and Large Language Model to Enhance Annotation, Profiling, and Classification of Drug Adverse Events with AskFDALabel (2025) · cites this paper as [20]
Cites
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · UCL Discovery (University College London) · Cited by 1,187 · see citing works
Haleem, A., Javaid, M., & Singh, R. P. (2022) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100089 · Cited by 694 · see citing works
Gravel, J., D’Amours-Gravel, M., & Osmanlliu, E. (2023) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2023.05.004 · Cited by 222 · see citing works
Thakur, N., Reimers, N., Daxenberger, J., & Gurevych, I. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.28 · Cited by 215 · see citing works
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., & Launay, J. (2023) · arXiv · doi:10.48550/arxiv.2306.01116 · Cited by 164 · see citing works
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
Beware of references when using ChatGPT as a source of information to write scientific articles
Sanchez-Ramos, L., Lin, L., & Romero, R. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.04.004 · Cited by 60 · see citing works
Fang, H., Harris, S. C., Liu, Z., Zhou, G., Zhang, G., Xu, J., Rosario, L., Howard, P. C., & Tong, W. (2016) · Drug Discovery Today · doi:10.1016/j.drudis.2016.06.006 · Cited by 56 · see citing works
Self-citationPharmBERT: a domain-specific BERT model for drug labels
ValizadehAslani, T., Shi, Y., Ren, P., Wang, J., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad226 · Cited by 50 · see citing works
Qu, Y., Li, T., Liu, Z., Li, D., & Tong, W. (2023) · Drug Discovery Today · doi:10.1016/j.drudis.2023.103770 · Cited by 41 · see citing works
Self-citationFDALabel for drug repurposing studies and beyond
Fang, H., Harris, S., Liu, Z., Thakkar, S., Yang, J., Ingle, T., Xu, J., Lesko, L., Rosario, L., & Tong, W. (2020) · Nature Biotechnology · doi:10.1038/s41587-020-00751-0 · Cited by 19 · see citing works
Self-citationRxBERT: Enhancing drug labeling text mining and analysis with AI language modeling
Wu, L., Gray, M., Dang, O., Xu, J., Fang, H., & Tong, W. (2023) · Experimental Biology and Medicine · doi:10.1177/15353702231220669 · Cited by 16 · see citing works
Self-citation
Flagged first, then by citations
1 of the 15 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Artificial intelligence for drug delivery: Yesterday, today and tomorrow
Wu, Y., Wang, N., Xiong, P., Wang, R., Deng, J., & Ouyang, D. (2025) · Acta Pharmaceutica Sinica B · doi:10.1016/j.apsb.2025.09.022 · Cited by 34 · see citing works
Leveraging Large Language Models for Enhancing Safety in Maritime Operations
Miller, T. I., Durlik, I., Kostecka, E., Łobodzińska, A., Łazuga, K., & Kozlovska, P. (2025) · Applied Sciences · doi:10.3390/app15031666 · Cited by 31 · see citing works
Wu, L., Fang, H., Qu, Y., Xu, J., & Tong, W. (2025) · Drug Safety · doi:10.1007/s40264-025-01520-1 · Cited by 20 · see citing works
“5 (F1-score = 0.756) [20].”
InfluentialSelf-citationHe, J., Zhang, B., Rouhizadeh, H., Chen, Y., Yang, R., Lu, J. S., Chen, X., Liu, N., & Teodoro, D. (2026) · Artificial Intelligence Review · doi:10.21203/rs.3.rs-8330917/v1 · Cited by 17 · see citing works
UniTox: Leveraging LLMs to Curate a Unified Dataset of Drug-Induced Toxicity from FDA Labels
Silberg, J., Swanson, K., Simon, E. P., Zhang, A., Ghazizadeh, Z., Ogden, S., Hamadeh, H. K., & Zou, J. (2024) · neural information processing systems · doi:10.52202/079017-0385 · Cited by 9 · see citing works
A Bibliometric Analysis of Trust in Conversational Agents over the Past Fifteen Years
Aksoy, M., & Bush, A. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00076 · Cited by 6 · see citing works
Kadi, H., Abdellatif, A., Njamen, D. I. K., & Pereme, F. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1598979 · Cited by 5 · see citing works
Is ChatGPT Ready for Public Use in Organ-Specific Drug Toxicity Research?
Connor, S., Wu, L., Roberts, R. A., & Tong, W. (2025) · Drug Discovery Today · doi:10.1016/j.drudis.2025.104297 · Cited by 4 · see citing works
Self-citationOptimization Techniques in Large Language Models for News Report Generation
Bhatia, T., Gupta, A. V., Varshney, D., Mutreja, T., Kumar, S., & Vijh, S. (2024) · International Conference on Emerging Technologies and Innovation for Sustainability (EmergIN) · doi:10.1109/emergin63207.2024.10961084 · Cited by 3 · see citing works
Martins, B., Pereira, L. H. N., de Azevedo, R., & Costa, P. (2025) · Pharmacophore · doi:10.51847/vwhtdaetbq · Cited by 2 · see citing works
Park, H., Rees, M., Kruger, N., Fuse, K., Castro, V. M., Gainer, V. S., Wattanasin, N., Benoit, B., Wagholikar, K. B., & Murphy, S. N. (2025) · medRxiv · doi:10.1101/2025.04.29.25326696 · Cited by 1 · see citing works
PED-X-Bench: A Benchmark of Adult-to-Pediatric Extrapolation Decisions in FDA Drug Labels
Srinivasan, A., Berkowitz, J., Friedrich, N. A., Tsang, K., Kuchi, A., Acitores, J., Zietz, M., Czarny, R. S., Liu, H., & Tatonetti, N. P. (2025) · medRxiv · doi:10.1101/2025.05.22.25328187
“On the other hand, recent NLP work on FDA labels targets toxicity or adverse-event extraction [12, 13] or general summarisation [14], not extrapolation.”
Large language models and retrieval-augmented generation-based chatbot for adolescent mental health
Riansyah, A., Subroto, I. M. I., Nur'aini, I., Supradewi, R., & Suyanto, S. (2025) · International Journal of Advances in Applied Sciences · doi:10.11591/ijaas.v14.i3.pp849-858
Procópio, D. C., & Silva, P. N. (2026) · Information Research · doi:10.47989/ir31iconf64289
Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domains
Shrestha, M., & Kim, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30813-9_68
Organização e Arquitetura da Informação em Large Language Models
Procópio, D. C., & Silva, P. N. (2026) · Informação & Informação · doi:10.5433/1981-8920.2026v31n1p149
Advancing Patient‐Centred AI With Adaptive Machine Learning
Kalogeropoulos, D., Barach, P. R., Downing, A., Verhulst, S., & Lustberg, M. B. (2026) · Cancer Medicine · doi:10.1002/cam4.72371
AI–Assisted Process for Hospital New Drug Inclusion: Development and Validation (Preprint)
Yang, Z., Wu, A., Huang, F., Zhong, G., Zhu, H., Xie, H., He, J. L., Lu, J., Zhong, J., Lan, J., et al. (2026) · JMIR Medical Informatics · doi:10.2196/93972
LLM-Based IAS: Facilitating Trustworthiness Through Transparency
Black, A., Smith, G., Francia, G., & El-Sheikh, E. (2026) · Studies in big data · doi:10.1007/978-3-032-23522-0_27
- Authors
- Wu, L., Xu, J., Thakkar, S., Gray, M., Qu, Y., Li, D., & Tong, W.
- Affiliations
- National Center for Toxicological Research (US)
- United States Food and Drug Administration (US)
- Corresponding author
- Wu, Leihong
- Tong, Weida
- Venue
- Regulatory Toxicology and Pharmacology
- Publisher
- Elsevier BV
- Published
- 2024-05
- Volume
- 149
- Pages
- 105613
- ISSN
- 0273-2300 (ISSN-L)
- OpenAlex
- W4393952524
- Cited by
- 16
- References
- 15
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 2
- FWCI
- 2.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Regulatory Toxicology and Pharmacology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- Oak Ridge Institute for Science and Education
- U.S. Food and Drug Administration (E0777301, E0782201)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Keywords
- large language models, FDA drug review, semantic search, word embeddings, question answering, text generation, proprietary information, explainability
- MeSH terms
- Drug Labeling, Humans, United States, United States Food and Drug Administration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
