Improving large language models for clinical named entity recognition via prompt engineering
Open accessAbstract
IMPORTANCE: The study highlights the potential of large language models, specifically GPT-3.5 and GPT-4, in processing complex clinical data and extracting meaningful information with minimal training data. By developing and refining prompt-based strategies, we can significantly enhance the models' performance, making them viable tools for clinical NER tasks and possibly reducing the reliance on extensive annotated datasets. OBJECTIVES: This study quantifies the capabilities of GPT-3.5 and GPT-4 for clinical named entity recognition (NER) tasks and proposes task-specific prompts to improve their performance. MATERIALS AND METHODS: We evaluated these models on 2 clinical NER tasks: (1) to extract medical problems, treatments, and tests from clinical notes in the MTSamples corpus, following the 2010 i2b2 concept extraction shared task, and (2) to identify nervous system disorder-related adverse events from safety reports in the vaccine adverse event reporting system (VAERS). To improve the GPT models' performance, we developed a clinical task-specific prompt framework that includes (1) baseline prompts with task description and format specification, (2) annotation guideline-based prompts, (3) error analysis-based instructions, and (4) annotated samples for few-shot learning. We assessed each prompt's effectiveness and compared the models to BioClinicalBERT. RESULTS: Using baseline prompts, GPT-3.5 and GPT-4 achieved relaxed F1 scores of 0.634, 0.804 for MTSamples and 0.301, 0.593 for VAERS. Additional prompt components consistently improved model performance. When all 4 components were used, GPT-3.5 and GPT-4 achieved relaxed F1 socres of 0.794, 0.861 for MTSamples and 0.676, 0.736 for VAERS, demonstrating the effectiveness of our prompt framework. Although these results trail BioClinicalBERT (F1 of 0.901 for the MTSamples dataset and 0.802 for the VAERS), it is very promising considering few training samples are needed. DISCUSSION: The study's findings suggest a promising direction in leveraging LLMs for clinical NER tasks. However, while the performance of GPT models improved with task-specific prompts, there's a need for further development and refinement. LLMs like GPT-4 show potential in achieving close performance to state-of-the-art models like BioClinicalBERT, but they still require careful prompt engineering and understanding of task-specific knowledge. The study also underscores the importance of evaluation schemas that accurately reflect the capabilities and performance of LLMs in clinical settings. CONCLUSION: While direct application of GPT models to clinical NER tasks falls short of optimal performance, our task-specific prompt framework, incorporating medical knowledge and training samples, significantly enhances GPT models' feasibility for potential clinical applications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models.
“Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”
From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [53]
The Results section reports that fine-tuned BioClinicalBERT outperformed few-shot GPT models on two clinical NER datasets, supporting the general performance comparison. But the Discussion says the study was limited to GPT models and identifies LLaMA as future work; it does not demonstrate superior performance for fine-tuned LLaMA models. Page numbers are unavailable in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLLMs show significant potential in health informatics tasks such as name entity extraction, patient trial matching, biomedical reasoning and classification, prediction of admissions, and automation of administrative tasks.
“LLMs show significant potential in health informatics, including tasks such as name entity extraction [11], patient trial matching [1213], biomedical reasoning and classification [14], prediction of admissions [15], automation of administrative tasks [16], and so forth.”
From Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study (2025) · cites this paper as [11]
Recent advances in large language models present an opportunity to streamline the informed consent form creation process while improving readability, understandability, and actionability.
“The recent advances in large language models (LLMs) present an opportunity to streamline the ICF creation process while improving readability, understandability, and actionability.”
From Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study (2025) · cites this paper as [11]
The Introduction and Discussion address using LLMs to streamline clinical named entity recognition, not informed consent form creation. Searches for informed consent forms and patient-facing document writing found no passage addressing ICF creation or improvements in readability, understandability, or actionability. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 1,984 · see citing works
Savova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967 · see citing works
Mining electronic health records: towards better research applications and clinical care
Jensen, P. B., Jensen, L. J., & Brunak, S. (2012) · Nature Reviews Genetics · doi:10.1038/nrg3208 · Cited by 1,680 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works
2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,258 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · see citing works
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910 · see citing works
Complexity-Based Prompting for Multi-Step ReasoningPreprint
Fu, Y., Peng, H.-C., Sabharwal, A., Clark, P., & Khot, T. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.00720 · Cited by 670 · see citing works
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · see citing works
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563 · see citing works
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549 · see citing works
Rao, A. S., Kim, J., Kamineni, M., Pang, M., Lie, W., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.05.003 · Cited by 481 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Self-citationBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
Clinical Natural Language Processing in languages other than English: opportunities and challenges
Aurélie Névéol, A., Dalianis, H., Velupillai, S. U., Savova, G., & Zweigenbaum, P. (2018) · Journal of Biomedical Semantics · doi:10.1186/s13326-018-0179-8 · Cited by 313 · see citing works
Jiang, M., Chen, Y., Liu, M., Rosenbloom, S. T., Mani, S., Denny, J. C., & Xu, H. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000163 · Cited by 292 · see citing works
Self-citationJin, Q., Yang, Y., Chen, Q., & Lu, Z. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae075 · Cited by 176 · see citing works
Self-citationPrompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150 · see citing works
Self-citationChen, W., Ma, X., Wang, X., & Cohen, W. W. (2022) · arXiv · doi:10.48550/arxiv.2211.12588 · Cited by 125 · see citing works
Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J., Savova, G. K., & Bitterman, D. S. (2023) · arXiv · doi:10.48550/arxiv.2304.02496 · Cited by 91 · see citing works
The AI revolution in medicine: GPT-4 and beyond
Craig, K. (2024) · Journal of IT Cases and Applications · doi:10.1080/15228053.2024.2417522 · Cited by 85 · see citing works
Du, J., Xiang, Y., Sankaranarayanapillai, M., Zhang, M., Wang, J., Si, Y., Pham, H. A., Xu, H., Chen, Y., & Tao, C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab014 · Cited by 34 · see citing works
Self-citationAteia, S., & Kruschwitz, U. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.16108 · Cited by 10 · see citing works
Prompt Engineering and Calibration for Zero-Shot Commonsense ReasoningPreprint
Ma, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.06962 · Cited by 3 · see citing works
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language ModelsPreprint
Sun, J., Luo, Y., Gong, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.11657 · Cited by 2 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 5
- [5]Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF models for sequence tagging. arXiv, arXiv:1508.01991, 2015, preprint: not peer reviewed. · arXiv
- [13]Brown T, Mann B, Ryder N, et al Language models are few-shot learners. Adv Neural Inform Process Syst. 2020; 33: 1877- 1901.
- [21]Chen Q, Du J, Hu Y, et al Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv, arXiv:2305.16326, 2023, preprint: not peer reviewed. · arXiv
- [25]Yu F, Quartey L, Schilder F. Exploring the effectiveness of prompt engineering for legal reasoning tasks. In: Findings of the Association for Computational Linguistics: ACL 2023; 2023.
- [33]Wolf T, Debut L, Sanh V, et al. Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations; 2020.
Cited by
Sorted by citations
Large language models for generative information extraction: a survey
Xu, D., Chen, W., Peng, W., Zhang, C., Xu, T., Zhao, X., Wu, X., Zheng, Y., Wang, Y., & Chen, E. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40555-y · Cited by 326 · see citing works
“…information from some specific domains, such as mulitmodal [57, 94, 166, 167], multilingual [83, 133, 163], medical [85, 91, 162, 163, 171, 172, 179, 183, 184, 185, 186, 187, 188, 234, 235], scientific [47, 80, 180, 181, 182], astronomical [164, 173], historical [126, 189], and legal [78,…”
Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-56989-2 · Cited by 219 · see citing works
“This approach aligns with previous studies in the biomedical domain, which have demonstrated that incorporating task guidance into the prompt leads to improved performance 74,76 and was also employed and evaluated in our previous study, speci fi cally focusing on named entity recognition 77 .”
Self-citationBenefits and Risks of AI in Health Care: Narrative Review
Chustecki, M. (2024) · Interactive Journal of Medical Research · doi:10.2196/53616 · Cited by 208 · see citing works
“These models leverage vast amounts of text data to learn rich representations of language, enabling tasks ranging from clinical documentation improvement to automated summarization of medical literature [15,16].”
Harnessing EHR data for health research
Tang, A. S., Woldemariam, S. R., Miramontes, S., Norgeot, B., Oskotsky, T. T., & Sirota, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-03074-8 · Cited by 87 · see citing works
Shen, M., Shen, Y., Liu, F., & Jin, J. (2025) · BMC Nursing · doi:10.1186/s12912-025-03115-8 · Cited by 76 · see citing works
Fleurence, R. L., Bian, J., Wang, X., Xu, H., Dawoud, D., Higashi, M., & Chhatwal, J. (2025) · Value in Health · doi:10.1016/j.jval.2024.10.3846 · Cited by 54 · see citing works
Self-citationBenchmarking Large Language Models in Evidence-Based Medicine
Li, J., Deng, Y., Sun, Q., Zhu, J., Tian, Y., Li, J., & Zhu, T. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483816 · Cited by 35 · see citing works
Keloth, V. K., Selek, S., Chen, Q., Gilman, C., Fu, S., Dang, Y., Chen, X., Hu, X., Zhou, Y., He, H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01645-8 · Cited by 24 · see citing works
Self-citationTransforming Informed Consent Generation Using Large Language Models: Mixed Methods Study
Shi, Q., Luzuriaga, K., Allison, J. J., Oztekin, A., Faro, J. M., Lee, J. L., Hafer, N., McManus, M., & Zai, A. H. (2025) · JMIR Medical Informatics · doi:10.2196/68139 · Cited by 23 · see citing works
“LLMs show significant potential in health informatics, including tasks such as name entity extraction [11], patient trial matching [12,13], biomedical reasoning and classification [14], prediction of admissions [15], automation of administrative tasks [16], and so forth.”
LLM-IE: a python package for biomedical generative information extraction with large language models
Hsu, E., & Roberts, K. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf012 · Cited by 16 · see citing works
Self-citationSun, D., Hadjiiski, L., Bruno, G., Gormley, J., Chan, H.-P., Caoili, E. M., Cohan, R. H., Alva, A., Mihalcea, R., Zhou, C., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-27593-7 · Cited by 9 · see citing works
Chen, D., Li, P., Khoshkish, E., Lee, S., Ning, T., Tahir, U., Wong, H. C. Y., Lee, M. S. F., & Raman, S. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf223 · Cited by 6 · see citing works
- Authors
- Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., Roberts, K., & Xu, H.
- Affiliations
- McWilliams School of Biomedical Informatics , Houston, TX, United States
- National Institutes of Health (US)
- Yale University (US)
- Corresponding author
- Xu, Hua
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-01-27
- Volume
- 31
- Issue
- 9
- Pages
- 1812-1820
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC11339492
- OpenAlex
- W4391292768
- Cited by
- 314
- References
- 35
- Citations by year
- 2026: 89 · 2025: 150 · 2024: 75
- FWCI
- 29.60
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- PubMed Central (submittedVersion)
- Funders
- National Institutes of Health (R21EB029575, R21AI164100, R01LM011934, 1K99LM01402, R01AG066749, R01AG066749-03S1, R01LM013712, U01TR002062)
- National Institute on Aging (1RF1AG072799, 1R01AG080429)
- Cancer Prevention and Research Institute of Texas (RR180012)
- National Science Foundation (2124789)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical named entity recognition, prompt engineering, large language models, GPT-3.5, GPT-4, clinical notes, adverse event reporting, VAERS, few-shot learning, annotation guidelines, error analysis, BioClinicalBERT, neurological disorders, safety reports
- MeSH terms
- Humans, Natural Language Processing, Data Mining, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
