CoChat
Sign Up
Cited by 104Biomedical Text Mining and OntologiesTopic ModelingMultimodal Machine Learning Applications

Advancing entity recognition in biomedicine via instruction tuning of large language models

Open access
Keloth, Vipina K, Hu, Yan, Xie, Qianqian, Peng, Xueqing, Wang, Yan, Zheng, Andrew, Selek, Melih, Raja, Kalpana +5 more·2024·Bioinformatics
DOI 10.1093/bioinformatics/btae163PMC PMC11001490
Open this paper in CoChat — free accountOpen PDF

Abstract

Motivation: Large Language Models (LLMs) have the potential to revolutionize the field of Natural Language Processing, excelling not only in text generation and reasoning tasks but also in their ability for zero/few-shot learning, swiftly adapting to new tasks with minimal fine-tuning. LLMs have also demonstrated great promise in biomedical and healthcare applications. However, when it comes to Named Entity Recognition (NER), particularly within the biomedical domain, LLMs fall short of the effectiveness exhibited by fine-tuned domain-specific models. One key reason is that NER is typically conceptualized as a sequence labeling task, whereas LLMs are optimized for text generation and reasoning tasks. Results: We developed an instruction-based learning paradigm that transforms biomedical NER from a sequence labeling task into a generation task. This paradigm is end-to-end and streamlines the training and evaluation process by automatically repurposing pre-existing biomedical NER datasets. We further developed BioNER-LLaMA using the proposed paradigm with LLaMA-7B as the foundational LLM. We conducted extensive testing on BioNER-LLaMA across three widely recognized biomedical NER datasets, consisting of entities related to diseases, chemicals, and genes. The results revealed that BioNER-LLaMA consistently achieved higher F1-scores ranging from 5% to 30% compared to the few-shot learning capabilities of GPT-4 on datasets with different biomedical entities. We show that a general-domain LLM can match the performance of rigorously fine-tuned PubMedBERT models and PMC-LLaMA, biomedical-specific language model. Our findings underscore the potential of our proposed paradigm in developing general-domain LLMs that can rival SOTA performances in multi-task, multi-domain scenarios in biomedical and health applications. Availability and implementation: Datasets and other resources are available at https://github.com/BIDS-Xu-Lab/BioNER-LLaMA.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/bioinformatics/btae163

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models.

    “Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”

    From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [51]

    The Results (§4.1, Table 1; page unavailable) compare zero- and five-shot GPT models with instruction-tuned BioNER-LLaMA and report lower performance for the GPT models. This supports superior performance for the tuned LLaMA models in that comparison, but the paper does not compare zero- or few-shot LLaMA models with fine-tuned LLaMA models. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 27 · Cited by 104

Research works printed but not linked: 60

  1. Achiam J, Adler S, Agarwal S et al Gpt-4 technical report. arXiv, arXiv:2303.08774, 2023, preprint: not peer reviewed. · arXiv
  2. Agrawal M, Hegselmann S, Lang H et al Large language models are few-shot clinical information extractors. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, UAE. 2022.
  3. Ashok D, Lipton ZC. PromptNER: prompting for named entity recognition. arXiv, arXiv:2305.15444, 2023, preprint: not peer reviewed. · arXiv
  4. Beltagy I, Lo K, Cohan A. SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620, Hong Kong, China. Association for Computational Linguistics. 2019.
  5. Biderman S, Schoelkopf H, Anthony QG et al Pythia: a suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning, Honolulu, Hawaii, USA. 2023.
  6. Chaudhry MC, Kazmi A, Jatav S et al Reducing inference time of biomedical NER tasks using multi-task learning. In: Proceedings of the 19th International Conference on Natural Language Processing (ICON), New Delhi, India. 2022.
  7. Chen Q, Du J, Hu Y et al Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv, arXiv:2305.16326, 2023, preprint: not peer reviewed. · arXiv
  8. Cho H, Lee H. Biomedical named entity recognition using deep neural networks with contextual information. BMC Bioinformatics 2019; 20: 735. 31881938 PMC6935215
  9. Dang TH, Le H-Q, Nguyen TM et al D3NER: biomedical named entity recognition using CRF-biLSTM improved with fine-tuned embeddings of various linguistic information. Bioinformatics 2018; 34: 3539– 46. 29718118
  10. Dettmers T, Lewis M, Belkada Y et al LLM.int8(): 8-bit matrix multiplication for transformers at scale. 2022. CoRR abs/2208.07339. · arXiv
  11. Devlin J, Chang M-W, Lee K et al BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. 2018.
  12. Doğan RI, Leaman R, Lu Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. J Biomed Inform 2014; 47: 1– 10. 24393765 PMC3951655
  13. Fang L, Chen Q, Wei C-H et al Bioformer: an efficient transformer language model for biomedical text mining. arXiv, arXiv:2302.01588, 2023, preprint: not peer reviewed. · arXiv
  14. Fukuda K-i, Tsunoda T, Tamura A et al Toward information extraction: identifying protein names from biological papers. Pac Symp Biocomput 1998: 707– 18. 9697224
  15. Gilson A, Safranek CW, Huang T et al How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023; 9: e45312. 36753318 PMC9947764
  16. Gu Y, Tinn R, Cheng H et al Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthcare 2021; 3: 1– 23.
  17. Hendy A, Abdelrehim M, Sharaf A et al How good are GPT models at machine translation? A comprehensive evaluation. arXiv, arXiv:2302.09210, 2023, preprint: not peer reviewed. · arXiv
  18. Hu EJ, Shen Y, Wallis P et al LoRA: low-rank adaptation of large language models. In: International Conference on Learning Representations, Virtual. 2022.
  19. Hu Y, Chen Q, Du J et al Improving large language models for clinical named entity recognition via prompt engineering. J Am Med Inf Assoc 2024: ocad259. PMC11339492 38281112
  20. Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF models for sequence tagging. arXiv, arXiv:1508.01991, 2015, preprint: not peer reviewed. · arXiv
  21. Islamaj R, Leaman R, Cissel D et al NLM-Chem-BC7: manually annotated full-text resources for chemical entity annotation and indexing in biomedical articles. Database 2022; 2022: baac102. 36458799 PMC9716560
  22. Islamaj R, Wei C-H, Cissel D et al NLM-Gene, a richly annotated gold standard dataset for gene entities that addresses ambiguity and multi-species gene recognition. J Biomed Inform 2021; 118: 103779. 33839304 PMC11037554
  23. Ji B. VicunaNER: zero/few-shot named entity recognition using vicuna. arXiv, arXiv:2305.03253, 2023, preprint: not peer reviewed. · arXiv
  24. Khambete MP, Su W, Garcia JC et al Quantification of BERT diagnosis generalizability across medical specialties using semantic dataset distance. In: AMIA Summits on Translational Science Proceedings, p. 345. 2021. PMC8378651 34457149
  25. Kojima T, Gu SS, Reid M et al Large language models are zero-shot reasoners. Adv Neural Inf Process Syst 2022; 35: 22199– 213.
  26. Košprdić M, Prodanović N, Ljajić A et al Zero-and few-shot machine learning for named entity recognition in biomedical texts. In: 4th Belgrade Bioinformatics Conference, Belgrade, Serbia. 2023.
  27. Labrak Y, Rouvier M, Dufour R. A zero-shot and few-shot study of instruction-finetuned large language models applied to clinical and biomedical tasks. arXiv, arXiv:2307.12114, 2023, preprint: not peer reviewed. · arXiv
  28. Lee J, Yoon W, Kim S et al BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 2020; 36: 1234– 40. 31501885 PMC7703786
  29. Li J, Sun Y, Johnson RJ et al BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016; 2016: baw068. 27161011 PMC4860626
  30. Li J, Wei Q, Ghiasvand O et al A comparative study of pre-trained language models for named entity recognition in clinical trial eligibility criteria from multiple corpora. BMC Med Inform Decis Mak 2022; 22: 235. 36068551 PMC9450226
  31. Loukas L, Stogiannidis I, Malakasiotis P et al Breaking the bank with ChatGPT: few-shot text classification for finance. In Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial Forecasting, pages 74–80, Macao. 2023
  32. Luo L, Wei C-H, Lai P-T et al AIONER: all-in-one scheme-based biomedical named entity recognition using deep learning. Bioinformatics 2023; 39: btad310. 37171899 PMC10212279
  33. Manakhimova S, Avramidis E, Macketanz V et al Linguistically motivated evaluation of the 2023 state-of-the-art machine translation: can ChatGPT outperform NMT? In: Proceedings of the Eighth Conference on Machine Translation. Singapore. 2023.
  34. Margatina K, Schick T, Aletras N et al Active learning principles for in-context learning with large language models. In Findings of the Association of Computational Linguistics: EMNLP 2023, pp. 5011-5034, Singapore. 2023
  35. Morwal S, Jahan N, Chopra D. Named entity recognition using hidden Markov model (HMM). In: International Journal on Natural Language Computing (IJNLC) Vol, 1. 2012
  36. Peng Y, Chen Q, Lu Z. An empirical study of multi-task learning on BERT for biomedical text mining. In: Proceedings of the BioNLP 2020 workshop, pages 205–214. Online. 2020
  37. Ponomareva N, Rosso P, Pla F et al Conditional random fields vs. hidden Markov models in a biomedical named entity recognition task. In: Proceedings of International Conference on Recent Advances in Natural Language Processing, RANLP, Borovets, Bulgaria. 2007.
  38. Rodriguez NE, Nguyen M, McInnes BT. Effects of data and entity ablation on multitask learning models for biomedical entity recognition. J Biomed Inform 2022; 130: 104062. 35413440
  39. Settles B. ABNER: an open source tool for automatically tagging genes, proteins and other entity names in text. Bioinformatics 2005; 21: 3191– 2. 15860559
  40. Shao Y, Lin JC-W, Srivastava G et al Self-attention-based conditional random fields latent variables model for sequence labeling. Pattern Recognit Lett 2021; 145: 157– 64.
  41. Smith L, Tanabe LK, Kuo C-J et al Overview of BioCreative II gene mention recognition. Genome Biol 2008; 9 Suppl 2: S2– 19. PMC2559986 18834493
  42. Sutton C, McCallum A. An introduction to conditional random fields. Found Trends Mach Learn 2012; 4: 267– 373.
  43. Touvron H, Lavril T, Izacard G et al LLaMA: open and efficient foundation language models. arXiv, arXiv:2302.13971, 2023a, preprint: not peer reviewed. · arXiv
  44. Touvron H, Martin L, Stone K et al LLaMA 2: open foundation and fine-tuned chat models. arXiv, arXiv:2307.09288, 2023b, preprint: not peer reviewed. · arXiv
  45. Tsai RT-H, Sung C-L, Dai H-J et al NERBio: using selected word conjunctions, term normalization, and global patterns to improve biomedical named entity recognition. BMC Bioinformatics 2006; 7: S11– 14. PMC1764467 17254295
  46. Tsuruoka Y, Tsujii J. Boosting precision and recall of dictionary-based protein name recognition. In: Proceedings of the ACL 2003 Workshop on Natural Language Processing in Biomedicine. Budapest, Hungary. 2003.
  47. Wang J, Liang Y, Meng F et al Is ChatGPT a good NLG evaluator? A preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Linguistics. 2023a.
  48. Wang J, Liang Y, Meng F et al Zero-shot cross-lingual summarization via large language models. In: Proceedings of the 4th New Frontiers in Summarization Workshop. Singapore. 2023b.
  49. Wang S, Sun X, Li X et al GPT-NER: named entity recognition via large language models. arXiv, arXiv:2304.10428, 2023c, preprint: not peer reviewed. · arXiv
  50. Wang X, Zhang Y, Ren X et al Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics 2019; 35: 1745– 52. 30307536
  51. Weber L, Münchmeyer J, Rocktäschel T et al HUNER: improving biomedical NER with pretraining. Bioinformatics 2020; 36: 295– 302. 31243432
  52. Wei J, Bosma M, Zhao VY et al Finetuned language models are zero-shot learners. arXiv, In: International Conference on Learning Representations. Virtual. 2022.
  53. Wei X, Cui X, Cheng N et al Zero-shot information extraction via chatting with ChatGPT. arXiv, arXiv:2302.10205, 2023, preprint: not peer reviewed. · arXiv
  54. Wu C, Zhang X, Zhang Y et al PMC-LLaMA: further finetuning LLaMA on medical papers. arXiv, arXiv:2304.14454, 2023, preprint: not peer reviewed. · arXiv
  55. Yadav S, Ramesh S, Saha S et al Relation extraction from biomedical and clinical text: unified multitask learning framework. IEEE/ACM Trans Comput Biol Bioinform 2022; 19: 1105– 16. 32853152
  56. Yang Z, Lin H, Li Y. Exploiting the performance of dictionary-based bio-entity name recognition in biomedical literature. Comput Biol Chem 2008; 32: 287– 91. 18467180
  57. Zhang R, Han J, Zhou A et al LLaMA-adapter: efficient fine-tuning of language models with zero-init attention. arXiv, arXiv:2303.16199, 2023, preprint: not peer reviewed. · arXiv
  58. Zhao X, Ouyang S, Yu Z et al Pre-trained language models can be fully zero-shot learners. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15590–15606, Toronto, Canada. Association for Computational Linguistics 2022.
  59. Zhou B, Cai X, Zhang Y et al An end-to-end progressive multi-task learning framework for medical named entity recognition and normalization. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Online. 2021.
  60. Zhou W, Zhang S, Gu Y et al UniversalNER: targeted distillation from large language models for open named entity recognition. arXiv, arXiv:2308.03279, 2023, preprint: not peer reviewed. · arXiv

Authors
Keloth, V. K., Hu, Y., Xie, Q., Peng, X., Wang, Y., Zheng, A., Selek, M., Raja, K., Wei, C. H., Jin, Q., Lu, Z., Chen, Q., & Xu, H.
Affiliations
  • Yale University (US)
  • The University of Texas Health Science Center at Houston (US)
  • William P. Clements High School , Sugar Land, TX-77479, United States
  • Stephen F. Austin High School , Sugar Land, TX-77498, United States
  • National Institutes of Health (US)
Corresponding author
  • Chen, Qingyu
  • Xu, Hua
ORCID
Venue
Bioinformatics
Publisher
Oxford University Press (OUP)
Published
2024-03-21
Volume
40
Issue
4
ISSN
  • 1367-4803 (ISSN-L)
  • 1367-4811
DOI
10.1093/bioinformatics/btae163
PMC
PMC11001490
OpenAlex
W4393094733
Cited by
104
References
27
Citations by year
2026: 27 · 2025: 60 · 2024: 17
FWCI
14.11
Type
journal article
Language
en
Open access
Licence
Locations
  • Bioinformatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Institutes of Health (R01AG078154, R01LM013519, 1K99LM01402)
  • Intramural Research Program of the National Library of Medicine
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
biomedical named entity recognition, instruction tuning, large language models, llamas, sequence labeling, few-shot learning, disease entity, chemical entity
MeSH terms
Deep Learning, Animals, Camelids, New World, Language, Natural Language Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review