CoChat
Sign Up
Cited by 542Biomedical Text Mining and OntologiesTopic ModelingMachine Learning in Bioinformatics

Deep learning with word embeddings improves biomedical named entity recognition

Open access
Habibi, Maryam, Weber, Leon, Neves, Mariana, Wiegandt, David Luis, Leser, Ulf·2017·Bioinformatics
DOI 10.1093/bioinformatics/btx228PMC PMC5870729
Open this paper in CoChat — free accountOpen PDF

Abstract

Motivation: Text mining has become an important tool for biomedical research. The most fundamental text-mining task is the recognition of biomedical named entities (NER), such as genes, chemicals and diseases. Current NER methods rely on pre-defined features which try to capture the specific surface properties of entity types, properties of the typical local context, background knowledge, and linguistic information. State-of-the-art tools are entity-specific, as dictionaries and empirically optimal feature sets differ between entity types, which makes their development costly. Furthermore, features are often optimized for a specific gold standard corpus, which makes extrapolation of quality measures difficult. Results: We show that a completely generic method based on deep learning and statistical word embeddings [called long short-term memory network-conditional random field (LSTM-CRF)] outperforms state-of-the-art entity-specific NER tools, and often by a large margin. To this end, we compared the performance of LSTM-CRF on 33 data sets covering five different entity classes with that of best-of-class NER tools and an entity-agnostic CRF implementation. On average, F1-score of LSTM-CRF is 5% above that of the baselines, mostly due to a sharp increase in recall. Availability and implementation: The source code for LSTM-CRF is available at https://github.com/glample/tagger and the links to the corpora are available at https://corposaurus.github.io/corpora/.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/bioinformatics/btx228

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Conventional NLP approaches for temporal information extraction suffer from limited scalability and generalizability and struggle to interpret implicit or vague temporal expressions.

    “Conventional natural language processing (NLP) approaches suffer from limited scalability and generalizability, and struggle to interpret implicit or vague temporal expressions.”

    From TimeX: Phenotype Onset Extraction from Clinical Narratives (2026) · cites this paper as [22]

    The Introduction says conventional biomedical named-entity recognition tools are costly to develop and highly specialized, supporting a limited claim about their generalizability and ease of scaling. The paper does not address temporal information extraction or difficulty interpreting implicit or vague temporal expressions, so those parts of the attributed claim are unsupported. Page numbers are not supplied in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Aerts S. et al (2006) Gene prioritization through genomic data fusion. Nat. Biotechnol., 24, 537– 544. 16680138
  2. Akhondi S.A. et al (2014) Annotated chemical patent corpus: a gold standard for text mining. PloS One, 9, 1– 8. PMC4182036 25268232
  3. Bagewadi S. et al (2014) Detecting miRNA mentions and relations in biomedical literature. F1000Research, 3. PMC4602280 26535109
  4. Batista-Navarro R. et al (2015) Optimising chemical named entity recognition with pre-processing analytics, knowledge-rich features and heuristics. J. Cheminform., 7. PMC4331696 25810777
  5. Campos D. et al (2012) Theory and Applications for Advanced Text Mining, Chapter Biomedical Named Entity Recognition: A Survey of Machine-Learning Tools. INTECH Open Access Publisher, Rijeka, Croatia, pp. 175– 195.
  6. Campos D. et al (2013) Gimli: open source and high-performance biomedical name recognition. BMC Bioinformatics, 14. PMC3651325 23413997
  7. Chalapathy R. et al (2016a) Bidirectional LSTM-CRF for clinical concept extraction. In Proceedings of the Clinical Natural Language Processing Workshop, Osaka, Japan, pp. 7–12.
  8. Chalapathy R. et al (2016b) An investigation of recurrent neural architectures for drug name recognition. In Proceedings of the Seventh International Workshop on Health Text Mining and Information Analysis, Austin, TX.
  9. Culotta A., McCallum A. (2004) Confidence estimation for information extraction. In Proceedings of NAACL-HLT, Boston, MA, pp. 109–112.
  10. Dai A.M., Le Q.V. (2015) Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, Montreal, Canada, pp. 3079–3087.
  11. Ding J. et al (2002) Mining MEDLINE: abstracts, sentences, or phrases. In Proceedings of the Pacific Symposium on Biocomputing, Lihue, Hawaii, USA, pp. 326–337. 11928487
  12. Doğan R.I. et al (2014) NCBI disease corpus: a resource for disease name recognition and concept normalization. J. Biomed. Inform., 47, 1– 10. 24393765 PMC3951655
  13. Eltyeb S., Salim N. (2014) Chemical named entities recognition: a review on approaches and applications. J. Cheminform., 6. PMC4022577 24834132
  14. Furlong L.I. et al (2008) OSIRISv1. 2: a named entity recognition system for sequence variants of genes in biomedical literature. BMC Bioinformatics, 9. PMC2277400 18251998
  15. Gerner M. et al (2010) LINNAEUS: a species name identification system for biomedical literature. BMC Bioinformatics, 11. PMC2836304 20149233
  16. Goldberg T. et al (2015) Linked annotations: a middle ground for manual curation of biomedical databases and text corpora. BMC Proc., 9, 1– 3.
  17. Graves A., Schmidhuber J. (2005) Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw., 18, 602– 610. 16112549
  18. Gurulingappa H. et al (2010). An empirical evaluation of resources for the identification of diseases and adverse effects in biomedical literature. In Proceedings of the 2nd Workshop on Building and evaluating resources for biomedical text mining, Valletta, Malta, pp. 15–22.
  19. Habibi M. et al (2016) Recognizing chemicals in patents - a comparative analysis. J. Cheminform., 8, 1– 15. 27843493 PMC5086069
  20. Hahn U. et al (2010). A proposal for a configurable silver standard. In Proceedings of the 4th Linguistic Annotation Workshop at ACL, Uppsala, Sweden, pp. 235–242.
  21. Hastie T. et al (2001) The Elements of Statistical Learning, Vol. 1 Springer series in statistics Springer, Berlin.
  22. Hettne K.M. et al (2009) A dictionary to identify small molecules and drugs in free text. Bioinformatics, 25, 2983– 2991. 19759196
  23. Jozefowicz R. et al (2016). Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410v2. · arXiv
  24. Kaewphan S. et al (2016) Cell line name recognition in support of the identification of synthetic lethality in cancer from text. Bioinformatics, 32, 276– 282. 26428294 PMC4708107
  25. Kim J.-D. et al (2004). Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications, Geneva, Switzerland, pp. 70–75.
  26. Krallinger M. et al (2013) Overview of the chemical compound and drug name recognition (CHEDNER) task. In BioCreative Challenge Evaluation Workshop, Washington, DC, vol. 2, pp. 2–33.
  27. Krallinger M. et al (2015a) The CHEMDNER corpus of chemicals and drugs and its annotation principles. J. Cheminform., 7, 1– 17. 25810773 PMC4331692
  28. Krallinger M. et al (2015b) Overview of the CHEMDNER patents task. In Proceedings of the 5th BioCreative Challenge Evaluation Workshop, Sevilla, Spain, pp. 63–75.
  29. Kulick S. et al (2004) Integrated annotation for biomedical information extraction. In Proceedings of NAACL-HLT, Boston, MA, pp. 61–68.
  30. Lafferty J. et al (2001) Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML, pp. 282–289.
  31. Lample G. et al (2016) Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, San Diego, CA, pp. 260–270.
  32. Leaman R., Gonzalez G. (2008) BANNER: an executable survey of advances in biomedical named entity recognition. In Proceedings of the Pacific Symposium on Biocomputing, Big Island, Hawaii, USA, pp. 652– 663. 18229723
  33. Leaman R., Lu Z. (2016) TaggerOne: Joint named entity recognition and normalization with Semi-Markov models. Bioinformatics, 2839– 2846. 27283952 PMC5018376
  34. Leaman R. et al (2009). Enabling recognition of diseases in biomedical text with machine learning: corpus and benchmark. In Proceedings of the Symposium on Languages in Biology and Medicine, Seogwipo-si, Jeju Island, South Korea, pp. 82–89.
  35. Leaman R. et al (2013) DNorm: disease name normalization with pairwise learning to rank. Bioinformatics, 29, 2909– 2917. 23969135 PMC3810844
  36. Leaman R. et al (2015) tmChem: a high performance approach for chemical named entity recognition and normalization. J. Cheminform., 7. PMC4331693 25810774
  37. Leser U., Hakenberg J. (2005) What makes a gene name? Named entity recognition in the biomedical literature. Brief. Bioinform., 6, 357– 369. 16420734
  38. Li J. et al (2016) BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database 2016. PMC4860626 27161011
  39. Mackin R. (1978) On collocations: Words shall be known by the company they keep. In Honour of as Hornby, pp. 149–165.
  40. Narayanaswamy M. et al (2003) A biological named entity recognizer. In Proceedings of the Pacific Symposium on Biocomputing, Lihue, Hawaii, USA, pp. 427–438. 12603047
  41. Neves M. et al (2012) Annotating and evaluating text for stem cell research. In Proceedings of the 3rd Workshop on Building and Evaluation Resources for Biomedical Text Mining at Language Resources and Evaluation, Istanbul, Turkey, pp. 16–23.
  42. Pafilis E. et al (2013) The SPECIES and ORGANISMS resources for fast and accurate identification of taxonomic names in text. PLoS One, 8, 1–6. PMC3688812 23823062
  43. Pascanu R. et al (2014) How to construct deep recurrent neural networks. In Proceedings of the 2nd International Conference on Learning Representations, Banff, Canada.
  44. Pyysalo S. et al (2007) BioInfer: a corpus for information extraction in the biomedical domain. BMC Bioinformatics, 8. PMC1808065 17291334
  45. Pyysalo S. et al (2013) Distributional semantics resources for biomedical text processing. In Proceedings of the 5th International Symposium on Languages in Biology and Medicine, Tokyo, Japan.
  46. Rabiner L.R. (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, Vol. 77, pp. 257– 286.
  47. Rocktäschel T. et al (2012) ChemSpot: a hybrid system for chemical named entity recognition. Bioinformatics, 28, 1633– 1640. 22500000
  48. Segura-Bedmar I. et al (2015). Combining conditional random fields and word embeddings for the CHEMDNER-patents task. In Proceedings of the 5th BioCreative challenge evaluation workshop, Sevilla, Spain, pp. 90–93.
  49. Settles B. (2005) ABNER: an open source tool for automatically tagging genes, proteins and other entity names in text. Bioinformatics, 21, 3191– 3192. 15860559
  50. Smith L. et al (2008) Overview of BioCreative II gene mention recognition. Genome Biol., 9, 1– 19. PMC2559986 18834493
  51. Stenetorp P. et al (2012). Size (and domain) matters: Evaluating semantic word space representations for biomedical text. In Proceedings of the 5th International Symposium on Semantic Mining in Biomedicine, Zurich, Switzerland, pp. 42–49.
  52. Tang B. et al (2014) Evaluating word representation features in biomedical named entity recognition tasks. BioMed Res. Int., 2014, 1– 6. PMC3963372 24729964
  53. Tjong Kim Sang E.F., De Meulder F. (2003). Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the 7th conference on Natural language learning at HLT-NAACL, Edmonton, Canada, pp. 142–147.
  54. Uzuner Ö. et al (2011) 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. J. Am. Med. Inform. Assoc., 18, 552– 556. 21685143 PMC3168320
  55. Verspoor K. et al (2013) Annotating the biomedical literature for the human variome. Database 2013. PMC3676157 23584833
  56. Wang X. et al (2010) Disambiguating the species of biomedical named entities using natural language parsers. Bioinformatics, 26, 661– 667. 20053840 PMC2828111
  57. Wang Z.-Y., Zhang H.-Y. (2013) Rational drug repositioning by medical genetics. Nat. Biotechnol., 31, 1080– 1082. 24316641
  58. Wei C.-H. et al (2015) Overview of the BioCreative V chemical disease relation (CDR) task. In Proceedings of the 5th BioCreative challenge evaluation workshop, Sevilla, Spain, pp. 154–166.
  59. Zhou X. et al (2014) Human symptoms-disease network. Nat. Commun., 5. 24967666

Cited by

Sorted by citations

Authors
Habibi, M., Weber, L., Neves, M., Wiegandt, D. L., & Leser, U.
Affiliations
  • Humboldt-Universität zu Berlin (DE)
  • Hasso Plattner Institute (DE)
Corresponding author
Habibi, Maryam
ORCID
Venue
Bioinformatics
Publisher
Oxford University Press (OUP)
Published
2017-07-12
Volume
33
Issue
14
Pages
i37-i48
ISSN
  • 1367-4803 (ISSN-L)
  • 1367-4811
DOI
10.1093/bioinformatics/btx228
PMC
PMC5870729
OpenAlex
W2734608416
Cited by
542
References
3
Citations by year
2026: 14 · 2025: 19 · 2024: 37 · 2023: 55 · 2022: 81 · 2021: 93 · 2020: 101 · 2019: 91 · 2018: 45 · 2017: 5 · 2016: 1
FWCI
27.04
Type
journal article
Language
en
Open access
Licence
Locations
  • Bioinformatics (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Funders
  • BMBF
  • DFG
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Keywords
biomedical named entity recognition, word embeddings, LSTM-CRF, text mining, gene identification, chemical recognition, disease identification
MeSH terms
Machine Learning, Animals, Humans, Software, Mice, Data Mining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review