CoChat
Sign Up
Cited by 62Biomedical Text Mining and OntologiesBioinformatics and Genomic NetworksTopic Modeling

BioConceptVec: Creating and evaluating literature-based biomedical concept embeddings on a large scale

Open access
Chen, Qingyu, Lee, Kyubum, Yan, Shankai, Kim, Sun, Wei, Chih-Hsuan, Lu, Zhiyong·2020·PLOS Computational Biology
DOI 10.1371/journal.pcbi.1007617PMC PMC7237030
Open this paper in CoChat — free accountOpen PDF

Abstract

A massive number of biological entities, such as genes and mutations, are mentioned in the biomedical literature. The capturing of the semantic relatedness of biological entities is vital to many biological applications, such as protein-protein interaction prediction and literature-based discovery. Concept embeddings-which involve the learning of vector representations of concepts using machine learning models-have been employed to capture the semantics of concepts. To develop concept embeddings, named-entity recognition (NER) tools are first used to identify and normalize concepts from the literature, and then different machine learning models are used to train the embeddings. Despite multiple attempts, existing biomedical concept embeddings generally suffer from suboptimal NER tools, small-scale evaluation, and limited availability. In response, we employed high-performance machine learning-based NER tools for concept recognition and trained our concept embeddings, BioConceptVec, via four different machine learning models on ~30 million PubMed abstracts. BioConceptVec covers over 400,000 biomedical concepts mentioned in the literature and is of the largest among the publicly available biomedical concept embeddings to date. To evaluate the validity and utility of BioConceptVec, we respectively performed two intrinsic evaluations (identifying related concepts based on drug-gene and gene-gene interactions) and two extrinsic evaluations (protein-protein interaction prediction and drug-drug interaction extraction), collectively using over 25 million instances from nine independent datasets (17 million instances from six intrinsic evaluation tasks and 8 million instances from three extrinsic evaluation tasks), which is, by far, the most comprehensive to our best knowledge. The intrinsic evaluation results demonstrate that BioConceptVec consistently has, by a large margin, better performance than existing concept embeddings in identifying similar and related concepts. More importantly, the extrinsic evaluation results demonstrate that using BioConceptVec with advanced deep learning models can significantly improve performance in downstream bioinformatics studies and biomedical text-mining applications. Our BioConceptVec embeddings and benchmarking datasets are publicly available at https://github.com/ncbi-nlp/BioConceptVec.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pcbi.1007617

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This relevance filtering step is critical, as biomedical concepts frequently exhibit high ambiguity in free text, and exact string-matching approaches like SKiM frequently retrieve abstracts containing ambiguous or irrelevant matches.

    “This relevance filtering step is critical, as biomedical concepts frequently exhibit high ambiguity in free text [37], and exact string-matching approaches like SKiM frequently retrieve abstracts containing ambiguous or irrelevant matches [8].”

    From SKiM-GPT: combining biomedical literature-based discovery with large language model hypothesis evaluation (2025) · cites this paper as [8]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 69 · Cited by 62

Research works printed but not linked: 61

  1. [1]Singhal A, Leaman R, Catlett N, Lemberger T, McEntyre J, Polson S, et al Pressing needs of biomedical text mining in biocuration and beyond: opportunities and challenges. Database. 2016. PMC5199160 28025348
  2. [2]Lu Z, Hirschman L. Biocuration workflows and text mining: overview of the BioCreative 2012 Workshop Track II. Database. 2012. PMC3500522 23160416
  3. [3]Henry S, McInnes BT. Literature based discovery: models, methods, and trends. Journal of biomedical informatics. 2017; 74: 20– 32. 10.1016/j.jbi.2017.08.011 28838802 · DOI · Paper page
  4. [4]Ningthoujam D, Yadav S, Bhattacharyya P, Ekbal A. Relation extraction between the clinical entities based on the shortest dependency path based LSTM. arXiv preprint arXiv:190309941. 2019.
  5. [5]Zheng JG, Howsmon D, Zhang B, Hahn J, McGuinness D, Hendler J, et al Entity linking for biomedical literature. BMC medical informatics and decision making. 2015; 15(1): S4. PMC4460707 26045232
  6. [6]Doğan RI, Kim S, Chatr-aryamontri A, Wei C-H, Comeau DC, Antunes R, et al Overview of the BioCreative VI Precision Medicine Track: mining protein interactions and mutations for precision medicine. Database: the journal of biological databases and curation. 2019. PMC6348314 30689846
  7. [7]Erk K. Vector space models of word meaning and phrase meaning: A survey. Language and Linguistics Compass. 2012; 6(10): 635– 53.
  8. [8]Li Y, Yang T. Word embedding for understanding natural language: a survey Guide to Big Data Applications: Springer; 2018 p. 83– 104.
  9. [9]Park J, Kim K, Hwang W, Lee D. Concept Embedding to Measure Semantic Relatedness for Biomedical Information Ontologies. Journal of Biomedical Informatics. 2019: 103182 10.1016/j.jbi.2019.103182 31009761 · DOI
  10. [10]Xiang Y, Xu J, Si Y, Li Z, Rasmy L, Zhou Y, et al Time-sensitive clinical concept embeddings learned from large electronic health records. BMC medical informatics and decision making. 2019; 19(2): 58. 30961579 PMC6454598
  11. [11]Beam AL, Kompa B, Fried I, Palmer NP, Shi X, Cai T, et al. Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data. arXiv preprint arXiv:180401486. 2018. PMC6922053 31797605
  12. [12]Choi Y, Chiu CY-I, Sontag D. Learning low-dimensional representations of medical concepts. AMIA Summits on Translational Science Proceedings. 2016; 2016: 41. PMC5001761 27570647
  13. [13]Ma Y, Cambria E. Concept-Based Embeddings for Natural Language Processing. arXiv preprint arXiv:180705519. 2018.
  14. [14]Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J, editors. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems; 2013.
  15. [15]Pennington J, Socher R, Manning C, editors. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP); 2014.
  16. [16]Mikolov T, Grave E, Bojanowski P, Puhrsch C, Joulin A. Advances in pre-training distributed word representations. arXiv preprint arXiv:171209405. 2017.
  17. [17]Aggarwal CC, Zhai C. Mining text data: Springer Science & Business Media; 2012.
  18. [18]Lee K, Famiglietti ML, McMahon A, Wei C-H, MacArthur JAL, Poux S, et al Scaling up data curation using deep learning: An application to literature triage in genomic variation resources. PLoS computational biology. 2018; 14(8): e1006390 10.1371/journal.pcbi.1006390 30102703 PMC6107285 · DOI · Paper page
  19. [19]Allot A, Chen Q, Kim S, Vera Alvarez R, Comeau DC, Wilbur WJ, et al LitSense: making sense of biomedical literature at sentence level. Nucleic acids research. 2019. PMC6602490 31020319
  20. [20]Dimitriadis D, Tsoumakas G. Word embeddings and external resources for answer processing in biomedical factoid question answering. Journal of biomedical informatics. 2019; 92: 103118 10.1016/j.jbi.2019.103118 30753948 · DOI
  21. [21]Wei C-H, Lee K, Leaman R, Lu Z, editors. Biomedical Mention Disambiguation using a Deep Learning Approach. Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics; 2019: ACM.
  22. [22]Du J, Jia P, Dai Y, Tao C, Zhao Z, Zhi D. Gene2vec: distributed representation of genes based on co-expression. BMC genomics. 2019; 20(1): 82. 30712510 PMC6360648
  23. [23]Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research. 2004; 32(suppl_1): D267– D70. 14681409 PMC308795
  24. [24]Hassanzadeh H, Nguyen A, Koopman B, editors. Evaluation of medical concept annotation systems on clinical records. Proceedings of the Australasian Language Technology Association Workshop 2016; 2016.
  25. [25]Lin Y-C, Christen V, Groß A, Cardoso SD, Pruski C, Da Silveira M, et al., editors. Evaluating and improving annotation tools for medical forms. International Conference on Data Integration in the Life Sciences; 2017: Springer.
  26. [26]Reátegui R, Ratté S. Comparison of MetaMap and cTAKES for entity extraction in clinical notes. BMC medical informatics and decision making. 2018; 18(3): 74. 30255810 PMC6157281
  27. [27]Suominen H, Zhou L, Hanlen L, Ferraro G. Benchmarking clinical speech recognition and information extraction: new data, methods, and evaluations. JMIR medical informatics. 2015; 3(2): e19 10.2196/medinform.4321 25917752 PMC4427705 · DOI
  28. [28]Pradhan S, Elhadad N, South BR, Martinez D, Christensen L, Vogel A, et al Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. Journal of the American Medical Informatics Association. 2014; 22(1): 143– 54. 10.1136/amiajnl-2013-002544 25147248 PMC4433360 · DOI
  29. [29]Schnabel T, Labutov I, Mimno D, Joachims T, editors. Evaluation methods for unsupervised word embeddings. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing; 2015.
  30. [30]Choi E, Bahadori MT, Sun J, Kulas J, Schuetz A, Stewart W, editors. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in Neural Information Processing Systems; 2016.
  31. [31]Pakhomov S, McInnes B, Adam T, Liu Y, Pedersen T, Melton GB, editors. Semantic similarity and relatedness between clinical terms: an experimental study. AMIA annual symposium proceedings; 2010: American Medical Informatics Association. PMC3041430 21347043
  32. [32]Wei C-H, Kao H-Y, Lu Z. PubTator: a web-based text mining tool for assisting biocuration. Nucleic acids research. 2013; 41(W1): W518– W22. 23703206 PMC3692066
  33. [33]Szklarczyk D, Gable AL, Lyon D, Junge A, Wyder S, Huerta-Cepas J, et al STRING v11: protein–protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. Nucleic acids research. 2018; 47(D1): D607– D13. PMC6323986 30476243
  34. [34]Chen Q, Peng Y, Lu Z, editors. BioSentVec: creating sentence embeddings for biomedical texts. 2019 IEEE International Conference on Healthcare Informatics (ICHI); 2019: IEEE.
  35. [35]Leaman R, Lu Z. TaggerOne: joint named entity recognition and normalization with semi-Markov Models. Bioinformatics. 2016; 32(18): 2839– 46. Epub 2016/06/11. 10.1093/bioinformatics/btw343 27283952 PMC5018376 · DOI
  36. [36]Wei C-H, Kao H-Y, Lu Z. GNormPlus: an integrative approach for tagging genes, gene families, and protein domains. BioMed research international. 2015; 2015. PMC4561873 26380306
  37. [37]Wei C-H, Phan L, Feltz J, Maiti R, Hefferon T, Lu Z. tmVar 2.0: integrating genomic variant information from literature with dbSNP and ClinVar for precision medicine. Bioinformatics. 2017; 34(1): 80– 7. PMC5860583 28968638
  38. [38]Wei CH, Kao HY, Lu Z. SR4GN: a species recognition software tool for gene normalization. PLoS One. 2012; 7(6): e38460 Epub 2012/06/09. 10.1371/journal.pone.0038460 22679507 PMC3367953 · DOI
  39. [39]Wei C-H, Allot A, Leaman R, Lu Z. PubTator central: automated concept annotation for biomedical full text articles. Nucleic acids research 2019. PMC6602571 31114887
  40. [40]Wang Y, Wang J, Lin H, Tang X, Zhang S, Li L. Bidirectional long short-term memory with CRF for detecting biomedical event trigger in FastText semantic space. BMC bioinformatics. 2018; 19(20): 507. 30577839 PMC6302454
  41. [41]Jin D, Szolovits P. Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts. arXiv preprint arXiv:180806161. 2018.
  42. [43]Yu Z, Wallace BC, Johnson T, Cohen T. Retrofitting concept vector representations of medical concepts to improve estimates of semantic similarity and relatedness. arXiv preprint arXiv:170907357. 2017. PMC6464117 29295178
  43. [44]Chen Q, Du J, Kim S, Wilbur WJ, Lu Z. Deep learning with sentence embeddings pre-trained on biomedical corpora improves the performance of finding similar sentences in electronic medical records. arXiv preprint arXiv:190903044. 2019. PMC7191680 32349758
  44. [45]Jang M, Kang P. Paraphrase Thought: Sentence Embedding Module Imitating Human Language Recognition. arXiv preprint arXiv:180805505. 2018.
  45. [46]Chiu B, Crichton G, Korhonen A, Pyysalo S, editors. How to train good word embeddings for biomedical NLP. Proceedings of the 15th Workshop on Biomedical Natural Language Processing; 2016.
  46. [47]De Vine L, Zuccon G, Koopman B, Sitbon L, Bruza P, editors. Medical semantic similarity with a neural language model. Proceedings of the 23rd ACM international conference on conference on information and knowledge management; 2014: ACM.
  47. [49]Barabasi A-L, Oltvai ZN. Network biology: understanding the cell's functional organization. Nature reviews genetics. 2004; 5(2): 101 10.1038/nrg1272 14735121 · DOI · Paper page
  48. [50]Hartwell LH, Hopfield JJ, Leibler S, Murray AW. From molecular to modular cell biology. Nature. 1999; 402(6761supp): C47. 10591225
  49. [51]Davis AP, Grondin CJ, Johnson RJ, Sciaky D, McMorran R, Wiegers J, et al The comparative toxicogenomics database: Update 2019. Nucleic acids research. 2018; 47(D1): D948– D54. PMC6323936 30247620
  50. [52]Liberzon A, Subramanian A, Pinchback R, Thorvaldsdóttir H, Tamayo P, Mesirov JP. Molecular signatures database (MSigDB) 3.0. Bioinformatics. 2011; 27(12): 1739– 40. 10.1093/bioinformatics/btr260 21546393 PMC3106198 · DOI · Paper page
  51. [54]Segura Bedmar I, Martínez P, Herrero Zazo M, editors. Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013)2013: Association for Computational Linguistics.
  52. [56]Smaili FZ, Gao X, Hoehndorf R. Onto2vec: joint vector-based representation of biological entities and their ontology-based annotations. Bioinformatics. 2018; 34(13): i52– i60. 10.1093/bioinformatics/bty259 29949999 PMC6022543 · DOI
  53. [57]Smaili FZ, Gao X, Hoehndorf R. Opa2vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction. arXiv preprint arXiv:180410922. 2018. 30407490
  54. [58]Wishart DS, Feunang YD, Guo AC, Lo EJ, Marcu A, Grant JR, et al DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic acids research. 2017; 46(D1): D1074– D82. PMC5753335 29126136
  55. [59]Herrero-Zazo M, Segura-Bedmar I, Martínez P, Declerck T. The DDI corpus: An annotated corpus with pharmacological substances and drug–drug interactions. Journal of biomedical informatics. 2013; 46(5): 914– 20. 10.1016/j.jbi.2013.07.011 23906817 · DOI
  56. [61]Chauhan G, McDermott M, Szolovits P. Reflex: Flexible Framework for Relation Extraction in Multiple Domains. arXiv preprint arXiv:190608318. 2019.
  57. [62]Peters ME, Ammar W, Bhagavatula C, Power R. Semi-supervised sequence tagging with bidirectional language models. arXiv preprint arXiv:170500108. 2017.
  58. [63]Zhang Y, Zheng W, Lin H, Wang J, Yang Z, Dumontier M. Drug–drug interaction extraction via hierarchical RNNs on sequence and shortest dependency paths. Bioinformatics. 2017; 34(5): 828– 35. PMC6030919 29077847
  59. [64]Choi E, Bahadori MT, Searles E, Coffey C, Thompson M, Bost J, et al., editors. Multi-layer representation learning for medical concepts. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016: ACM.
  60. [65]Cai X, Gao J, Ngiam KY, Ooi BC, Zhang Y, Yuan X. Medical concept embedding with time-aware attention. arXiv preprint arXiv:180602873. 2018.
  61. [66]Nguyen K, Ichise R, editors. Learning Effective Distributed Representation of Complex Biomedical Concepts. 2018 IEEE 18th International Conference on Bioinformatics and Bioengineering (BIBE); 2018: IEEE.

Authors
Chen, Q., Lee, K., Yan, S., Kim, S., Wei, C.-H., & Lu, Z.
Affiliations
National Institutes of Health (US)
Corresponding author
Lu, Zhiyong
ORCID
Venue
PLOS Computational Biology
Publisher
Public Library of Science (PLoS)
Published
2020-04-23
Volume
16
Issue
4
Pages
e1007617
ISSN
  • 1553-734X (ISSN-L)
  • 1553-7358
DOI
10.1371/journal.pcbi.1007617
PMC
PMC7237030
OpenAlex
W3017463390
Cited by
62
References
69
Citations by year
2026: 4 · 2025: 8 · 2024: 5 · 2023: 10 · 2022: 10 · 2021: 19 · 2020: 5 · 2018: 1
FWCI
3.65
Type
journal article
Language
en
Update notices
new_version (2020-05-19)
Open access
Licence
Locations
  • PLoS Computational Biology (publishedVersion, public-domain)
  • PubMed Central (publishedVersion, public-domain)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
the Intramural Research Program of the NIH, National Library of Medicine
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
named entity recognition, biomedical literature, gene-gene interaction, drug-gene interactions, protein-protein interaction prediction, drug-drug interaction extraction, PubMed abstracts, biomedical text mining, benchmark dataset
MeSH terms
Deep Learning, Algorithms, Drug Interactions, Humans, Publications, Semantics, Computational Biology, Protein Interaction Mapping, Databases, Protein, PubMed, Data Mining, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review